Skip to content

fix(haystack): record actual generator replies instead of the "replies" key - #4721

Open
DZH166 wants to merge 1 commit into
traceloop:mainfrom
DZH166:codex/haystack-replies-20261119
Open

DZH166 wants to merge 1 commit into
traceloop:mainfrom
DZH166:codex/haystack-replies-20261119

Conversation

@DZH166

@DZH166 DZH166 commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

In Haystack 2.x, OpenAIChatGenerator.run() and OpenAIGenerator.run() return their results as a dict: {"replies": [ChatMessage, ...]} / {"replies": [str, ...]} (verified against haystack-ai 2.3.1, the version this repo's lock file pins).

The OpenAI wrapper in this instrumentation passed that whole dict to the completion serializer, which does for index, message in enumerate(choices). Enumerating a dict iterates its keys, so on every Haystack OpenAI call the span recorded:

  • gen_ai.completion.0.content = "replies" (the literal key name)
  • no attribute for the actual reply text; with multiple replies, everything except the key was dropped.

So the haystack.openai.chat / haystack.openai.completion spans never carried what the model actually answered.

The fix unpacks the replies list before serializing, and for the chat path serializes a ChatMessage via its .content string (previously a raw ChatMessage object would have been set as the attribute value, which the OTel API silently drops). Behavior when the wrapped call returns a non-dict is unchanged.

Validation

Inside packages/opentelemetry-instrumentation-haystack (uv-managed, haystack-ai==2.3.1):

  • Added two regression tests that invoke the wrapper with a stubbed generator run returning a Haystack-shaped result:
    • test_chat_generator_records_reply_content (chat path, ChatMessage replies)
    • test_generator_records_reply_content (plain generator path, string replies)
  • Before the fix, both fail with assert 'replies' == 'Hello there' / assert 'replies' == 'first reply' (red log attached).
  • After the fix: uv run --no-sync pytest tests/ → 4 passed (2 existing vcr tests + 2 new) (green log attached).
  • uv run --no-sync ruff check . → All checks passed. git diff --check clean.

AI assistance

This change was prepared with AI coding assistance (bug located by code inspection against the pinned haystack-ai version; tests and fix written and verified locally). The CI workflow does not forbid AI-assisted contributions; disclosing here for transparency.

No public issue tracked this specific defect (the related open issue #1169 reports missing traceloop.entity.input/output on the workflow root span, a different facet).

Summary by CodeRabbit

  • Bug Fixes
    • Haystack chat and text generator replies are now recorded more accurately in telemetry, including message content, assistant roles, and text completions.
    • Responses wrapped in a dictionary are now handled correctly, while other response formats continue to be supported.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The Haystack instrumentation now processes dictionary responses through their replies value and records chat message content when available. Tests cover recorded attributes for chat and text generator replies.

Changes: Generator reply attributes

Layer / File(s) Summary
Process generator replies
packages/opentelemetry-instrumentation-haystack/opentelemetry/instrumentation/haystack/wrap_openai.py, packages/opentelemetry-instrumentation-haystack/tests/test_simple_pipeline.py
Dictionary responses are unwrapped through replies. Chat completion attributes use message.content when available, with the original message as fallback. Tests check chat content, assistant role, and text completion content.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: nirga

Merge Risk: Minimal · up to 4b1d7

This fixes Haystack OpenAI spans so they record the generator's actual reply text instead of the literal "replies" key. The change is narrow, covered by new regression tests, and has no identified merge-blocking risk.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title says the fix records the actual generator replies instead of the literal "replies" key. The raw summary confirms this: _set_response_attributes now unwraps dict responses via their "replies"…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title says the fix records the actual generator replies instead of the literal "replies" key. The raw summary confirms this: _set_response_attributes now unwraps dict responses via their "replies" value, and chat messages use their content. The added tests check that reply content is recorded. The title matches the main change and uses the conventional fix(haystack) prefix.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 926422fd-97c4-4a73-8122-a4773790785e
Commits

Reviewing files that changed from the base of the PR and between 285c7bf and 4b1d7de.

Files selected for processing: 2

  • packages/opentelemetry-instrumentation-haystack/opentelemetry/instrumentation/haystack/wrap_openai.py
  • packages/opentelemetry-instrumentation-haystack/tests/test_simple_pipeline.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants