Skip to content

Drop client-side validation of tool results against the output schema - #3865

Open
Kludex wants to merge 1 commit into
mainfrom
drop-tool-output-validation
Open

Kludex wants to merge 1 commit into
mainfrom
drop-tool-output-validation

Conversation

@Kludex

@Kludex Kludex commented Nov 16, 2026

Copy link
Copy Markdown
Member

ClientSession.call_tool checked every tool result's structured_content against the outputSchema from tools/list, using jsonschema. The server writes both the schema and the result, so the check could only tell you that a server disagrees with itself. It also cost a dependency tree (jsonschema, jsonschema-specifications, referencing, rpds-py) and an implicit tools/list the first time you called each tool, which a server that only handles tools/call answered with METHOD_NOT_FOUND.

This removes it. call_tool now returns structured_content exactly as the server sent it and sends no tools/list of its own. To rely on the shape, validate it against a model you own:

book = Book.model_validate(result.structured_content)

ClientSession.validate_tool_result() is public, so it stays as a deprecated no-op. The session still caches each tool's output_schema from tools/list, because downstream clients such as FastMCP read it to build typed results.

The spec says clients SHOULD validate structured results, so the interaction suite now records this as a divergence. The Go, Rust and C# SDK clients don't validate them either.

Most of the test diff is list_tools handlers that only existed to answer the implicit tools/list.

AI Disclaimer

This PR was developed with the assistance of either Claude or Codex. I've reviewed and verified the changes.

Session: claude --resume 25d1e11b-aaa2-4dbf-bb33-c05d1f09b27a

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T11:47:43.504813Z 6ad1a7a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

📚 Documentation preview

Preview https://pr-3865.mcp-python-docs.pages.dev
Deployment https://1c1607cf.mcp-python-docs.pages.dev
Commit 6ad1a7a
Triggered by @Kludex
Updated 2026-11-16 11:48:57 UTC

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 46 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/interaction/_requirements.py">

<violation number="1" location="tests/interaction/_requirements.py:963">
P2: This `behavior` field describes the SDK’s non-validation, but the requirement cites the spec and its divergence note says clients SHOULD validate. State the spec’s validation expectation here so the manifest does not record the divergence as required behavior.</violation>
</file>

<file name="tests/interaction/transports/test_legacy_wire.py">

<violation number="1" location="tests/interaction/transports/test_legacy_wire.py:39">
P3: This server no longer handles `tools/list`, so `_server()`'s docstring incorrectly says the exchange covers it. Update the docstring to describe the `tools/call`-only exchange.</violation>
</file>

<file name="tests/interaction/lowlevel/test_list_changed.py">

<violation number="1" location="tests/interaction/lowlevel/test_list_changed.py:52">
P2: This removes the server's `tools` capability while the test still emits `notifications/tools/list_changed`, so the notification no longer comes from a server that declares tool support. Keep a non-invoked `on_list_tools` handler to advertise that capability, as the resource and prompt cases do for theirs.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

behavior=(
"Calling a tool whose output schema is not yet cached issues an implicit tools/list to "
"populate the cache; subsequent calls of the same tool do not."
"is returned to the caller unchanged; the client does not validate it."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This behavior field describes the SDK’s non-validation, but the requirement cites the spec and its divergence note says clients SHOULD validate. State the spec’s validation expectation here so the manifest does not record the divergence as required behavior.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/interaction/_requirements.py, line 963:

<comment>This `behavior` field describes the SDK’s non-validation, but the requirement cites the spec and its divergence note says clients SHOULD validate. State the spec’s validation expectation here so the manifest does not record the divergence as required behavior.</comment>

<file context>
@@ -956,35 +956,24 @@ def __post_init__(self) -> None:
-        behavior=(
-            "Calling a tool whose output schema is not yet cached issues an implicit tools/list to "
-            "populate the cache; subsequent calls of the same tool do not."
+            "is returned to the caller unchanged; the client does not validate it."
         ),
         divergence=Divergence(
</file context>
Suggested change
"is returned to the caller unchanged; the client does not validate it."
"should be validated by the client against the declared output schema."

return CallToolResult(content=[TextContent(text="installed")])

server = Server("registry", on_list_tools=list_tools, on_call_tool=call_tool)
server = Server("registry", on_call_tool=call_tool)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This removes the server's tools capability while the test still emits notifications/tools/list_changed, so the notification no longer comes from a server that declares tool support. Keep a non-invoked on_list_tools handler to advertise that capability, as the resource and prompt cases do for theirs.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/interaction/lowlevel/test_list_changed.py, line 52:

<comment>This removes the server's `tools` capability while the test still emits `notifications/tools/list_changed`, so the notification no longer comes from a server that declares tool support. Keep a non-invoked `on_list_tools` handler to advertise that capability, as the resource and prompt cases do for theirs.</comment>

<file context>
@@ -44,17 +44,12 @@ async def collect(message: IncomingMessage) -> None:
         return CallToolResult(content=[TextContent(text="installed")])
 
-    server = Server("registry", on_list_tools=list_tools, on_call_tool=call_tool)
+    server = Server("registry", on_call_tool=call_tool)
 
     async with connect(server, message_handler=collect) as client:
</file context>

return CallToolResult(content=[TextContent(text=str(params.arguments["text"]))])

return Server("legacy", on_list_tools=list_tools, on_call_tool=call_tool)
return Server("legacy", on_call_tool=call_tool)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This server no longer handles tools/list, so _server()'s docstring incorrectly says the exchange covers it. Update the docstring to describe the tools/call-only exchange.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/interaction/transports/test_legacy_wire.py, line 39:

<comment>This server no longer handles `tools/list`, so `_server()`'s docstring incorrectly says the exchange covers it. Update the docstring to describe the `tools/call`-only exchange.</comment>

<file context>
@@ -34,15 +31,12 @@
         return CallToolResult(content=[TextContent(text=str(params.arguments["text"]))])
 
-    return Server("legacy", on_list_tools=list_tools, on_call_tool=call_tool)
+    return Server("legacy", on_call_tool=call_tool)
 
 
</file context>

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline findings, I also checked the jsonschema dependency removal for leftovers: no Python file in the checkout still imports jsonschema or referencing, and no pyproject.toml still lists it. I also checked for remaining callers of the now-deprecated ClientSession.validate_tool_result() under src/ (which would trip filterwarnings = ["error"]) — the only reference left is its definition at src/mcp/client/session.py:1101.

Extended reasoning...

The change removes client-side output-schema validation from ClientSession.call_tool and Client.call_tool, turns validate_tool_result into a deprecated no-op, and drops the jsonschema runtime dependency, with most of the 46-file diff being test cleanup. It touches no auth, crypto or permission code, but it does change the observable behaviour of a released 2.x public API, and the inline findings are left for a maintainer to weigh.

Comment thread src/mcp/client/session.py
if isinstance(result, types.CallToolResult) and not result.is_error:
await self.validate_tool_result(name, result)

# The input_required arm stays first; a claimed shape is terminal for the multi-round-trip driver.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Direct ClientSession.call_tool callers on 2026-07-28 HTTP can now get HEADER_MISMATCH for tools with x-mcp-header arguments that worked before. The implicit tools/list removed here was also what filled _x_mcp_header_maps after a first successful call. Without it _resolve_param_headers (session.py:1089) sends no Mcp-Param-* headers until the caller lists tools. Fix: make Mcp-Param-* headers available for a called tool without an explicit listing, e.g. give ClientSession.call_tool the re-list-and-resend recovery at client.py:843, or document that list_tools() must come first. [also at: src/mcp/client/session.py:1080 - Code that calls ClientSession.call_tool directly on a 2026-07-28 HTTP connection can now get HEADER_MISMATCH for x-mcp-header tools where the base recovered by itself.; src/mcp/client/client.py:853 - Client users on a 2026-07-28 HTTP connection now hit a rejected call, a re-list and a resend for some calls that succeeded in one round trip on the base.; +1 more]

Why this was flagged

A caller holds a ClientSession on a 2026-07-28 Streamable HTTP connection, never calls list_tools(), calls tool A, then calls tool B whose input schema marks an argument x-mcp-header. On the base, A's non-error result ran validate_tool_result, whose await self.list_tools() went through _absorb_tool_listing and set self._x_mcp_header_maps[tool.name] (src/mcp/client/session.py:1227) for every listed tool, so B's call carried its header. After the change nothing lists tools, _resolve_param_headers returns {} (src/mcp/client/session.py:1092-1093), and the server rejects B with HEADER_MISMATCH "header is missing but the request body's ... argument is present" (src/mcp/shared/inbound.py:579-583). ClientSession.call_tool has no retry, so the caller gets an MCPError where the base returned the result. Client.call_tool recovers via _relist_tool (src/mcp/client/client.py:843-853) but pays one rejected tools/call the base did not.

Verification: _x_mcp_header_maps is written in only one place, _absorb_tool_listing (src/mcp/client/session.py:1227). _resolve_param_headers (lines 1089-1094) returns {} when the tool has no cached map. ClientSession.call_tool has no recovery. So on the base, call A then call B (annotated argument) succeeded; on head, B is rejected with HEADER_MISMATCH.

Finding ID: f-748c4898

Comment thread src/mcp/client/session.py
Comment on lines +1096 to +1102
@deprecated(
"The client no longer validates tool results against the server's output schema; "
"validate `structured_content` against your own model instead.",
category=MCPDeprecationWarning,
)
async def validate_tool_result(self, name: str, result: types.CallToolResult) -> None:
"""Revalidate a `CallToolResult` against the tool's declared output schema.

Raises:
RuntimeError: Structured content is missing or does not conform to the schema, or the
schema is invalid or has a `$ref` that does not resolve within the schema document.
"""
if name not in self._tool_output_schemas:
# refresh output schema cache
await self.list_tools()

output_schema = None
if name in self._tool_output_schemas:
output_schema = self._tool_output_schemas.get(name)
else:
logger.warning(f"Tool {name} not listed by server, cannot validate any structured content")

if output_schema is not None:
from jsonschema import exceptions as jsonschema_exceptions
from referencing.exceptions import Unresolvable

if result.structured_content is None and "structured_content" not in result.model_fields_set:
raise RuntimeError(f"Tool {name} has an output schema but did not return structured content")
validator = self._output_schema_validator(name, output_schema)
# `best_match` picks the same error the previous `jsonschema.validate()` call raised,
# so the message a caller sees is unchanged. It is untyped upstream.
errors = validator.iter_errors(result.structured_content)
try:
error = cast(
"Exception | None",
jsonschema_exceptions.best_match(errors), # pyright: ignore[reportUnknownMemberType]
)
except Unresolvable as e:
# A `$ref` did not resolve within the schema document.
raise RuntimeError(f"Invalid schema for tool {name}: {e}") from e
if error is not None:
raise RuntimeError(f"Invalid structured content returned by tool {name}: {error}") from error

def _output_schema_validator(self, name: str, output_schema: dict[str, Any]) -> Validator:
"""Compiled validator for the tool's cached output schema, built once per schema value.

Compiling is ~60x the cost of validating, so a one-shot `jsonschema.validate()` per
result dominates `call_tool`; the compiled validator is cached instead. It stays valid
because `_absorb_tool_listing` evicts a tool's validator whenever it absorbs a different
schema for that tool, so a cached entry always matches `output_schema`.

Raises:
RuntimeError: The schema is not a valid JSON Schema. Raised on every call, since a
failed compile is never cached.
"""
from jsonschema import SchemaError
from jsonschema.validators import validator_for
from referencing import Registry

if (validator := self._tool_output_validators.get(name)) is not None:
return validator

validator_cls = validator_for(output_schema)
try:
validator_cls.check_schema(output_schema)
except SchemaError as e:
raise RuntimeError(f"Invalid schema for tool {name}: {e}")
# An explicit empty registry: `$ref`s resolve within the schema document and the bundled metaschemas.
validator = validator_cls(output_schema, registry=Registry())
self._tool_output_validators[name] = validator
return validator
"""Does nothing. The client no longer validates tool results against the server's output schema."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Callers upgrading within 2.x silently lose a check: schema-violating structured_content is now returned instead of raising RuntimeError. AGENTS.md says a change to a released API's observable behaviour, even behind a @ deprecated shim, is an explicit maintainer decision and should generally be avoided. validate_tool_result at session.py:1101 keeps its name but accepts everything, so code using it as a gate fails open. Fix: keep validate_tool_result validating (jsonschema as an optional import with a clear error when absent), or record the maintainer decision for both it and call_tool. [also at: src/mcp/client/session.py:1096 - nit: AGENTS.md (Branching Model) says a change to an existing 2.x API's observable behaviour, even one softened by a @ deprecated shim, is a design decision a maintainer makes explicitly and should generally be avoided.]

Why this was flagged

Trigger: a 2.x application calls ClientSession.call_tool / Client.call_tool on a tool whose listed output_schema the result violates or whose result has no structuredContent, or calls the public ClientSession.validate_tool_result(name, result) directly as a gate. On the base, session.py raised RuntimeError("Invalid structured content returned by tool ...") or "Tool ... has an output schema but did not return structured content", and the claim path in client.py revalidated the resolver product. After the change session.py:1080-1087 returns the result unchecked, client.py:860 returns claim.resolve(...) directly, and session.py:1101-1102 is an empty body that only emits MCPDeprecationWarning. The caller's except RuntimeError branch never runs and malformed data flows on as if validated. The @ deprecated shim does not prevent this: it warns but still returns None, the same value a passing validation returned.

Verification: src/mcp/client/session.py:1096-1102 is a @ deprecated(..., category=MCPDeprecationWarning) method whose body is only the docstring "Does nothing." A tool lists {"x": integer, required} and returns {"x": "no"}. On the base, call_tool raises RuntimeError; on the head, the result is returned and the bad value reaches application code. Callers of call_tool get no signal at all.

Finding ID: f-f4472c1d

Comment on lines 31 to +39
def _server() -> Server:
"""A low-level server with one echo tool, so the recorded exchange covers tools/list and tools/call."""

async def list_tools(ctx: ServerRequestContext, params: PaginatedRequestParams | None) -> ListToolsResult:
return ListToolsResult(tools=[Tool(name="echo", description="Echo text.", input_schema={"type": "object"})])

async def call_tool(ctx: ServerRequestContext, params: CallToolRequestParams) -> CallToolResult:
assert params.name == "echo"
assert params.arguments is not None
return CallToolResult(content=[TextContent(text=str(params.arguments["text"]))])

return Server("legacy", on_list_tools=list_tools, on_call_tool=call_tool)
return Server("legacy", on_call_tool=call_tool)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): The legacy-wire vocabulary guard no longer records a tools/list exchange, so a ttlMs or cacheScope leak on a legacy listing would now pass it. The diff removed the list_tools handler from _server(), and the client at test_legacy_wire.py:64 only sends tools/call. The _server() docstring at line 32 still says the exchange covers tools/list. Fix: keep the handler and call await client.list_tools() explicitly, as test_hosting_http_modern.py:405 now does, and restore len(recorded.frames) >= 6.

Why this was flagged

The trigger is a future change that leaks ttlMs, cacheScope or resultType onto a tools/list result on a connection negotiated at the legacy protocol version. On the base, test_legacy_streamable_http_exchange_carries_no_modern_protocol_vocabulary recorded the implicit tools/list that call_tool sent, so assert_no_modern_vocabulary(recorded) scanned a listing request and response. After this diff _server() at tests/interaction/transports/test_legacy_wire.py:31-39 registers only on_call_tool, the client at line 64 only calls echo, and the frame floor at line 77 drops from 6 to 5. The requirement hosting:http:legacy-no-modern-vocabulary is then pinned only for the handshake and tools/call, while the docstring at line 32 still claims tools/list coverage. The sibling test in tests/interaction/transports/test_hosting_http_modern.py kept its coverage by adding an explicit await session.list_tools(); this one did not. Nothing fails at runtime.

Verification: nit — triggers when a later change leaks ttlMs/cacheScope onto a legacy tools/list result; the legacy-wire test no longer records any listing exchange. The only client call is await client.call_tool("echo", {"text": "legacy"}) (test_legacy_wire.py:64), so the recorded exchange is now handshake + tools/call only.

Finding ID: f-a0aacc5e

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant