Repository navigation
Conversation
…ceholder sep-9998) agents-wg PR 22 has no number in modelcontextprotocol/modelcontextprotocol yet, and both traceability filters are numeric, so the rows and check IDs carry the placeholder 9998. The header explains why and how to rename. 31 graded rows and 18 excluded. Every row quotes the proposal at 9b16840e, matched with line wrapping ignored, and points at the section it sits under.
…ged scenarios Discovery grades the capability, the roster (shape, paging, cache hints, cards, unique names), the details for every listed agent, tool parity with tools/list, and the two error reasons. The hidden-agent and broken-agent rows read their names from the server's agents_conformance_fixture control and report untestable without it. List-changed opens a subscriptions/listen stream with agentsListChanged, grades the acknowledgement either way round, changes the roster through agents_conformance_add_agent, and waits for the notification and the re-listed roster. Registered as pending, since the everything-server does not implement the extension.
A stateless fixture built per test, conformant by default, with one option per rule. 40 tests: a conformant run with every check passing, then one broken rule per test asserting its check flips.
commit: |
agents-wg#22 merged on 2026-10-20 as 137d88bd. The proposal file is byte-identical to head 9b16840e (blob 7534146f), so every row keeps its quote and only the citation moves.
| if (typeof card.name !== 'string' || card.name === '') { | ||
| nameErrs.push(`${label}: name is ${describeValue(card.name)}`); |
There was a problem hiding this comment.
Technically the spec doesn't say agent names need to be non-empty, though that does make sense. Probably worth correcting the extension spec for this one.
| if (got.error.message !== unknownFailure.message) { | ||
| errs.push( | ||
| `message ${JSON.stringify(got.error.message)} for the hidden agent, ${JSON.stringify(unknownFailure.message)} for an unknown name` | ||
| ); | ||
| } |
There was a problem hiding this comment.
Do we want to do an exact match on the error message here? Seems unreasonably strict to disallow e.g. agent "<requested name>" not found (might warrant a spec adjustment)
| /** Every tool `tools/list` serves, following cursors. */ | ||
| export async function toolsListAll( | ||
| conn: Connection, | ||
| maxPages = 50 | ||
| ): Promise<Record<string, unknown>[] | RpcFailure> { | ||
| const out: Record<string, unknown>[] = []; | ||
| let cursor: string | undefined; | ||
| for (let i = 0; i < maxPages; i++) { | ||
| let res: { tools?: unknown; nextCursor?: unknown }; | ||
| try { | ||
| res = await conn.request('tools/list', cursor ? { cursor } : {}); | ||
| } catch (err) { | ||
| return toRpcFailure(err); | ||
| } | ||
| for (const t of Array.isArray(res.tools) ? res.tools : []) { | ||
| if (isObject(t)) out.push(t); | ||
| } | ||
| if (typeof res.nextCursor !== 'string' || res.nextCursor === '') break; | ||
| cursor = res.nextCursor; | ||
| } | ||
| return out; | ||
| } |
There was a problem hiding this comment.
Feels like this should report tool list truncation and flag the server as untestable (a la agentsListAll) rather than silently using an incomplete tool list.
| if (!ack) { | ||
| const why = session.error | ||
| ? `${SUBSCRIPTIONS_LISTEN_METHOD} failed: ${session.error.code} ${session.error.message}` | ||
| : `${SUBSCRIPTIONS_LISTEN_METHOD} did not confirm with ${SUBSCRIPTIONS_ACKNOWLEDGED_NOTIFICATION} within ${LISTEN_ACK_MS}ms (HTTP ${session.status}, ${session.contentType ?? 'no content type'})`; | ||
| if (!declaredTrue) { | ||
| return [ | ||
| untestable(NOT_DECLARED_ID, why), |
There was a problem hiding this comment.
If a server doesn't declare listChanged and answers subscriptions/listen with -32601, or if it doesn't serve it at all, isn't it already not acknowledging the filter key? I think this should be brought in line with the precedent here which marks this case as skipped.
| const hit = await session.waitFor( | ||
| (n) => n.method === AGENTS_LIST_CHANGED_NOTIFICATION, | ||
| LIST_CHANGED_MS | ||
| ); |
There was a problem hiding this comment.
nit: this waitFor will pass on servers that happen to emit a list_changed notification immediately upon connection - I believe I've seen servers do that with tools sometimes. Might be worth having an overload that only considers notifications starting from an arbitrary execution breakpoint, or something? Might be overengineering, though.
Closes #576. agents-wg#22 merged on 2026-10-20, so this is ready for review.
What changes
Adds a requirement-traceability yaml for the agent discovery extension, agents-wg#22, and two server scenarios that grade an implementation against it, plus 40 negative controls.
sep-9998-agents-discoveryio.modelcontextprotocol/agentscapability;agents/listshape, cards and pagination; the flatagents/getresult andmodelRoutingHint; every returned tool matching itstools/listentry;agent_not_foundandinvalid_agent_definition; a hidden agent indistinguishable from an unknown onesep-9998-agents-list-changednotifications/agents/list_changedoversubscriptions/listenwithagentsListChanged, and its acknowledgementThe yaml declares 49 rows quoting the proposal at its merge commit
137d88bd, each with aurlto its section, checked by script against the pinned file. 31 are graded. The other 18 are declared and excluded, each with its reason: client and host obligations, rules that compare two authorization contexts (a run holds one credential), and step-up, a MAY that needs a credential the harness can raise.Things to look at first
modelcontextprotocol/modelcontextprotocolPR number, and the traceability tooling drops non-numeric ids. The yaml header explainssep-9998(9999 was the MCP Events placeholder before SEP-3415 existed) and the scripted rename once a number exists.list_changedrows need the server to provoke a state over the wire. The scenarios look foragents_conformance_fixtureandagents_conformance_add_agentintools/list, and report untestable without them, the same pattern as feat(sep-3415): Conformance suite for triggers and events #521.--force. The extension isn't on the spec timeline, so the runner selects these scenarios only with--force.ttlMsandcacheScopeare graded as required, because both results areCacheableResult, where the draft schema requires them.-32021row yet. Luca said on Discord that a client that didn't declare the extension should get-32021. The proposal text doesn't say so yet, so I'll add the row once that follow-up merges.Evidence
Against mcpkit's implementation (
examples/agents/discovery --conformance-agents), the result is 28/28 and 4/4. Breaking that implementation four ways, by showing the hidden agent, never sendinglist_changed, returning a wrongagent_not_foundreason, or omittingttlMs, turns the matching rows red. Running against the Python reference fork is next, once it serves the extension.npm run build,npm run lintandnpm testpass.Prerequisite knowledge
137d88bdis the text every row quotes. Retrieving Agent Details, Error Handling and Security Implications carry most of the graded rows.AGENTS.md§ Scenario design and § Check conventions explain why this is two scenarios, and why a missing control fails rather than skips.