You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CI: conformance-weekly failing: unexpected failure auth/resource-mismatch (not in baseline) #648
This isn't a regression in the SDK. The scenario's verdict changed upstream.
auth/resource-mismatch passes when the client reads the mismatched resource from the PRM and then doesn't go on to authorize. Our client has no OAuth yet: it gets the 401 on initialize and stops without ever fetching the PRM. Up to 0.2.0-alpha.11 that counted as a pass. modelcontextprotocol/conformance#483 changed it to depend on the PRM fetch, and without it the check now reports "Not testable" as a FAILURE (resource-mismatch.ts:100-111). That change shipped in 0.2.0-alpha.12 on 2026-10-01, so 10-05 was the first weekly run to pick it up.
So the old pass was a false pass, and the baseline is the right fix. The line can't go in on its own, though. The PR pipeline pins alpha.11, where the scenario still passes, and the runner fails a baseline entry that passes as stale (expected-failures.ts:203-207). It needs the pin bump in the same change:
.github/workflows/pipeline.yaml: CONFORMANCE_PKG to @modelcontextprotocol/conformance@0.2.0-alpha.12
Makefile: CONFORMANCE_VERSION ?= 0.2.0-alpha.12
tests/Conformance/conformance-baseline-2026-07-28.yml: - auth/resource-mismatch under client:
What I ran locally on main at 5d311d6, PHP 8.3, with tests/Conformance/client.php:
auth/resource-mismatch, 2026-07-28
alpha.11
alpha.12
current baseline
PASS
FAIL: "client never requested the Protected Resource Metadata document"
baseline + the new line
exit 1, stale entry
exit 0
On alpha.12 with the new baseline, the full client suite exits 0 for both spec versions: 2026-07-28 has 63 passed and 55 expected failures, and 2025-11-25 has 15 passed and 49 expected failures. The server side I can't vouch for. My local server ran under php -S instead of the nginx/php-fpm fixture, and that setup already fails three 2025-11-25 scenarios on alpha.11 that CI passes. The weekly job does show server 2026-07-28 green on alpha.12 for four weeks. After a bump, server 2025-11-25 would be the one job running alpha.12 for the first time.
I can't push workflow changes from this account, so this is a description rather than a PR. Once feat-client-auth lands, the scenario should run for real and the line can come out again.
I'll take this on, building on @aton-of-data's diagnosis. I'll verify the version change and cover the server checks with nginx/PHP-FPM for both protocol revisions, then send the pin and baseline update together.
Activity
aton-of-data commentedon Nov 2, 2026
This isn't a regression in the SDK. The scenario's verdict changed upstream.
auth/resource-mismatchpasses when the client reads the mismatchedresourcefrom the PRM and then doesn't go on to authorize. Our client has no OAuth yet: it gets the 401 oninitializeand stops without ever fetching the PRM. Up to0.2.0-alpha.11that counted as a pass. modelcontextprotocol/conformance#483 changed it to depend on the PRM fetch, and without it the check now reports "Not testable" as a FAILURE (resource-mismatch.ts:100-111). That change shipped in0.2.0-alpha.12on 2026-10-01, so 10-05 was the first weekly run to pick it up.So the old pass was a false pass, and the baseline is the right fix. The line can't go in on its own, though. The PR pipeline pins
alpha.11, where the scenario still passes, and the runner fails a baseline entry that passes as stale (expected-failures.ts:203-207). It needs the pin bump in the same change:.github/workflows/pipeline.yaml:CONFORMANCE_PKGto@modelcontextprotocol/conformance@0.2.0-alpha.12Makefile:CONFORMANCE_VERSION ?= 0.2.0-alpha.12tests/Conformance/conformance-baseline-2026-07-28.yml:- auth/resource-mismatchunderclient:What I ran locally on main at 5d311d6, PHP 8.3, with
tests/Conformance/client.php:auth/resource-mismatch, 2026-07-28On
alpha.12with the new baseline, the full client suite exits 0 for both spec versions: 2026-07-28 has 63 passed and 55 expected failures, and 2025-11-25 has 15 passed and 49 expected failures. The server side I can't vouch for. My local server ran underphp -Sinstead of the nginx/php-fpm fixture, and that setup already fails three 2025-11-25 scenarios onalpha.11that CI passes. The weekly job does show server 2026-07-28 green onalpha.12for four weeks. After a bump, server 2025-11-25 would be the one job runningalpha.12for the first time.I can't push workflow changes from this account, so this is a description rather than a PR. Once
feat-client-authlands, the scenario should run for real and the line can come out again.euvoor commentedon Nov 4, 2026
I'll take this on, building on @aton-of-data's diagnosis. I'll verify the version change and cover the server checks with nginx/PHP-FPM for both protocol revisions, then send the pin and baseline update together.