Skip to content

CI: conformance-weekly failing: unexpected failure auth/resource-mismatch (not in baseline) #648

Closed

Description

@fitzyracing1

What fails

Workflow .github/workflows/conformance-weekly.yaml fails on main in job conformance / client (2026-07-28, alpha).

Error

  ~ auth/token-endpoint-auth-none
  ~ auth/pre-registration
  ~ auth/offline-access-scope
  ~ auth/offline-access-not-supported
  ~ auth/authorization-server-migration:sep-2352-reregister-on-as-change
  ~ auth/iss-supported
  ~ auth/iss-not-advertised
  ~ auth/iss-supported-missing
  ~ auth/iss-wrong-issuer
  ~ auth/iss-unexpected
  ~ auth/iss-normalized
  ~ auth/metadata-issuer-mismatch
Unexpected failures (not in baseline):
  ✗ auth/resource-mismatch
Unexpected failures detected: these are not in your expected-failures baseline.
##[error]Process completed with exit code 1.

When it started

Likely cause

New conformance scenario auth/resource-mismatch fails.

Suggested fix

Implement RFC 9728 resource check or add to the expected-failures baseline.

Activity

  1. aton-of-data commented on Nov 2, 2026

    @aton-of-data
    Contributor

    This isn't a regression in the SDK. The scenario's verdict changed upstream.

    auth/resource-mismatch passes when the client reads the mismatched resource from the PRM and then doesn't go on to authorize. Our client has no OAuth yet: it gets the 401 on initialize and stops without ever fetching the PRM. Up to 0.2.0-alpha.11 that counted as a pass. modelcontextprotocol/conformance#483 changed it to depend on the PRM fetch, and without it the check now reports "Not testable" as a FAILURE (resource-mismatch.ts:100-111). That change shipped in 0.2.0-alpha.12 on 2026-10-01, so 10-05 was the first weekly run to pick it up.

    So the old pass was a false pass, and the baseline is the right fix. The line can't go in on its own, though. The PR pipeline pins alpha.11, where the scenario still passes, and the runner fails a baseline entry that passes as stale (expected-failures.ts:203-207). It needs the pin bump in the same change:

    • .github/workflows/pipeline.yaml: CONFORMANCE_PKG to @modelcontextprotocol/conformance@0.2.0-alpha.12
    • Makefile: CONFORMANCE_VERSION ?= 0.2.0-alpha.12
    • tests/Conformance/conformance-baseline-2026-07-28.yml: - auth/resource-mismatch under client:

    What I ran locally on main at 5d311d6, PHP 8.3, with tests/Conformance/client.php:

    auth/resource-mismatch, 2026-07-28 alpha.11 alpha.12
    current baseline PASS FAIL: "client never requested the Protected Resource Metadata document"
    baseline + the new line exit 1, stale entry exit 0

    On alpha.12 with the new baseline, the full client suite exits 0 for both spec versions: 2026-07-28 has 63 passed and 55 expected failures, and 2025-11-25 has 15 passed and 49 expected failures. The server side I can't vouch for. My local server ran under php -S instead of the nginx/php-fpm fixture, and that setup already fails three 2025-11-25 scenarios on alpha.11 that CI passes. The weekly job does show server 2026-07-28 green on alpha.12 for four weeks. After a bump, server 2025-11-25 would be the one job running alpha.12 for the first time.

    I can't push workflow changes from this account, so this is a description rather than a PR. Once feat-client-auth lands, the scenario should run for real and the line can come out again.

  2. euvoor commented on Nov 4, 2026

    @euvoor
    Contributor

    I'll take this on, building on @aton-of-data's diagnosis. I'll verify the version change and cover the server checks with nginx/PHP-FPM for both protocol revisions, then send the pin and baseline update together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions