Skip to content

Preserve existing findings when scans are incomplete or skipped #304

Open

Description

@helen

Parent: github/accessibility-scanner#300

Goal

Do not close an accessibility issue as resolved merely because the corresponding check timed out, failed, or was not run. Continue resolving genuine fixes from successfully completed checks.

Current behavior

updateFilingsWithNewFindings clears cached findings before repopulating them from the current scan. isResolvedFiling treats a cached filing with no findings as resolved. Neither decision currently proves that the relevant URL/scanner check completed.

This becomes especially important when adding timeout recovery: continuing with a partial findings file must not create false resolutions.

Scope

  • Consume the explicit per-URL/per-scanner completion contract from the reporting task; do not infer completion from an empty findings array.
  • Reconcile successful checks normally while retaining cached findings for timed-out, failed, skipped, not-started, or unknown coverage.
  • Handle disabled scan types and URLs not scanned in the current run conservatively. Any intentional retirement of historical findings must be explicit, not an accidental consequence of missing coverage.
  • Preserve findings at the correct granularity for every supported grouping mode. For an issue spanning multiple URLs, update successfully checked portions without deleting unverified portions or closing the whole issue prematurely.
  • Carry retained findings through the saved cache so later successful runs still reconcile with the existing issue rather than losing its identity or creating duplicates.
  • Document compatibility with older cache/result formats; missing completion metadata must not silently authorize closing unchecked findings.

Acceptance criteria

  • Tests cover timeout, error, disabled scanner, omitted URL, absent/unknown coverage, and mixed successful/incomplete runs.
  • Existing open findings for unchecked URL/scanner pairs remain tracked; issues whose relevant checks completed successfully and no longer reproduce still resolve.
  • Grouped filings retain unverified findings while refreshing completed portions across supported grouping modes.
  • A later successful scan of previously incomplete work updates or resolves the original filing without duplication.
  • Existing manual-closure/wontfix behavior is not unintentionally changed, and a partial run does not reopen a deliberately closed issue merely because its cached finding was retained.

Dependency and boundary

This task depends on the completion-reporting contract in github/accessibility-scanner#301, and can be implemented/tested against fixtures before runtime timeout recovery in github/accessibility-scanner#302 is ready. It owns filing/cache reconciliation, not timeout enforcement or browser recovery. Safe reconciliation must be connected before partially completed recovered runs are allowed to perform ordinary issue resolution.

🤖 Assisted by Copilot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions