Skip to content

[ANE-Bot] Strip leading UTF-8 BOM before parsing JSON manifests - #1748

Merged
tjugdev merged 2 commits into
masterfrom
fossa-cli-fix-2026-08-17-2
Aug 18, 2026
Merged

tjugdev merged 2 commits into
masterfrom
fossa-cli-fix-2026-08-17-2

Conversation

@fossa-ane-bot

Copy link
Copy Markdown
Contributor

Overview

FOSSA CLI fails to parse JSON manifest files that begin with a UTF-8 byte order mark (0xEF 0xBB 0xBF). readContentsJson in Effect.ReadFS passes raw file bytes directly to Aeson, and Aeson rejects a leading BOM, so analysis of an otherwise-valid manifest fails with an error like:

Error: parsing file: C:\...\project.json
Unexpected "\239\187\191{\r ...

Windows tooling (notably Visual Studio) writes UTF-8 files with a BOM by default, so this hits real, valid NuGet project.json manifests — this exact signature accounts for ~300 occurrences over the last 7 days in the CLI error telemetry (version-filtered to v3.17.x/v3.16.x), all from Windows paths. RFC 8259 §8.1 permits JSON parsers to ignore a leading BOM.

The fix strips a leading UTF-8 BOM from file contents in readContentsJson before decoding, so every JSON manifest read through it (NuGet project.json, project.assets.json, package.json, composer.lock, etc.) tolerates BOM'd input. One helper (stripUtf8Bom) plus a one-line change to readContentsJson.

This is an internal parsing fix: no user-visible schema, CLI flag, or documented behavior change, so the docs/schema checklist items don't apply. Changelog.md is updated.

This is distinct from #1747 (missing dependencies key in project.json) — same file in telemetry, different root cause and different code path.

Acceptance criteria

fossa analyze on a project containing a JSON manifest with a leading UTF-8 BOM (e.g. a project.json saved by Visual Studio on Windows) no longer fails with parsing file: ... Unexpected "\239\187\191..."; the manifest parses and its dependencies are reported normally.

Testing plan

  1. cabal build (or make build-cli).
  2. Create a minimal reproduction: a directory containing a project.json with a leading BOM, e.g.
    printf '\xEF\xBB\xBF{\r\n "dependencies": { "Newtonsoft.Json": "13.0.1" }\r\n}\r\n' > repro/project.json
  3. Run fossa analyze repro --output. Before this change the projectjson target fails with Unexpected "\239\187\191{...; after it, the target analyzes and reports Newtonsoft.Json.
  4. cabal test unit-tests — the new Effect.ReadFSSpec covers readContentsJson with and without a BOM (the BOM fixture reproduces the exact observed input shape: BOM + CRLF). The BOM case was verified to fail against the unfixed code.

Risks

Low. A leading UTF-8 BOM is never valid JSON syntax, so stripping it cannot change the result of any parse that previously succeeded. The change is scoped to readContentsJson only; other readers (readContentsParser, YAML, TOML, XML) are untouched.

Metrics

The Error patterns widget on the fossa-cli-dashboard tracks this signature (Unexpected "\239\187\191... parse failures, ~300/week); it should drop off after this releases.

References

Checklist

  • I added tests for this PR's change (or explained in the PR description why tests don't make sense).
  • If this PR introduced a user-visible change, I added documentation into docs/.
  • If this PR added docs, I added links as appropriate to the user manual's ToC in docs/README.ms and gave consideration to how discoverable or not my documentation is.
  • If this change is externally visible, I updated Changelog.md. If this PR did not mark a release, I added my changes into an ## Unreleased section at the top.
  • If I made changes to .fossa.yml or fossa-deps.{json.yml}, I updated docs/references/files/*.schema.json AND I have updated example files used by fossa init command. You may also need to update these if you have added/removed new dependency type (e.g. pip) or analysis target type (e.g. poetry).
  • If I made changes to a subcommand's options, I updated docs/references/subcommands/<subcommand>.md.

This PR description was generated with Claude Code


Generated by Claude Code

JSON manifests written by Windows tooling (e.g. Visual Studio) often start
with a UTF-8 byte order mark, which Aeson rejects, failing analysis of
otherwise-valid files (seen as 'Unexpected "\239\187\191...' parse errors
on NuGet project.json). RFC 8259 permits ignoring a leading BOM, so strip
it before decoding.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138826PipdEg1cEcc1sGYaW
@GauravB159
GauravB159 marked this pull request as ready for review August 18, 2026 00:04
@GauravB159
GauravB159 requested a review from a team as a code owner August 18, 2026 00:04
@GauravB159
GauravB159 requested a review from tjugdev August 18, 2026 00:04
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 956fc00a-1eb3-490b-847e-6b156c2173bb

📥 Commits

Reviewing files that changed from the base of the PR and between 1928041 and ff3f3a4.

📒 Files selected for processing (6)
  • Changelog.md
  • spectrometer.cabal
  • src/Effect/ReadFS.hs
  • test/Effect/ReadFSSpec.hs
  • test/Effect/testdata/no-bom.json
  • test/Effect/testdata/utf8-bom.json

Included review availability: Your plan includes up to 4 reviews per rolling hour; 2 remain after this review.


Walkthrough

readContentsJson now strips a leading UTF-8 BOM before decoding JSON. New tests compare standard and BOM-prefixed manifests using matching fixtures. The test module is registered in spectrometer.cabal. The changelog documents the new parsing behavior.

Merge Risk: ⚪ Minimal · up to ff3f3

This localized parsing fix strips a leading UTF-8 BOM from JSON manifests without changing previously successful parses. No actionable merge-blocking risk remains after normal checks and review.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely identifies the main change: stripping a leading UTF-8 BOM before parsing JSON manifests.
Description check ✅ Passed The description includes all required sections, explains the change and impact, provides repeatable tests, documents risks, references, metrics, and checklist status.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

The utf8-bom.json fixture's leading bytes are invisible in a diff and in
most editors. If an editor, a text=auto gitattributes rule, or a "normalize
line endings" commit ever strips them, the existing BOM test would keep
passing while testing nothing.

Verified that is not hypothetical: with the BOM removed from the fixture,
"parses a JSON file with a leading UTF-8 byte order mark" still passes and
only this new assertion fails.
@tjugdev
tjugdev merged commit 65142c7 into master Aug 18, 2026
19 checks passed
@tjugdev
tjugdev deleted the fossa-cli-fix-2026-08-17-2 branch August 18, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants