Repository navigation
[ANE-Bot] Strip leading UTF-8 BOM before parsing JSON manifests - #1748
Conversation
JSON manifests written by Windows tooling (e.g. Visual Studio) often start with a UTF-8 byte order mark, which Aeson rejects, failing analysis of otherwise-valid files (seen as 'Unexpected "\239\187\191...' parse errors on NuGet project.json). RFC 8259 permits ignoring a leading BOM, so strip it before decoding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138826PipdEg1cEcc1sGYaW
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
Included review availability: Your plan includes up to 4 reviews per rolling hour; 2 remain after this review. Walkthrough
Merge Risk: ⚪ Minimal · up to This localized parsing fix strips a leading UTF-8 BOM from JSON manifests without changing previously successful parses. No actionable merge-blocking risk remains after normal checks and review. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The utf8-bom.json fixture's leading bytes are invisible in a diff and in most editors. If an editor, a text=auto gitattributes rule, or a "normalize line endings" commit ever strips them, the existing BOM test would keep passing while testing nothing. Verified that is not hypothetical: with the BOM removed from the fixture, "parses a JSON file with a leading UTF-8 byte order mark" still passes and only this new assertion fails.
Overview
FOSSA CLI fails to parse JSON manifest files that begin with a UTF-8 byte order mark (
0xEF 0xBB 0xBF).readContentsJsoninEffect.ReadFSpasses raw file bytes directly to Aeson, and Aeson rejects a leading BOM, so analysis of an otherwise-valid manifest fails with an error like:Windows tooling (notably Visual Studio) writes UTF-8 files with a BOM by default, so this hits real, valid NuGet
project.jsonmanifests — this exact signature accounts for ~300 occurrences over the last 7 days in the CLI error telemetry (version-filtered to v3.17.x/v3.16.x), all from Windows paths. RFC 8259 §8.1 permits JSON parsers to ignore a leading BOM.The fix strips a leading UTF-8 BOM from file contents in
readContentsJsonbefore decoding, so every JSON manifest read through it (NuGetproject.json,project.assets.json,package.json,composer.lock, etc.) tolerates BOM'd input. One helper (stripUtf8Bom) plus a one-line change toreadContentsJson.This is an internal parsing fix: no user-visible schema, CLI flag, or documented behavior change, so the docs/schema checklist items don't apply.
Changelog.mdis updated.This is distinct from #1747 (missing
dependencieskey inproject.json) — same file in telemetry, different root cause and different code path.Acceptance criteria
fossa analyzeon a project containing a JSON manifest with a leading UTF-8 BOM (e.g. aproject.jsonsaved by Visual Studio on Windows) no longer fails withparsing file: ... Unexpected "\239\187\191..."; the manifest parses and its dependencies are reported normally.Testing plan
cabal build(ormake build-cli).project.jsonwith a leading BOM, e.g.printf '\xEF\xBB\xBF{\r\n "dependencies": { "Newtonsoft.Json": "13.0.1" }\r\n}\r\n' > repro/project.jsonfossa analyze repro --output. Before this change the projectjson target fails withUnexpected "\239\187\191{...; after it, the target analyzes and reportsNewtonsoft.Json.cabal test unit-tests— the newEffect.ReadFSSpeccoversreadContentsJsonwith and without a BOM (the BOM fixture reproduces the exact observed input shape: BOM + CRLF). The BOM case was verified to fail against the unfixed code.Risks
Low. A leading UTF-8 BOM is never valid JSON syntax, so stripping it cannot change the result of any parse that previously succeeded. The change is scoped to
readContentsJsononly; other readers (readContentsParser, YAML, TOML, XML) are untouched.Metrics
The Error patterns widget on the fossa-cli-dashboard tracks this signature (
Unexpected "\239\187\191...parse failures, ~300/week); it should drop off after this releases.References
Checklist
docs/.docs/README.msand gave consideration to how discoverable or not my documentation is.Changelog.md. If this PR did not mark a release, I added my changes into an## Unreleasedsection at the top..fossa.ymlorfossa-deps.{json.yml}, I updateddocs/references/files/*.schema.jsonAND I have updated example files used byfossa initcommand. You may also need to update these if you have added/removed new dependency type (e.g.pip) or analysis target type (e.g.poetry).docs/references/subcommands/<subcommand>.md.This PR description was generated with Claude Code
Generated by Claude Code