Skip to content

Return 400 instead of crashing on JSON with an unknown charset - #1121

Closed
hemtilva wants to merge 1 commit into
marshmallow-code:devfrom
hemtilva:fix-unknown-json-charset
Closed

hemtilva wants to merge 1 commit into
marshmallow-code:devfrom
hemtilva:fix-unknown-json-charset

Conversation

@hemtilva

Copy link
Copy Markdown

When a JSON request's Content-Type names a codec Python doesn't know (for example application/json; charset=not-a-charset), PyramidParser and AIOHTTPParser let a LookupError escape, and the server returns a 500. This PR makes them return the documented 400 {"json": ["Invalid JSON body."]} instead, the same way undecodable bodies are already handled since #427.

Fixes #1120

Changes

  • webargs.core.parse_json: if bytes.decode(encoding) raises LookupError (unknown codec), raise json.JSONDecodeError instead. UnicodeDecodeError is already converted the same way. Parser.load_json then sends it to _handle_invalid_json_error. This fixes PyramidParser, which passes req.charset as encoding.
  • AIOHTTPParser.load_json: catch the LookupError that aiohttp's Request.text() raises for an unknown charset and pass it to _handle_invalid_json_error. The annotation of _handle_invalid_json_error is widened to match.
  • Tests:
    • tests/test_core.py::test_parse_json_with_unknown_encoding_raises_json_decode_error
    • test_parse_json_with_unknown_charset in tests/test_pyramidparser.py and tests/test_aiohttpparser.py
    • All three fail on dev and pass with this change.
  • CHANGELOG entry under 9.0.0 "Bug fixes".

Flask, Django, Bottle and Falcon ignore the charset parameter and decode as UTF-8, so they are unaffected. This PR does not change their behaviour.

Testing

  • pytest: 506 passed, 4 skipped (dev baseline: 503 passed, 4 skipped)
  • ruff check / ruff format --check: clean
  • mypy src/webargs: no new errors (the 2 existing unreachable errors in tornadoparser.py are also on dev)

PyramidParser passes the request charset to core.parse_json and aiohttp's
Request.json() decodes with the request charset. When the Content-Type
header names a codec Python does not know (e.g.
'application/json; charset=not-a-charset'), bytes.decode() raises
LookupError, which was not handled and surfaced as a 500.

parse_json now converts LookupError into JSONDecodeError (as it already
does for UnicodeDecodeError), and AIOHTTPParser.load_json routes
LookupError to _handle_invalid_json_error, so both frameworks respond with
the documented 400 'Invalid JSON body.' error.
Copilot AI balanced review requested due to automatic review settings November 15, 2026 22:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@sirosen

sirosen commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Blanket catching LookupError like this is not a valid solution.

Please read our AI Policy. Sending us LLM outputs without any investment of real effort is actively harmful to the project maintainers and therefore harmful to the project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unknown charset in a JSON request's Content-Type crashes PyramidParser and AIOHTTPParser (500 instead of 400)

3 participants