Skip to content

feat: add coerce_primitive in json - #26561

Open
ion-elgreco wants to merge 1 commit into
apache:mainfrom
ion-elgreco:feat/json-coerce-primitive
Open

ion-elgreco wants to merge 1 commit into
apache:mainfrom
ion-elgreco:feat/json-coerce-primitive

Conversation

@ion-elgreco

@ion-elgreco ion-elgreco commented Oct 21, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

When a JSON file has a number or a boolean in a column that the schema declares as a string, the scan fails with an error such as expected string got false. arrow-json can keep these values as text with ReaderBuilder::with_coerce_primitive, but JsonSource does not expose that option. We now have to write a custom JSON source.

What changes are included in this PR?

  • Add JsonSource::with_coerce_primitive and the getter JsonSource::coerce_primitive. The default is false, so current behavior does not change.
  • Add JsonOpener::with_coerce_primitive. The opener passes the value to ReaderBuilder::with_coerce_primitive on every read path: byte range, local file, and stream, for newline-delimited JSON and JSON arrays.
  • Add optional bool coerce_primitive = 3 to the JsonScanExecNode protobuf message

What is the testing strategy for this PR?

  • test_coerce_primitive in datafusion/datasource-json/src/source.rs covers each read path
  • json_exec_coerce_primitive in datafusion/core/src/datasource/physical_plan/json.rs e2e test through DataSourceExec.
  • roundtrip_json_scan_preserves_format_options in datafusion/proto/tests/cases/plans/sources.rs now checks that the option survives a protobuf round trip

Are there any user-facing changes?

Yes. New public API:

  • JsonSource::with_coerce_primitive
  • JsonSource::coerce_primitive
  • JsonOpener::with_coerce_primitive
  • JsonScanExecNode::coerce_primitive (new optional protobuf field)

The default behavior does not change. Code that builds or destructures JsonScanExecNode with all fields named must add the new field.

@github-actions github-actions Bot added core Core DataFusion crate proto Related to proto crate datasource Changes to the datasource crate labels Oct 21, 2026
@ion-elgreco
ion-elgreco force-pushed the feat/json-coerce-primitive branch from fc6e6e4 to 2ded6ca Compare October 21, 2026 12:27
@ion-elgreco
ion-elgreco force-pushed the feat/json-coerce-primitive branch from 2ded6ca to 67d16ec Compare October 24, 2026 09:58
@ion-elgreco
ion-elgreco force-pushed the feat/json-coerce-primitive branch from 67d16ec to 503e0c3 Compare November 10, 2026 23:17
@kosiew

kosiew commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

🚀
@ion-elgreco
Thank you for your contribution.

Can you amend the PR description as per .github/pull_request_template.md?

@ion-elgreco
ion-elgreco force-pushed the feat/json-coerce-primitive branch from 503e0c3 to 1513988 Compare November 12, 2026 09:57
@ion-elgreco

Copy link
Copy Markdown
Author

@kosiew done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate datasource Changes to the datasource crate proto Related to proto crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow coerce_primitives in JsonSource

2 participants