Skip to content

Fix schemas for flattened tuple-struct remainders - #39

Merged
ahl merged 7 commits into
oxidecomputer:mainfrom
sekaiser:fix/flattened-vec-remainder-schema
Nov 12, 2026
Merged

ahl merged 7 commits into
oxidecomputer:mainfrom
sekaiser:fix/flattened-vec-remainder-schema

Conversation

@sekaiser

@sekaiser sekaiser commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

A tuple struct with a flattened Vec<String> tail serializes as ["hello",42,"a","b"], but its schema expects arrays such as ["hello",42,["a"],["b"]]. It rejects valid output and accepts data that the deserializer rejects.

Draft-07 §6.4.2 applies additionalItems to each trailing element. This uses the element schema for direct standard Vec<T> remainders with conditional JsonSchema. The Rust representation and Serde behavior stay the same.

Custom containers and indirect remainders are unchanged. A vector remainder behind a type alias still gets the container schema instead of the element schema.

Note This can break consumers that depend on the old schema. Nested tails that used to validate will fail, and generated clients or workarounds may need changes.

Use the element schema for direct standard Vec remainders so draft-07
additionalItems describes each trailing value. Preserve the Rust
representation and Serde behavior. Custom containers and indirect
remainders retain their existing schema behavior.

Add serializer/schema regression checks and coverage for nested and
referenced elements, crate-path overrides, and custom-container traits.

Verified with Rust 1.97.1: 492 tests passed, 14 ignored; formatting,
Clippy and documentation checks passed.

@ahl ahl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice find (did you hit this by using it?)

I'd like to fix up all the cases, not just the Vec case.

Comment on lines +1211 to +1220
let additional_item_id = match typespace.types.get(rest_id) {
Some(Type::Vec(element_id))
if path_text(vec_type.path()) == "::std::vec::Vec"
&& vec_type.provision(TypespaceTrait::JsonSchema)
== TraitProvision::IfParameters =>
{
element_id
}
_ => rest_id,
};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need something more like array_bounds below

Custom
// containers (including transparent Vec wrappers or aliases) and
// other remainder shapes retain their existing schema behavior,
// which may still describe an extra array level.

yeah: let's fix this.

@sekaiser sekaiser Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for taking the time to review my pull request. I really appreciate your time and feedback. Although the PR ended up being much larger than I originally anticipated, I hope the contribution proves useful.

nice find (did you hit this by using it?)

No, I stumbled across Oxide’s RFDs and was inspired to try contributing; I really enjoyed your Typify2 post too.

I've followed the array_bounds approach for the other remainder types. The schema fix handles item schemas and length bounds for vectors, sets, fixed arrays, tuples, nested tuple structs and array-valued literals. It walks aliases, boxes and newtypes; custom and opaque containers supply their own schemas without adding element bounds. Nested element schemas stay unchanged, and all known positional fields remain required.

Native tests exercise transparent wrappers, aliases and nested custom containers. One container deliberately implements JsonSchema independently of its parameter; its test checks the item schema and minimum length.

A private utility shared within each output handles the runtime translation. Its helpers separate reference handling, logical branches, literals and array constraints. Most of the work comes from constraints on the entire tail: contains must find a match after the prefix, and an array-valued const or enum describes the complete tail. Logical combinations need the same translation.

Supporting fewer schema constructs would make the helper smaller. I believe we could keep the same traversal for aliases, wrappers and custom containers, and decide how to handle unsupported constraints. I'd like feedback on whether to limit the scope that way.

Sequence references must resolve to ready definitions registered with Schemars. Unsupported references cause a panic during schema generation that names the generated type. A referenced false is rejected because Schemars also uses false for definitions still being built; inline false works.

Existing json-serde adapters cannot round-trip every supported schema shape. Fixed arrays and tuples have schema-only coverage. For a cycle consisting only of wrappers, the item layout stays unknown: the walk stops with an unconstrained item schema and retains the known prefix. That test checks termination and prefix bounds.

Suffix uniqueness is left for a separate change. A tuple with a String title and flattened BTreeSet<String> tags has two independent fields written into one JSON array. Only the tags must be unique; the title may equal a tag.

JSON Expected with unique tags Current schema Whole-array uniqueItems
["Report"] Accept Accept Accept
["Report", "red", "blue"] Accept Accept Accept
["Report", "red", "red"] Reject Accept Reject
["red", "red"] Accept Accept Reject
["red", "red", "blue"] Accept Accept Reject

In both draft-07 and 2020-12, uniqueItems checks the whole array. It would reject valid serialized values where the title equals a tag. Uniqueness and dependent logical constraints remain relaxed here; a follow-up must decide how to express uniqueness within the tail or report unsupported cases.

For {"not":{"uniqueItems":true}}, a valid tail must have duplicates. Removing only uniqueItems would make the inner schema accept every tail and not reject every tail. The helper records which translations are exact, so it can omit affected negations, oneOf or conditionals and keep accepting values allowed by the original constraint.

Note Consumers relying on the old schema may need to update validators, generated clients or workarounds.

@sekaiser
sekaiser force-pushed the fix/flattened-vec-remainder-schema branch from 095c773 to a2ffe6f Compare November 9, 2026 21:01
Resolve chains of ready definitions registered with Schemars when shifting
an opaque remainder schema past a nonempty tuple prefix. Unavailable or
cyclic references now cause a schema-generation panic naming the generated
type. Reject referenced false targets because Schemars also uses false for
in-progress definitions; intentional inline false remains supported.
Leave references within item schemas unchanged.

Extend flattened remainder handling through aliases, wrappers, nested
tuples, fixed sequences and custom containers without adding element
JsonSchema bounds. Share the generated translator and split it into
documented private helpers.

Add native regression cases and generated snapshots. Keep suffix
uniqueness relaxed and leave existing json-serde behavior unchanged.

Validation: 532 workspace tests passed, plus Clippy, formatting and docs.
Assert that a flattened set schema accepts duplicate tail values while an
ordinary set schema rejects them. Document that suffix uniqueness and
logical constraints depending on it remain relaxed, with enforcement
deferred to a separate follow-up.

Validation: 42 relevant tests, targeted Clippy and formatting pass.
@sekaiser sekaiser changed the title Fix schemas for flattened standard Vec tuple remainders Fix schemas for flattened tuple-struct remainders Nov 10, 2026
@sekaiser
sekaiser requested a review from ahl November 10, 2026 11:57

@ahl ahl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The general case turned out to be more involved than anticipated. I suggest we address the initial, narrower case, and then consider the right generalized approach.

Let's go back to a2ffe6f and I'll review that more narrowly. Sorry for the churn.

Comment thread typespace/src/lib.rs
Comment on lines -2383 to -2386
// We could do something cute here, like export its schema and
// then--for tuples with flattened remainders--modify the
// schema at runtime... but that's pretty fragile since we have
// no idea of the structure of the output schema.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the approach you've taken is more or less what we had discarded as fragile.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! I got a bit carried away exploring the general case and probably should have checked in before expanding the scope. I've restored the narrower fix, and I agree that tackling this incrementally makes more sense. Thanks again for taking the time to review it!

Comment thread typespace/src/build/structs.rs Outdated
Comment on lines +1197 to +1200
// TODO 11/10/2026
// Consider deriving JsonSchema for remainder-free tuple structs where
// the derive matches the generated schema. One-field TupleStructs must
// still describe arrays, unlike the default derive for newtypes.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain this change?

@sekaiser sekaiser Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certainly! While revisiting the existing TODO about potentially using #[derive(JsonSchema)] for tuple structs without a rest field, I noticed a possible representation mismatch.

My understanding was based on the existing implementation:

  1. test_tuple_marker_struct verifies that a single-field TupleStruct serializes as [7], whereas a newtype serializes as 7.

  2. TupleStruct::render generates a custom sequence serializer regardless of whether rest is present.

  3. The custom JsonSchema implementation explicitly describes the array representation.

Furthermore, I checked Schemars 0.8.22. Its derive implementation treats a struct with one unnamed field as a newtype and generates the schema from the field's type.

This can be reproduced with a small addition inside the existing test_tuple_marker_struct test's inner() function:

#[derive(schemars::JsonSchema)]
struct DerivedListed(u32);

let schema = serde_json::to_value(
    schemars::schema_for!(DerivedListed)
).unwrap();

let value = serde_json::to_value(
    import::Listed(7)
).unwrap();

assert_eq!(schema["type"], "integer");
assert_eq!(value, serde_json::json!([7]));

assert!(
    !jsonschema::draft7::new(&schema)
        .unwrap()
        .is_valid(&value)
);

// Read the schema-only helper's field to avoid an unused-field warning.
let _ = DerivedListed(7).0;
println!("Derived schema type: {}", schema["type"]);
println!("Typespace Wrapped(7) JSON: {}", serde_json::to_value(import::Wrapped(7)).unwrap());
println!("Typespace Listed(7) JSON: {}", value);
println!("Derived schema accepts Listed(7): {}", jsonschema::draft7::new(&schema).unwrap().is_valid(&value));
    



// Rust 1.97.1, targeting aarch64-apple-darwin
// running 1 test
// Derived schema type: "integer"
// Typespace Wrapped(7) JSON: 7
// Typespace Listed(7) JSON: [7]
// Derived schema accepts Listed(7): false
// test test_tuple_marker_struct ... ok

Here, DerivedListed has the same Rust struct shape as the generated Listed, but uses Schemars' automatic derivation. The comparison illustrates why replacing the custom schema implementation with a plain #[derive(JsonSchema)] could introduce a mismatch: the derived schema expects an integer, while Typespace’s serializer produces an array.

That was my reasoning behind the additional qualification in the TODO. It was intended to document a consideration for a possible future cleanup, not to introduce a functional change.

Revert c6b6468 and 3083cd3, restoring the exact a2ffe6f tree.

Keep the direct standard-Vec correction in this PR. Leave generalized
remainder handling and opaque schema translation for separate design
discussion, as requested in review.

Validation: 492 workspace tests passed; Clippy, formatting and docs passed.

@ahl ahl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for this

Comment on lines +9 to +12
// Exercise generated paths through a facade as well as the default crate name.
pub mod schema_facade {
pub use schemars::*;
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this unrelated?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point the facade probably adds unnecessary complexity.

}
}

#[test]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want a test with no comment that asserts behavior that we know to be incorrect.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

}

#[test]
fn indirect_vec_remainders_keep_existing_schema() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as above

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we shouldn't preserve assertions about schema behavior we already know to be incorrect.

However, I think there's a separate regression worth guarding against here: custom containers may implement JsonSchema independently of their element type.

I've narrowed this test to verify only that invariant. It uses a custom OpaqueVec<T> with an unconditional JsonSchema implementation and an element type that deliberately does not implement JsonSchema.

If schema generation accidentally introduces an additional T: JsonSchema requirement, the test will fail to compile.

The revised test no longer makes any assertions about the existing additionalItems representation. I've also added a comment explaining the distinction.

Resolve changelog and import conflicts while preserving the direct standard
Vec remainder schema fix and upstream string-constrained newtype changes.

@ahl ahl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for this

@ahl
ahl merged commit 8a03a5f into oxidecomputer:main Nov 12, 2026
10 checks passed
@sekaiser sekaiser mentioned this pull request Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants