Repository navigation
Integrate JSONBench into vortex benchmarking - #11390
Draft
robert3005 wants to merge 22 commits into
Draft
robert3005 wants to merge 22 commits into
robert3005 wants to merge 22 commits into
Conversation
JSONBench runs ClickHouse's five queries over Bluesky events held in a single Variant column. DataFusion reads it from Parquet, Vortex, and Vortex Compact; DuckDB reads it from Parquet and its native format. - vortex-bench: download the NDJSON source files and write Parquet with a Parquet Variant column, optionally shredded (`--opt shredding=`). Add `Benchmark::queries_for_engine`, because Variant access has no SQL syntax that both engines accept. - datafusion-bench: register `variant_get(variant, path, type)`. It has the signature of datafusion-variant, which has no release for DataFusion 55 and Arrow 59. - bench-orchestrator: add the suite, and reject DuckDB over Vortex, which cannot export a Variant to DuckDB yet. Reading a Variant column from a Vortex file into Arrow did not work: - Exporting a chunked Variant recursed until the stack overflowed. - Rebuilding the shredded fields failed when the storage of a nested Variant or the shredded object was chunked. - vortex-datafusion rejected a Variant dtype against its Arrow storage struct. - The Parquet to Vortex conversion panicked for lack of a Variant builder. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
The default Vortex preset leaves an unshredded Variant value without general-purpose compression. Vortex Compact applies zstd to it, which roughly halves the file. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
An unshredded Variant value is an opaque binary blob that the lightweight schemes cannot compress, so the default preset stored it raw. The JSONBench file was 244 MB against 129 MB of zstd Parquet. `CascadingCompressor` can now hand the storage children of Variant arrays to a second compressor. The default BtrBlocks preset builds one that keeps the binary Zstd scheme. Shredded children and every other column are compressed as before. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
DataFusion evaluated `variant_get` above the scan, so a Vortex scan had to export the whole Variant, merging shredded fields back into it. - vortex-datafusion converts a typed `variant_get(variant, path, type)` over plain object fields to the Vortex `variant_get`. - datafusion-bench marks the function as cheap to move, so DataFusion hoists it into a projection that the scan absorbs. Running it inside the scan exposed two problems in `variant_get`: - Over a lazy input, such as a filtered or chunked Variant, no bulk kernel matched and every row was extracted as a scalar. Chunked inputs now extract per chunk, and other inputs canonicalize first. - A path through a partially shredded object returned nulls: the walk stopped at the nested Variant and fell back to the residual value, which does not hold the shredded fields. The nested Variant now resolves the rest of the path. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
Extracting a shredded field wrapped it in a mask of its parent's validity, even when the parent has no nulls. The mask hid the nested Variant from the kernels that extract in bulk, so every row was extracted as a scalar. A validity without nulls no longer adds a mask, and `variant_get` extracts from under a mask and reapplies it. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
- A shredded field is null in the typed column wherever an object lacks it. `variant_get` then extracted from every row of core storage to fill those rows. It now extracts from only those rows first, and returns the typed column when none of them holds a value. - Parquet Variant storage whose value is null in every row answers a typed extraction with nulls, without decoding its metadata. - The scan projection evaluates each distinct expression once. DataFusion asks for the same `variant_get` under several aliases when it hoists it out of both a filter and a group key. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
Add regression tests for extracting a nested shredded path from canonical, chunked, and filtered Variants, for converting DataFusion's `variant_get` to the Vortex one, and for evaluating duplicate scan projection expressions once. Record the measured results and the remaining costs in the suite doc. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
DataFusion plans the projection before the scan absorbs a filter, so it still asks for the expressions the filter tests. Where the filter equates one to a literal, every projected row has that value, and the scan now projects the literal. For JSONBench this drops the `variant_get` calls on `kind` and `commit.operation` from the projection, each a pass over the Variant. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
A Variant chunk is written as it is read, because there is no Variant builder to merge batches into. The Parquet reader yields 1,024 rows per batch by default, so every chunk of the Vortex file held dozens of Variants, each compressed on its own and each a subtree to deserialize. Reading a row group as one batch shrinks the JSONBench file by 5 to 10 percent and roughly halves the time of the shredded queries. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
When some rows that a shredded column leaves null did hold a value, `variant_get` extracted from every row of core storage again. The rows it already extracted are now scattered back to their positions. Update the JSONBench results and the remaining costs in the suite doc. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
- The flat layout reader decodes the storage of an unshredded Variant once per chunk when an expression extracts from it more than once. Each `variant_get` otherwise decompressed the metadata and the value again, where a Parquet reader decodes the column once. - Chunks of Parquet Variant storage execute to one storage array with chunked children, and the compressor compresses that instead of each chunk. A writer that feeds it small batches no longer stores a separately compressed Variant per batch. Shredded chunks stay apart. With these, JSONBench on the unshredded layout runs level with Parquet or faster on every query. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
Share the step that decodes the storage of an unshredded Variant before several extractions with the V2 scan's expression evaluation stage. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
The chunks of a shredded Variant's core storage are canonical Variants around the storage encoding, one level deeper than the compressor looked, so they still compressed chunk by chunk. It now unwraps them until the storage merges. Written from 1,024-row batches, the shredded JSONBench file shrinks from 149 MB to 135 MB, the size it has when written from whole row groups. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
The V2 scan evaluates each filter conjunct and the projection as their own scan of a segment, so a query that filters on three Variant paths decoded the Variant's storage four times. - The flat layout plans a Variant segment scan under a share, so a split decodes the segment once. - The decoded Variant's storage children are shared arrays, so the metadata and the value decompress once, and only if an extraction reads them. - `variant_get` canonicalizes its input one lazy layer at a time, up to a depth bound. The guard it had stopped at the first layer when layers nested, and extracted row by row from a shared array. JSONBench on the unshredded layout now runs faster than Parquet on every query with the V2 scan. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
A shredded Variant was written as one array per chunk: its metadata, its binary value, and every shredded field in a single segment. A scan that extracted one shredded field read and deserialized all of it. The Variant layout stores a shredded Variant as a struct of its core storage and its shredded child, and hands the struct to the struct layout. Each shredded field becomes a column of its own, with its own encoding and statistics. - `variant_pack(core, shredded)` builds the Variant back from the stored columns. The validity of a shredded struct is a column of its own, because a scan cannot push a field read through a nullable struct. - `variant_get` over `variant_pack` reduces to a read of the shredded column that holds the path, or of the core storage for a field that was not shredded. The writer records whether the shredded child is complete, that is whether no row left a value for it in the core storage; only then is the core storage left unread. - The layout plans as an evaluation of `variant_pack` over the struct's plan, and reads through the V1 scan by rewriting each expression over the stored struct. Fused plan expressions are optimized recursively, so that an extraction under a comparison reduces as well. - The compressor resolves core storage under a lazy array, such as a slice, as it does chunked storage. The layout is in the unstable edition. JSONBench enables it through an edition of its own. On its shredded layout, with the V2 scan, queries run at 0.08 to 0.25 of Parquet's time, from 0.21 to 0.44, and Q0 reads 0.5 MB of the 126 MB file. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
Extracting a path from unshredded Parquet Variant storage went through the `variant_get` kernel of `parquet-variant-compute`: an Arrow array of the storage, a `Variant` over every row, and a `Variant` over every object on the way down. That was half of the time of an unshredded JSONBench query, and converting the storage to Arrow another tenth. A string or a 64-bit integer under a path of object fields is now read from the encoded bytes in place. Anything else falls back to the kernel: a path through a list, another target dtype, and a value that needs a cast, so the kernel still decides how casts behave. Unshredded JSONBench queries run at 0.51 to 0.82 of Parquet's time with the V2 scan, from 0.71 to 1.06. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
The in-place extraction read every offset, field id and length by copying a variable number of bytes into a buffer. They are one byte wide far more often than not, and a read of a fixed width is a single load. Unshredded JSONBench queries run at 0.35 to 0.51 of Parquet's time with the V2 scan, from 0.51 to 0.82. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
Extracting a path from an unshredded Variant looked every field up by name in every object, a binary search that reads two offsets and compares a name at each step. The rows of a column share a handful of metadata values, which are dictionary encoded, so the names of the path are now resolved against each distinct metadata once, and an object is scanned for the field id. The extracted strings are also written to their views directly, where the builder checked the UTF-8 of each a second time. On JSONBench this takes the unshredded queries from 166, 303, 270, 223 and 228 ms to 155, 270, 246, 208 and 209 ms. Zstd decompression of the values is now about four fifths of what is left. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
A reader of an unshredded Variant decompresses and walks every value to extract one field. The Variant layout already stores a shredded Variant as columns, so the writer can now shred a Variant that arrives unshredded: VariantStrategy takes a shredder, which chooses a shredding from the first chunk of a stream and applies it to every chunk. InferredShredding in vortex-parquet-variant is such a shredder. It samples the values and shreds each object field, up to four objects deep, that holds a string, an integer, a double or a boolean in every sampled row that has it. Rows that do not fit keep their value in the residual binary, so the Variant holds what it held before. JSONBench gains the option shredding=inferred, which keeps the Parquet file unshredded and enables the shredder for the Vortex writer. Against that Parquet file the five queries take 0.04, 0.11, 0.04, 0.03 and 0.03 of the time, or 12 to 70 ms where the unshredded Vortex file takes 170 to 280 ms. The file grows from 126.5 MB to 163.2 MB with the default preset, and to 133.1 MB with the compact one. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
The suite's default dataset left the Variant unshredded, where reading is bound by zstd decompression of the values. With the Vortex writer shredding what the documents have in common, the five queries take 0.04, 0.09, 0.03, 0.03 and 0.02 of the time of the unshredded Parquet file. shredding=none still selects the unshredded dataset. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
There was no builder for Variant arrays, so a chunked Variant could never become one array: it canonicalized to a Variant over chunked core storage. Several places worked around that one at a time. VariantBuilder builds a canonical Variant. Its core storage keeps the encodings it was appended in, as other nested builders do, but a storage encoding can install a VariantStorageBuilder of its own, through which consecutive arrays of that encoding merge into one. Scalars become constant runs. The shredded children are concatenated, and rows without one are null in it, so chunks with and without a shredded child now canonicalize together instead of failing. Parquet Variant storage installs a storage builder that chunks its metadata, value and validity children without decoding them. With it: - A chunked Variant canonicalizes through the builder, and pack_variant_chunks is gone. - The execute-parent kernel that merged chunks of Parquet Variant storage is gone, and Arrow export concatenates chunks through the storage builder. - The compressor executes a Variant's core storage once instead of unwrapping up to four layers. - The benchmark conversion canonicalizes Variant columns like any other, and no longer reads each row group as one batch. - variant_get's two row-by-row fallbacks build through the builder. JSONBench results match Parquet on every query, with timings unchanged. The shredded files are 0.3 to 0.9 percent larger, since the inferred schema now comes from a 1,024-row first chunk. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SWbEtUd72h2VpGQ3DHqFC
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There's quite a lot of features and fixes in this branch to get the benchmark to work E2E. Will work through them, if anyone wants to pick up splitting changes out of this pr go ahead.