Repository navigation
Conversation
Add ContentStatsUtil for the v4 content_stats struct (spec section "Content Stats"): - stats ID math: base-id = 10_000 + 200 * field-id, the reserved 9000/9200 bases for _last_updated_sequence_number and _row_id, and the reverse mapping from a stats ID to its field ID and offset; - per-field stats structs with lower/upper bounds (geo_lower and geo_upper for geometry and geography), value_count, null_value_count, nan_value_count and total_bytes; - StatsWriteType from a schema and MetricsConfig, and StatsReadType for a set of field IDs. Fields in lists or maps have no stats. As the spec says, null_value_count is written for optional fields only. The read type also projects it for required fields, because Java writes it for every field (apache/iceberg#17413).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1127. Part of #1107.
Adds
ContentStatsUtil(src/iceberg/stats/content_stats_util.{h,cc}) for the v4content_statsstruct, following the spec's Content Stats section (apache/iceberg#14234;total_bytesfrom apache/iceberg#18308).BaseId,FieldIdandStatOffsetmap between table field IDs and stats IDs (10_000 + 200 * field-id, and 9000/9200 for_last_updated_sequence_number/_row_id).StatsWriteType(schema, MetricsConfig)builds the struct a writer puts in manifests: bounds fortruncate/full(geo_lower/geo_upperstructs for geo types), the counts,nan_value_countfor float/double,total_bytesfor variable-length types, and no stats for fields in lists or maps.StatsReadType(schema, field_ids)builds the projection a reader asks for.This ports Java's
StatsUtilwith one difference. The spec includesnull_value_countfor optional fields only, but Java writes it for every field since apache/iceberg#17413, and a required field inside an optional struct can still be null. The write type follows the spec, and the read type also projects the count for required fields, so stats written by Java are read.Testing
New
content_stats_util_test.cc(13 tests, inschema_test):BaseId,FieldIdandStatOffsetround-trip every offset for several field IDs, including the last storable ID (999_949), plus the reserved_last_updated_sequence_number/_row_idranges, the unassigned 9_400-9_999 (NotSupported) and out-of-range IDs (InvalidArgument).id/dataand geometrylocationexamples from the spec, compared field by field (IDs, names, types, docs), withgeo_lower/geo_upperfor geometry and geography.none/counts/truncate/fulland per-column overrides throughMetricsConfig._-joined names), fields in lists and maps skipped, field IDs too large to store.full, deduplicated and sorted, unknown IDs ignored, covers the write type; projected withProject()against Java-shaped stats (requirednull_value_countread) and spec-shaped stats (read as null).cmake --build, the full ctest suite,pre-commitand clang-tidy 22.1.8 pass locally.