Page header uncompressed_page_size exceeds the actual decompressed payload by 2 bytes on ZSTD + dictionary-encoded ARRAY<ARRAY<BIGINT>> columns #3712

Activity

1. weijietong

   added

   Type: bug

   on Aug 7, 2026
2. arnabnandy7 commented on Aug 13, 2026

   

   arnabnandy7

   on Aug 13, 2026

   More actions

   @weijietong the ZSTD frame contains 3125 uncompressed bytes, while the Parquet page header says 3127. The embedded RLE metadata also expects two more bytes than are present. This suggests the bytes were lost during compression or while the ZSTD frame was finalized, rather than being calculated incorrectly by the Parquet page writer.

   The file was written with zstd-jni 1.5.2-1, even though parquet-java 1.15.2 declares 1.5.6-6. Reading it with a newer zstd-jni version will not help because the two bytes were already missing when the file was created.

   Could you try writing the same data again with zstd-jni 1.5.6-6 or, preferably, 1.5.7-12? If the problem still occurs, a small input dataset or the affected page would help us reproduce it and add a regression test.

   Unfortunately, changing only uncompressed_page_size does not repair the file because the page contents are also truncated. The affected files will need to be regenerated from their source data.
3. weijietong commented on Aug 18, 2026

   

   weijietong

   on Aug 18, 2026

   Last edited by weijietong

   Author

   More actions

   @arnabnandy7 thanks for the response. I can't reproduce this bug. I can share the trimed bad page if you can find some clue from the bad page.It's hard to say the zstd-jni lost the compressed content or the parquet claimed a wrong larger length number.

   bad_page.parquet.zip

   👍React with 👍1Reacted by Arnab Nandy
4. arnabnandy7 commented on Aug 18, 2026

   

   arnabnandy7

   on Aug 18, 2026

   More actions

   @arnabnandy7 thanks for the response. I can't reproduce this bug. I can share the trimed bad page if you can find some clue from the bad page.It's hard to say the zstd-jni lost the compressed content or the parquet claimed a wrong larger length number.

   bad_page.parquet.zip

   I'll try in sometime and let you know.
5. arnabnandy7 commented on Aug 18, 2026

   

   arnabnandy7

   on Aug 18, 2026

   More actions

   @weijietong I confirmed that two bytes are missing from the final repetition-level bit-packed run. The ZSTD frame is valid and its stored CRC matches, so ZSTD compressed the page in its already-incomplete state.

   I also stress-tested the parquet-java 1.15.2 RLE encoder across 128,000 randomized inputs, including direct-buffer slab boundaries and the affected value count of 2898. In every case, the reported encoded size matched the serialized byte count, so I could not reproduce this through the normal encoder path.

   This may be a rare buffer lifecycle issue or a difference in Spark’s vendored writer path. It would be helpful to know which ByteBufferAllocator Spark used and whether the affected writer task reported retries, executor errors, or memory-pressure warnings.
6. weijietong commented on Aug 19, 2026

   

   weijietong

   on Aug 19, 2026

   Author

   More actions

   @arnabnandy7 The Paimon's vendor ParquetWriter doesn't configured a ByteBufferAllocator. So the RunLengthBitPackingHybridEncoder use the default ByteBufferAllocator from the ParquetProperties. The write path has no exception. The spark's writer task has no retries and errors, no full gc and oom.
7. edgarRd commented on Sep 18, 2026

   

   edgarRd

   on Sep 18, 2026

   Last edited by edgarRd

   More actions

   Edit (2026-09-29): corrected after further analysis. The earlier claim of a "foreign/stale tail" in the repetition levels was wrong. The damage is a single contiguous deletion from the rep-level stream, ending exactly at an internal buffer slab boundary. Also corrected: max repetition/definition levels are 5/22, not 4/20. Details below.

   We hit what looks like the same defect on an independent writer stack, with a much larger loss than 2 bytes. Adding it as a second data point.

   Environment

   • Writer: Apache Spark 3.1.1 on Java 8 (1.8.0_504), writing through Apache Iceberg's Parquet writer (an Iceberg 1.6.1-based build). Footer created_by = parquet-mr version 1.15.1 (build c7257b8faff5699e13bbc781679dc03f48c1102a).
   • Compression ZSTD, V1 data pages, PARQUET_1_0 writer version, default heap ByteBufferAllocator, page write checksums enabled.
   • zstd-jni on the executor classpath was 1.4.8-1 (Spark 3.1's pinned version; the parquet ZSTD codec ran against it).
   • Readers that fail on the file: parquet-mr 1.13.1 (could not decompress page → EOFException in DataInputStream.readFully from CodecFactory$HeapBytesDecompressor.decompress), DuckDB (ZSTD Decompression failure), Trino (out-of-bounds while decoding levels).

   The file

   • 485,647,873 bytes, 46,000 rows, 8 row groups, 4,602 leaf columns, all ZSTD. Object size on storage matches the size recorded by the writer, so nothing was truncated in transit.
   • Exactly one column chunk is bad: row group 1 of a deeply nested optional BINARY leaf under several levels of 3-level LIST groups (Edit: maxRepetitionLevel = 5, maxDefinitionLevel = 22; bit widths 3 and 5). The column is entirely null in every row group (null_count == num_values), so the page carries only repetition and definition levels: rlEnc=RLE dlEnc=RLE valEnc=PLAIN, no dictionary page, no value bytes. The same column's pages in the other 7 row groups are fine.

   The page

   Header: type=DATA_PAGE num_values=68593 compressed_page_size=21630 uncompressed_page_size=42334, CRC set.

   • CRC32 of the compressed bytes on disk == header crc, so the bytes are exactly what the writer produced.
   • The zstd payload is a single, complete, valid frame. It decompresses with no error to 40,756 bytes, i.e. 1,578 bytes short of uncompressed_page_size.
   • Inside the decompressed payload:
     • Repetition-level length prefix says 19,368 bytes.
     • The definition-level section is intact: its 4-byte prefix is at offset 17,794, length 22,958, and 17794 + 4 + 22958 == 40756. It decodes to exactly 68,593 values.
     • So only 17,790 rep-level bytes are present. 4 + 19368 + 4 + 22958 == 42334: the writer's size bookkeeping was consistent with 19,368 rep-level bytes, but only 17,790 reached the compressor.

   Edit: what actually happened to the repetition levels

   • 29 sibling columns in the same row group share the same repeated ancestry and value count. All of them have byte-identical 19,368-byte rep-level streams.

   • The bad stream is exactly one of those with one contiguous interval removed, with no fuzzy matching:

     bad_rep == healthy_rep[:14661] + healthy_rep[16239:]

     The "impossible" level values I reported earlier (5, 6, 7, 27) came from decoding a run whose middle had been cut out. All 3,129 bytes after the cut match the healthy suffix exactly.

   • Putting the 1,578 missing bytes back gives a 42,334-byte page whose decoded level histograms exactly match the page's size statistics (SizeStatistics) recorded by the writer:

     • repetition {0: 5906, 1: 46423, 2: 6675, 3: 9589}
     • definition {3: 274, 4: 30195, 6: 10268, 10: 27856}

   The cut ends exactly at a slab boundary

   Re-encoding the recovered 68,593 levels with the real RunLengthBitPackingHybridEncoder(3, 64, 1048576, HeapByteBufferAllocator) reproduces the healthy 19,368-byte stream with this slab layout in its CapacityByteArrayOutputStream:

   slab start	capacity	position
   0 … 4060	(8 slabs)	full
   8120	8119	8119
   16239	16239	3129

   The deleted interval [14661, 16239) is the tail of the slab [8120, 16239). If that slab's position() is lowered from 8,119 to 6,541 before serialization, parquet-java 1.15.1 + zstd-jni 1.4.8-1 reproduce both the decompressed payload and the 21,630-byte compressed page byte for byte. This is deliberate fault injection, not a natural reproduction.

   That fits the code: CapacityByteArrayOutputStream.size() returns the independent bytesUsed counter, which feeds the RLE length prefix and uncompressed_page_size. writeTo(), however, emits slab.position() bytes per slab, and nothing checks the two against each other.

   The slab in question was already full, so its position was 8,119 when the next slab was allocated. The next slab is complete. So the position was lowered after that slab was no longer being written to. I could not find any parquet-java code path that does this:

   • setByte writes at an absolute index and doesn't move the position.
   • writeInto flips the slab and then restores its position.
   • getInternalByteBuffer only applies when there is a single slab.
   • The heap allocator never reuses buffers.

   The Paimon page fits the same shape

   In the attached bad_page.parquet, the last bit-packed run of the rep levels has header 0x15 (10 groups = 20 bytes), but only 18 bytes follow before the def-level prefix 03 00 00 00. The sibling column's page has the same 18 bytes plus the final padded group 51 00, which RunLengthBitPackingHybridEncoder.toBytes() writes just before it sets that header. So again bytesUsed counted bytes that a slab's position() didn't cover. Here it's the final slab, the one still being written.

   What we tried

   • Fed the healthy page's rep/def payload through CodecFactory ZSTD compression (parquet-hadoop 1.15.1) as BytesInput.concat(fromInt(len), rl, fromInt(len), dl, empty()), with sizes varied around the real ones, single-threaded and with 8 concurrent compressors sharing the CodecFactory, on Java 8 and Java 17, with zstd-jni 1.4.8-1 and 1.5.7-6. Thousands of iterations, no size or content mismatch.
   • Replayed the exact recovered levels through the encoder and compressor over 120,000 iterations (stock and observing heap allocators, G1, small heap). No mismatch.

   Takeaways

   1. The failure is a slab position() lower than what was written, while bytesUsed is correct. The header and length prefix are right; the serialized payload is short.
   2. The zstd stage is very likely innocent: re-compressing the shortened input reproduces the stored page exactly. (A codec dropping the tail of a single write() call would look the same, so this isn't a strict proof.)
   3. Dictionary encoding and value bytes are not required: our column is all-null PLAIN with no value bytes.
   4. In our case the affected slab was not the one being written, which argues against a race inside the encoder. Something changed a finished slab's position afterwards.
   5. Different zstd-jni versions (1.5.2-1 vs 1.4.8-1) and writer frameworks (Paimon vs Iceberg) point below the integration layer.
8. cursor

   added a commit that references this issue on Oct 26, 2026

   apacheGH-3712: Force multi-slab layout in page-size regression tests

   ...

   fcbaec8