StackOverflowError when ParquetFileReader skips many consecutive row groups #3889

Component(s): Core

Description

A Spark read failed with StackOverflowError in org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup() while reading a Parquet file containing many consecutive empty row groups. The failure investigation recorded 3,263 row groups, 3,199 of them empty, with a longest consecutive run of 2,500 empty groups. The stack trace repeatedly contained ParquetFileReader.readNextRowGroup.

The expected behavior is to skip empty row groups and return the next non-empty group, or null at EOF, without growing the call stack with the number of skipped groups.

Cause and affected paths

In readNextRowGroup, an empty group raises ParquetEmptyBlockException. The catch block advances to the next group and recursively calls the same method:

} catch (ParquetEmptyBlockException e) {
  LOG.warn("Read empty block at index {} from {}", currentBlock, getFile());
  advanceToNextBlock();
  return readNextRowGroup();
}

Each consecutive empty group adds another stack frame. Java does not eliminate these recursive calls, so a sufficiently long run can exhaust the thread stack. The failure threshold depends on the JVM and thread stack size.

readNextFilteredRowGroup also calls itself recursively when a group is empty or its column-index-filtered row ranges contain no matching rows. The latter path also grows the call stack with each skipped group, even when the groups are non-empty.

The recursive empty-group handling is present in the inspected 1.13.1, 1.15.2, 1.16.0, 1.17.1, and 1.18.1 tags and the master revision linked above.

Suggested fix

Replace recursive skipping with iteration in both sequential read methods, preserving group advancement, dictionary-reader updates, and buffer ownership. Regression cases should cover many consecutive empty groups before a non-empty group, all-empty groups, trailing empty groups, and many consecutive groups with no matching column-index-filtered rows.

This follows #2728 (originally PARQUET-2219) and #1018, which added empty-group skipping. The discussion in #2728 notes that the specification does not prohibit zero-length row groups. The existing TestParquetReaderEmptyBlock covers up to two consecutive empty groups.

I used an AI agent to help analyze the source and draft this issue. I reviewed the final text.