Repository navigation
ParquetFileReader throws a runtime exception when a file contains only headers and now row data #2728
Description
Google BigQuery has an option to export table data to Parquet-formatted files, but some of these files are written with header data only. When this happens and these files are opened with the ParquetFileReader, an exception is thrown:
RuntimeException("Illegal row group of 0 rows");
It seems like the ParquetFileReader should not throw an exception when it encounters such a file.
Reporter: chris stockton
Assignee: Gang Wu / @wgtmac
Note: This issue was originally created as PARQUET-2219. Please see the migration documentation for further details.
Activity
asfimport commentedon Dec 16, 2022
Gang Wu / @wgtmac:
According to the error message, it seems that empty row group is deemed as malformed. CMIW, some downstream readers may use the parquet reader as an iterator to check if it can continue reading any row. However, the empty row group may terminate the iterator early which is an incorrect behavior. So I suggest not writing the parquet file if there is no row of data.
asfimport commentedon Dec 16, 2022
Gang Wu / @wgtmac:
cc @emkornfield
asfimport commentedon Jan 6, 2023
Micah Kornfield / @emkornfield:
I'm not aware of anything in the specification that prevents zero length row groups. We can try to prevent writing them out but I think readers should be robust to this if it isn't disallowed in the specification. For the iterator case, it seems like the rowgroup should just be discarded and the next one checked?