Skip to content

trims zsv related metadata from column names - #7963

Open
jangorecki wants to merge 3 commits into
masterfrom
zsv
Open

jangorecki wants to merge 3 commits into
masterfrom
zsv

Conversation

@jangorecki

@jangorecki jangorecki commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Background: it is possible to push limits of fread when reading from pre-indexed csv.

Reproducible example is taken from: Filter big csv early by partitions in zsv/README.md

stopifnot(nchar(system("zsv --version", intern=TRUE)) > 8L) # ensure zsv installed

library(data.table)
make_data = function(n = 1e6, years) {
  fwrite(data.frame(
    id = sort(sample(1.05*n, n)), # 5pct missing
    year = sort(sample(years, n, TRUE)),
    amount = sample(25, n, TRUE)
  ), "data.csv")
  # zsv is csv rows + zsv format header row.
  system("zsv data.csv --detect-types --index=id --partition-by=year > data.zsv")
  invisible(TRUE)
}
n = 1e9 # requires more than 25GiB, 17G csv on disk
make_data(n, years = 2020:2026)
system("head -3 data.csv")
system("head -3 data.zsv")

system.time(
  a1 <- fread("data.csv")[year %in% 2021:2023]
)
#   user  system elapsed
# 26.022   7.229   6.111
system.time(
  a2 <- fread("zsv data.zsv --partition=2021..2023")
)
#   user  system elapsed
#  8.887   3.503   3.306
stopifnot(all.equal(a1, a2))

make_data(n, years = 1970:2026)
system.time(
  a1 <- fread("data.csv")[year %in% 2021:2023]
)
#   user  system elapsed
# 27.542   5.777   5.597
system.time(
  a2 <- fread("zsv data.zsv --partition=2021..2023")
)
#   user  system elapsed
#  1.111   0.489   0.475
stopifnot(all.equal(a1, a2))

zsv internally is still csv, just longer column names.


This PR trims zsv related metadata from column names when reading such csv directly via fread("tiny.zsv"):

cat > tiny.csv << EOF
id,year,city,amount
1,2024,Bern,2.5
2,2024,Basel,
4,2025,Zug,1.5
5,2026,Bern,3
EOF
cat > tiny.zsv << EOF
id~u8-index~0~zsv,year~u16-partition{2024@0|2025@30|2026@45},city~str,amount~f64
1,2024,Bern,2.5
2,2024,Basel,
4,2025,Zug,1.5
5,2026,Bern,3
EOF
library(data.table)

# master
fread("tiny.csv")[1]
#      id  year   city amount
#1:     1  2024   Bern    2.5
fread("tiny.zsv")[1]
#   id~u8-index~0~zsv year~u16-partition{2024@0|2025@30|2026@45} city~str amount~f64
#1:                 1                                       2024     Bern        2.5

# branch
fread("tiny.csv")[1]
#      id  year   city amount
#1:     1  2024   Bern    2.5
fread("tiny.zsv")[1] # verbose: "Detected zsv header format: trim zsv to csv"
#      id  year   city amount
#1:     1  2024   Bern    2.5

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
  • HEAD=zsv much slower for Date-IDate regression in #7213

(Plot image upload to picrd.com failed; download the artifact below to view plots)

Generated via commit 8f8078b

Download link for the artifact containing the test results: ↓ atime-results.zip

Task Duration
R setup and installing dependencies 7 minutes and 34 seconds
Installing different package versions 26 seconds
Running and plotting the test cases 5 minutes and 46 seconds

@tdhock tdhock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is interesting! R code looks ok to me. add NEWS item?

Comment thread inst/tests/tests.Rraw Resolved
@jangorecki

Copy link
Copy Markdown
Member Author

NEWS entry is there

@jangorecki
jangorecki requested a review from tdhock November 21, 2026 12:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants