You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
I had expected rbindlist() to preserve attributes, but it doesn't. It'd be a nice feature; or an important limitation to disclose in the documentation. #5569
library(data.table)
x<- data.table(0)
setattr(x,"xa",0)
y<- data.table(1)
setattr(y,"ya",1)
z<- rbindlist(list(x,y))
attributes(z)
#> $names#> [1] "V1"#> #> $row.names#> [1] 1 2#> #> $class#> [1] "data.table" "data.frame"#> #> $.internal.selfref#> <pointer: 0x00000239858df980># I had expected attributes(z) to include at least "xa", or both "xa" and "ya".
Indeed, I'd be happy if rbindlist() was similar to either rbind() or dplyr::bind_rows() with respect to the preservation (or destruction, or copying) of table-level attributes. However I can't find any documentation of their intended behaviour! Furthermore, I wouldn't be surprised if the modify-in-place reference semantics of data.table methods will make it impossible (or undesirable) to mimic the behaviour of either of those data.frame methods.
My use-case for these methods is to add a row of experimental data to an existing table. I now realise that this is much more easily done with data.frame methods than with data.table methods:
xdf<-data.frame(data=c(0))
attr(xdf, "xa") <-0xdf<-dplyr::bind_rows(xdf, c(data=1))
# I expect attributes(xdf) to include "xa". (PASS)
attributes(xdf)
#> $names#> [1] "data"#> #> $class#> [1] "data.frame"#> #> $row.names#> [1] 1 2#> #> $xa#> [1] 0xdf<-data.frame(data=c(0))
attr(xdf, "xa") <-0xdf<- rbind(xdf, c(data=1))
# I expect attributes(xdf) to include "xa". (PASS)
attributes(xdf)
#> $names#> [1] "data"#> #> $row.names#> [1] 1 2#> #> $xa#> [1] 0#> #> $class#> [1] "data.frame"
library(data.table)
xdt<- data.table(data=0)
setattr(xdt, "xa", 0)
xdt<-dplyr::bind_rows(xdt, c(data=1))
# I expect class(xdt) to have "data.table" as its first element (PASS)
class(xdt)
#> [1] "data.table" "data.frame"# I expect xdt to have an attribute "xa" (PASS)
attributes(xdt)
#> $names#> [1] "data"#> #> $row.names#> [1] 1 2#> #> $class#> [1] "data.table" "data.frame"#> #> $.internal.selfref#> <pointer: 0x000002c36fbc4bb0>#> #> $xa#> [1] 0# I expect class(xdt) to have a valid .internal.selfref structure. (FAIL)data.table:::selfrefok(xdt)
#> [1] 0
library(data.table)
xdt<- data.table(data=0)
setattr(xdt, "xa", 0)
xdt<- rbindlist(list(xdt, data.table(data=1)))
# I expect class(xdt) to have a valid .internal.selfref structure. (PASS)data.table:::selfrefok(xdt)
#> [1] 1# I expect attributes(xdt) to include "xa". (FAIL)
attributes(xdt)
#> $names#> [1] "data"#> #> $row.names#> [1] 1 2#> #> $class#> [1] "data.table" "data.frame"#> #> $.internal.selfref#> <pointer: 0x000002c36fbc4bb0>
I'll leave it to the DT team to decide whether to continue treating this as a feature-request, or (as I'd now suggest) as a request to amend the documentation of rbindlist(). A sentence would suffice, I should think, to inform users that -- unlike DF <- rbind(DF, ...) and DF <- dplyr::bind_rows(DF, ... ) -- the new data.table DT created by DT <- rbindlist(l, ...) has none of the attributes of the tables in its argument l aside from their column names.
Your prompt response to my feature-request is very impressive, thanks! I'm extremely grateful for the bazillions of hours of volunteer labour, over the decades, which is making my current coding experience in R sososo much more pleasant and productive than my only prior experience with stochastic experimentation -- which was way back in the early 1990s, using S (yeah I'm an old codger! ;-)
Preserve attributes with rbindlist seems trickier than I initially thought, since we also store key and indices as attributes.
This is something that will definitely break a data.table when using dplyr::bind_rows together with a keyed data.table as first argument (similar to #5361)
Yeah, it'd be a big shift in semantics of rbindlist, if its first arg was redefined to be a DT that'll be modified by reference, followed by a list of DTs to be concatenated! There must be many people (and codes) who are currently using rbindlist as a UNION ALL, i.e. are relying on it not to modify its first argument.
So... maybe... it'd make sense to define a new method for DTs which (essentially) overloads append with the usual DT design pattern of modifying its first arg by reference and returning it invisibly? Maybe call it rappend? As with rbindlist I guess it'd be invalidating the keys, to avoid the performance overhead of rebuilding them.
Activity
ben-schwen commentedon Dec 20, 2022
To be fair the
baseequivalent does also not preserve all attributes.Neither does
dplyr::bind_rows()cthombor commentedon Dec 20, 2022
Indeed, I'd be happy if
rbindlist()was similar to eitherrbind()ordplyr::bind_rows()with respect to the preservation (or destruction, or copying) of table-level attributes. However I can't find any documentation of their intended behaviour! Furthermore, I wouldn't be surprised if the modify-in-place reference semantics ofdata.tablemethods will make it impossible (or undesirable) to mimic the behaviour of either of thosedata.framemethods.My use-case for these methods is to add a row of experimental data to an existing table. I now realise that this is much more easily done with
data.framemethods than withdata.tablemethods:I'll leave it to the DT team to decide whether to continue treating this as a feature-request, or (as I'd now suggest) as a request to amend the documentation of rbindlist(). A sentence would suffice, I should think, to inform users that -- unlike
DF <- rbind(DF, ...)andDF <- dplyr::bind_rows(DF, ... )-- the newdata.tableDT created byDT <- rbindlist(l, ...)has none of the attributes of the tables in its argumentlaside from their column names.Your prompt response to my feature-request is very impressive, thanks! I'm extremely grateful for the bazillions of hours of volunteer labour, over the decades, which is making my current coding experience in R so so so much more pleasant and productive than my only prior experience with stochastic experimentation -- which was way back in the early 1990s, using S (yeah I'm an old codger! ;-)
ben-schwen commentedon Dec 21, 2022
Preserve attributes with
rbindlistseems trickier than I initially thought, since we also storekeyandindicesas attributes.This is something that will definitely break a
data.tablewhen usingdplyr::bind_rowstogether with a keyeddata.tableas first argument (similar to #5361)cthombor commentedon Dec 21, 2022
Yeah, it'd be a big shift in semantics of
rbindlist, if its first arg was redefined to be a DT that'll be modified by reference, followed by a list of DTs to be concatenated! There must be many people (and codes) who are currently usingrbindlistas a UNION ALL, i.e. are relying on it not to modify its first argument.So... maybe... it'd make sense to define a new method for DTs which (essentially) overloads
appendwith the usual DT design pattern of modifying its first arg by reference and returning it invisibly? Maybe call itrappend? As withrbindlistI guess it'd be invalidating the keys, to avoid the performance overhead of rebuilding them.