You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Invalid .internal.selfref detected when a data.table is set as an attribute with setattr #3959
If I have an object (data.table or not) and I add a data.table attribute to it via attr<-, the address of the OBJECT changes, but the address of the ATTRIBUTE does not. I can later extract the data.table attribute, and modifications to it will also modify the attribute.
But if I add a data.table attribute to it via setattr, the address of the OBJECT does not change, whereas the address of the ATTRIBUTE does. If I later extract the attribute, it will be de-referenced:
x<-'x'dt<- data.table(a=1)
tracemem(dt)
#> [1] "<0x46d3dc0>"
tracemem(x)
#> [1] "<0x3207768>"
setattr(x, 'dt', dt)
#> tracemem[0x46d3dc0 -> 0x432dfc0]: setattrdt2<- attr(x, 'dt')
dt2[, b:=1]
#> Warning in `[.data.table`(dt2, , `:=`(b, 1)): Invalid .internal.selfref#> detected and fixed by taking a (shallow) copy of the data.table so#> that := can add this new column by reference. At an earlier point, this#> data.table has been copied by R (or was created manually using structure()#> or similar). Avoid names<- and attr<- which in R currently (and oddly)#> may copy the whole data.table. Use set* syntax instead to avoid copying: ?#> set, ?setnames and ?setattr. If this message doesn't help, please report#> your use case to the data.table issue tracker so the root cause can be#> fixed or this message improved.dt2[]
#> a b#> 1: 1 1
attr(x, 'dt')
#> a#> 1: 1
Note that in this second example I used setattr, not attr<-, and yet seemingly R copied the data.table somewhere. The message didn't help, so here I am.
sessionInfo:
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS
Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] data.table_1.12.2
loaded via a namespace (and not attached):
[1] compiler_3.5.2 tools_3.5.2 yaml_2.2.0
Note that in this second example I used setattr, not attr<-, and yet seemingly R copied the data.table somewhere. The message didn't help, so here I am.
It doesn't address the issue, but to clarify...
I believe that data (columns) have not been copied, but only list (data.table) been copied. That is fortunately a very cheap copy, and happens every time when selfref is invalid (after using names<- or attr<-).
> dataset[,unique_row_id := c(1:nrow(dataset))]
Warning message:
In `[.data.table`(dataset, , `:=`(unique_row_id, c(1:nrow(dataset)))) :
Invalid .internal.selfref detected and fixed by taking a (shallow) copy of the data.table so that := can add this new column by reference. At an earlier point, this data.table has been copied by R (or was created manually using structure() or similar). Avoid names<- and attr<- which in R currently (and oddly) may copy the whole data.table. Use set* syntax instead to avoid copying: ?set, ?setnames and ?setattr. If this message doesn't help, please report your use case to the data.table issue tracker so the root cause can be fixed or this message improved.
The step in my code that triggers this is the following:
Where csgo_live_round_winner is an xgboost model object and model_version is a string.
If I move dataset[,unique_row_id := c(1:nrow(dataset))] somewhere above the predict call in my script, i dont get the warning message, so I did that to avoid any potential problems. The order isnt really important.
If I change my code to use data.table syntax in that spot, like such:
I was pretty confused by the warning message, so I followed the instructions and posted here :). Please note im only an intermediate R user and relatively new to data.table
Hi @JB304245 for this error, reproducible examples are essential for debugging, so anything you could do to make an MRE of your issue would go a long way to it getting solved. Thanks!
Here is a MRE. It seems related to [[<-.data.table
library(data.table)
dt= data.table(x=1)
dt[["x"]] =2Ldt[, y:=3L]
#> Warning in `[.data.table`(dt, , `:=`(y, 3L)): Invalid .internal.selfref detected#> and fixed by taking a (shallow) copy of the data.table so that := can add this#> new column by reference. At an earlier point, this data.table has been copied#> by R (or was created manually using structure() or similar). Avoid names<- and#> attr<- which in R currently (and oddly) may copy the whole data.table. Use set*#> syntax instead to avoid copying: ?set, ?setnames and ?setattr. If this message#> doesn't help, please report your use case to the data.table issue tracker so the#> root cause can be fixed or this message improved.
It seems like the warning message should either add that dt$ <- can cause the issue. Or perhaps we could just remove the warning. I tried names(dt) = ... but I could not trigger the warning.
i.e. a copy is silently made by [[<- and the allocation gets out of sync. setDT after [[<- also fixes it.
In any case I don't think that's related to the current issue & AFAIK is known behavior. @JB304245 if the warning message was confusing for you, could you please open a separate issue for that? It's an intimidating message, probably it's worth some effort to making that message more accessible.
@MichaelChirico I agree that this sidebar regarding [[<- isn't about directly about this issue. At the same time, what is the resolution to this issue? Addressing #1218 and improving the warning message?
Activity
tdeenes commentedon Oct 10, 2019
I think this relates to #1218 (an issue I opened in 2015...). The point is that
setattrmakes a copy of ANY object which is given as the attribute.d-sci commentedon Oct 11, 2019
Indeed the same issue. Interesting to know that
bit::setattrhas a different (no-copy) behaviour.jangorecki commentedon May 27, 2020
It doesn't address the issue, but to clarify...
I believe that data (columns) have not been copied, but only list (data.table) been copied. That is fortunately a very cheap copy, and happens every time when selfref is invalid (after using names<- or attr<-).
JB304245 commentedon Sep 16, 2020
Hi,
I got the following:
The step in my code that triggers this is the following:
Where
csgo_live_round_winneris an xgboost model object andmodel_versionis a string.If I move
dataset[,unique_row_id := c(1:nrow(dataset))]somewhere above the predict call in my script, i dont get the warning message, so I did that to avoid any potential problems. The order isnt really important.If I change my code to use data.table syntax in that spot, like such:
There is also no warning message.
I was pretty confused by the warning message, so I followed the instructions and posted here :). Please note im only an intermediate R user and relatively new to data.table
MichaelChirico commentedon Sep 16, 2020
Hi @JB304245 for this error, reproducible examples are essential for debugging, so anything you could do to make an MRE of your issue would go a long way to it getting solved. Thanks!
ColeMiller1 commentedon Sep 16, 2020
Here is a MRE. It seems related to
[[<-.data.tableIt seems like the warning message should either add that
dt$ <-can cause the issue. Or perhaps we could just remove the warning. I triednames(dt) = ...but I could not trigger the warning.MichaelChirico commentedon Sep 20, 2020
Agreed about changing the warning. Not sure we can drop about
names(keep in mind it would have to be true for R 3.1.0 as well)...The error is indeed because of
[[<-:i.e. a copy is silently made by
[[<-and the allocation gets out of sync.setDTafter[[<-also fixes it.In any case I don't think that's related to the current issue & AFAIK is known behavior. @JB304245 if the warning message was confusing for you, could you please open a separate issue for that? It's an intimidating message, probably it's worth some effort to making that message more accessible.
ColeMiller1 commentedon Sep 23, 2020
@MichaelChirico I agree that this sidebar regarding
[[<-isn't about directly about this issue. At the same time, what is the resolution to this issue? Addressing #1218 and improving the warning message?