You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Overbroad issue about .. being defined in calling scope? #2988
idx = 5
DT = data.table(
ID = rep(1:3,each = 4),
flag = as.logical(rep(0:1, 6)),
N = 1:12,
count = 12:1
)
DT[ , dcast(.SD, ID ~ flag + N, value.var = 'count')
][ , {
idx = grep('^TRUE', names(.SD))
y1 = .SD[ , ..idx]
y2 = .SD[ , -..idx]
NULL
}]
Warning messages:
1: In [.data.table(.SD, , ..idx) :
Both 'idx' and '..idx' exist in calling scope. Please remove the '..idx' variable in calling scope for clarity.
2: In [.data.table(.SD, , -..idx) :
Both 'idx' and '..idx' exist in calling scope. Please remove the '..idx' variable in calling scope for clarity.
Yes, idx and ..idx exist in calling scope, but ..idx appears "down a level" (reverting to the analogy of .. as "up a level"), so the code as written seems pretty natural to me.
I get where the message comes from, and I'm not sure how much effort it's worth to fully account for this type of possible nesting, but wanted to raise attention anyway.
To get rid of this warning, I'm resorting to with = FALSE, so this is related to #2826
This becomes an error if there's no idx in globalenv:
Error in [.data.table(DT[, dcast(.SD, ID ~ flag + N, value.var = "count")], :
Variable 'idx' is not found in calling scope. Looking in calling scope because this symbol was prefixed with .. in the j= parameter.
Reacted by Frank, Matt Dowle, Arvind and Vassilis Kehayas
It is useful when the variable names are duplicate with data.table col names.
For example, after we fread a file, the col names or number could be not fixed and changeable,
with .. function, we do not have to worry about it may be same with a col name when we create a new variable.
for example.
library(data.table)
a<- data.table(x=1:2,y=2:1,z=c("x","y"))
x<-"y"# works in `i`a[get(..("x")) ==1] # a subset when `y==1`a[..x==1] # Error in eval(.massage i(isub), x, parent.frame()) : object '..x' not founda[get(..x)==1] # Error in get(..x) : object '..x' not found# works in `j`a[, get(..("x"))] # a vector of the col `y` a[, ..x] # a data.table with col `y`
Although we need use the additional get to obtain a col of data.table, it has advantages when we want to use it alone:
a[z==..("x")] # a subset data.table when `z=="y"`a[, z==..("x")] # a logical vector for `z=="y"`
Back to the question of @MichaelChirico, if we replace .. prefix with the .. function, the conflict will be eliminated.
I also found that the .. function can not be use in the LHS of := assignment, but we can use the get , for example.
a[, ..("x"):=9] # Error in get(x, envir = parent.env(env)) : object 'x' not founda[, get("x"):=9] # the `y` will be replaced with 9
And it does not work in by , for example,
a[, .N, by=..("x")]# Error
Last year, I put this solution in SO, The old version of .. function is:
The old version worked fine in data.table 1.10.4, but when I updated to 1.11.4, it failed in i, the I wrote the new one, which is correct in both versions. However, I am not a skilled R programer, I am not sure this approach is a reasonable way for such jobs.
In my opinion, maybe the .. prefix is too complicated, however, .. function explicitly tell R that it is a function calling but not a data.table col , which also allows the col names of data.table to start with "..".
Could we have an official support for .. function @mattdowle, @arunsrinivasan , and support it in i, j, and by?
..() function is very interesting idea, thanks.
The reason to choose .. prefix was imagine a situation where an expression is passed to ..(some expression) and that can make the internal output of all.vars() harder to deal with, and harder to know how to optimize. Whenever get is used (or something like it, like ..()) we have to turn off optimizations because get might be getting something that varies for each group. In contrast, a .. prefix on a variable means that it's for sure a symbol (baked into the syntax) which seems a bit more robust to me, and we know it's a constant. Although, yes, harder to implement. And the current situation where .. prefix is only available in j is of course confusing.
It will be difficult to put the toothpaste back in the tube on column names starting ... Since .. prefix on symbols seems to be liked and now used in the wild. We just need to implement .. prefix everywhere basically so it works for all the cases you showed.
We could have ..() too as well. Especially since it solves Michael's issue here. I haven't grasped that yet.
Maybe .. prefix needs a fix for compound queries.
I stumbled over the same warning message as Michael. I doubt my example contains any additional information to this issue, but since it was so simple I thought I just as well could add it here for illustration.
I wanted to update certain rows in certain columns with values from other rows in the same columns. Stripped down version:
d <- data.table(x = 1:3, y = 4:6, z = c("a", "b", "c"))
cols <- c("x", "z")
d[2, (cols) := d[3, ..cols]]
Warning message:
In [.data.table(d, 3, ..cols) :
Both 'cols' and '..cols' exist in calling scope. Please remove the '..cols' variable in calling scope for clarity.
Any progress on this issue? I also get the warning Both 'colname' and '..colname' exist in calling scope. Please remove the '..colname' variable in calling scope for clarity.
While I cannot see any use of colname anywhere else. ``
I will add my two cents having also run into these issues...
@liqg, .. as function makes a lot of sense to me and I would use it in i if available, and it additionally seems to solve the issue astutely raised by @MichaelChirico and others here.
However, it seems to me that .. and ..() existing is confusing. They would be used in non-overlapping places to non-overlapping effect. Given that .. is liked and used, it does seem that the path forward is implementing .. elsewhere vs. adding ..(). I think additional complexity to variable scoping in data.table would be a burden.
Thanks, do you mean something like this? dt[, sum(var), env = list(var = "mpg")]
I may have the variable defined somewhere already, for example I have a function to process a column in dt
convert_col<-function(dt, col_name) {
# how should I write this?dt[, col_name:=xxxx, env=list(col_name=col_name)]
}
Allweneedisjusttorecognizethatsomevariablesarefromoutsideofdtscope...
If you are passing character scalar into col_name, then yes, it will work exactly like this. Of course in this example you could easily wrap variable into ( and it would work as well.
Activity
jangorecki commentedon Aug 5, 2018
MichaelChirico commentedon Aug 15, 2018
liqg commentedon Sep 14, 2018
I wrote a
..function as an alternative to..prefix that can/may explicitly get variable outside the data.table scope.It is useful when the variable names are duplicate with data.table col names.
For example, after we
freada file, the col names or number could be not fixed and changeable,with
..function, we do not have to worry about it may be same with a col name when we create a new variable.for example.
Although we need use the additional
getto obtain a col of data.table, it has advantages when we want to use it alone:Back to the question of @MichaelChirico, if we replace
..prefix with the..function, the conflict will be eliminated.I also found that the
..function can not be use in the LHS of:=assignment, but we can use theget, for example.And it does not work in
by, for example,Last year, I put this solution in SO, The old version of
..function is:The old version worked fine in data.table 1.10.4, but when I updated to 1.11.4, it failed in
i, the I wrote the new one, which is correct in both versions. However, I am not a skilled R programer, I am not sure this approach is a reasonable way for such jobs.In my opinion, maybe the
..prefix is too complicated, however,..function explicitly tell R that it is a function calling but not a data.table col , which also allows the col names of data.table to start with "..".Could we have an official support for
..function @mattdowle, @arunsrinivasan , and support it ini,j, andby?Thanks!
mattdowle commentedon Sep 22, 2018
..()function is very interesting idea, thanks.The reason to choose
..prefix was imagine a situation where an expression is passed to..(some expression)and that can make the internal output ofall.vars()harder to deal with, and harder to know how to optimize. Whenevergetis used (or something like it, like ..()) we have to turn off optimizations becausegetmight be getting something that varies for each group. In contrast, a..prefix on a variable means that it's for sure a symbol (baked into the syntax) which seems a bit more robust to me, and we know it's a constant. Although, yes, harder to implement. And the current situation where..prefix is only available injis of course confusing.It will be difficult to put the toothpaste back in the tube on column names starting
... Since..prefix on symbols seems to be liked and now used in the wild. We just need to implement..prefix everywhere basically so it works for all the cases you showed.We could have
..()too as well. Especially since it solves Michael's issue here. I haven't grasped that yet.Maybe
..prefix needs a fix for compound queries.Henrik-P commentedon Oct 26, 2018
I stumbled over the same warning message as Michael. I doubt my example contains any additional information to this issue, but since it was so simple I thought I just as well could add it here for illustration.
I wanted to update certain rows in certain columns with values from other rows in the same columns. Stripped down version:
franknarf1 commentedon Nov 6, 2018
SO post to update (exactly the same as Henrik's above): https://stackoverflow.com/a/53180736
Another: https://stackoverflow.com/q/53227216
sanjmeh commentedon Mar 26, 2019
Any progress on this issue? I also get the warning
Both 'colname' and '..colname' exist in calling scope. Please remove the '..colname' variable in calling scope for clarity.While I cannot see any use of colname anywhere else. ``
andrewrech commentedon Mar 27, 2019
I will add my two cents having also run into these issues...
@liqg,
..as function makes a lot of sense to me and I would use it in i if available, and it additionally seems to solve the issue astutely raised by @MichaelChirico and others here.However, it seems to me that
..and..()existing is confusing. They would be used in non-overlapping places to non-overlapping effect. Given that..is liked and used, it does seem that the path forward is implementing..elsewhere vs. adding..(). I think additional complexity to variable scoping indata.tablewould be a burden.jangorecki commentedon Sep 2, 2019
related: https://stackoverflow.com/questions/57737697/how-to-use-data-table-a-reactive-expression-in-shiny
trying to use
..not on a symbol but on a call$(input, var)6 remaining items
dracodoc commentedon Apr 2, 2020
Thanks, do you mean something like this?
dt[, sum(var), env = list(var = "mpg")]I may have the variable defined somewhere already, for example I have a function to process a column in dt
jangorecki commentedon Apr 2, 2020
If you are passing character scalar into
col_name, then yes, it will work exactly like this. Of course in this example you could easily wrap variable into(and it would work as well.