Skip to content

Overbroad issue about .. being defined in calling scope? #2988

Open

Description

@MichaelChirico

Just encountered this...

idx = 5
DT = data.table(
  ID = rep(1:3,each = 4),
  flag = as.logical(rep(0:1, 6)),
  N = 1:12,
  count = 12:1
)

DT[ , dcast(.SD, ID ~ flag + N, value.var = 'count')
    ][ , {
      idx = grep('^TRUE', names(.SD))
      y1 = .SD[ , ..idx]
      y2 = .SD[ , -..idx]
      NULL
    }]

Warning messages:
1: In [.data.table(.SD, , ..idx) :
Both 'idx' and '..idx' exist in calling scope. Please remove the '..idx' variable in calling scope for clarity.
2: In [.data.table(.SD, , -..idx) :
Both 'idx' and '..idx' exist in calling scope. Please remove the '..idx' variable in calling scope for clarity.

Yes, idx and ..idx exist in calling scope, but ..idx appears "down a level" (reverting to the analogy of .. as "up a level"), so the code as written seems pretty natural to me.

I get where the message comes from, and I'm not sure how much effort it's worth to fully account for this type of possible nesting, but wanted to raise attention anyway.

To get rid of this warning, I'm resorting to with = FALSE, so this is related to #2826

This becomes an error if there's no idx in globalenv:

rm(idx)
DT[ , dcast(.SD, ID ~ flag + N, value.var = 'count')
    ][ , {
      idx = grep('^TRUE', names(.SD))
      y1 = .SD[ , ..idx]
      y2 = .SD[ , -..idx]
      NULL
    }]

Error in [.data.table(DT[, dcast(.SD, ID ~ flag + N, value.var = "count")], :
Variable 'idx' is not found in calling scope. Looking in calling scope because this symbol was prefixed with .. in the j= parameter.

Activity

  1. jangorecki commented on Aug 5, 2018

    @jangorecki
  2. MichaelChirico commented on Aug 15, 2018

    @MichaelChirico
    Author
  3. liqg commented on Sep 14, 2018

    @liqg

    I wrote a .. function as an alternative to .. prefix that can/may explicitly get variable outside the data.table scope.

    .. <- function (x, env = parent.frame()) 
    {
        stopifnot(inherits(x, "character"))
        stopifnot(length(x) == 1)
        get(x, envir = parent.env(env))
    }

    It is useful when the variable names are duplicate with data.table col names.
    For example, after we fread a file, the col names or number could be not fixed and changeable,
    with .. function, we do not have to worry about it may be same with a col name when we create a new variable.

    for example.

    library(data.table)
    a <- data.table(x=1:2,y=2:1,z=c("x","y"))
    x <- "y"
    # works in `i`
    a[get(..("x")) == 1] #  a subset when `y==1`
    a[..x==1] # Error in eval(.massage i(isub), x, parent.frame()) :   object '..x' not found
    a[get(..x)==1] # Error in get(..x) : object '..x' not found
    
    # works in `j`
    a[, get(..("x"))] # a vector of the col `y` 
    a[, ..x] # a data.table with col `y`

    Although we need use the additional get to obtain a col of data.table, it has advantages when we want to use it alone:

    a[z==..("x")] # a subset data.table when `z=="y"`
    a[, z==..("x")] # a logical vector for `z=="y"`

    Back to the question of @MichaelChirico, if we replace .. prefix with the .. function, the conflict will be eliminated.

    I also found that the .. function can not be use in the LHS of := assignment, but we can use the get , for example.

    a[, ..("x"):=9] # Error in get(x, envir = parent.env(env)) : object 'x' not found
    a[, get("x"):=9] # the `y` will be replaced with 9

    And it does not work in by , for example,

    a[, .N, by=..("x")]# Error

    Last year, I put this solution in SO, The old version of .. function is:

    `..old` <- function(x){
        stopifnot(inherits(x, "character"))
        stopifnot(length(x)==1)
        get(x, parent.frame(4))
    }

    The old version worked fine in data.table 1.10.4, but when I updated to 1.11.4, it failed in i, the I wrote the new one, which is correct in both versions. However, I am not a skilled R programer, I am not sure this approach is a reasonable way for such jobs.

    In my opinion, maybe the .. prefix is too complicated, however, .. function explicitly tell R that it is a function calling but not a data.table col , which also allows the col names of data.table to start with "..".

    Could we have an official support for .. function @mattdowle, @arunsrinivasan , and support it in i, j, and by?

    Thanks!

  4. mattdowle commented on Sep 22, 2018

    @mattdowle
    Member

    ..() function is very interesting idea, thanks.
    The reason to choose .. prefix was imagine a situation where an expression is passed to ..(some expression) and that can make the internal output of all.vars() harder to deal with, and harder to know how to optimize. Whenever get is used (or something like it, like ..()) we have to turn off optimizations because get might be getting something that varies for each group. In contrast, a .. prefix on a variable means that it's for sure a symbol (baked into the syntax) which seems a bit more robust to me, and we know it's a constant. Although, yes, harder to implement. And the current situation where .. prefix is only available in j is of course confusing.
    It will be difficult to put the toothpaste back in the tube on column names starting ... Since .. prefix on symbols seems to be liked and now used in the wild. We just need to implement .. prefix everywhere basically so it works for all the cases you showed.
    We could have ..() too as well. Especially since it solves Michael's issue here. I haven't grasped that yet.
    Maybe .. prefix needs a fix for compound queries.

  5. added this to the 1.12.0 milestone on Sep 22, 2018
  6. Henrik-P commented on Oct 26, 2018

    @Henrik-P

    I stumbled over the same warning message as Michael. I doubt my example contains any additional information to this issue, but since it was so simple I thought I just as well could add it here for illustration.

    I wanted to update certain rows in certain columns with values from other rows in the same columns. Stripped down version:

    d <- data.table(x = 1:3, y = 4:6, z = c("a", "b", "c"))
    cols <- c("x", "z")
    d[2, (cols) := d[3, ..cols]] 
    

    Warning message:
    In [.data.table(d, 3, ..cols) :
    Both 'cols' and '..cols' exist in calling scope. Please remove the '..cols' variable in calling scope for clarity.

  7. franknarf1 commented on Nov 6, 2018

    @franknarf1
    Contributor

    SO post to update (exactly the same as Henrik's above): https://stackoverflow.com/a/53180736

    Another: https://stackoverflow.com/q/53227216

  8. modified the milestones: 1.12.0, 1.12.2 on Jan 6, 2019
  9. modified the milestones: 1.12.2, 1.12.4 on Jan 24, 2019
  10. sanjmeh commented on Mar 26, 2019

    @sanjmeh

    Any progress on this issue? I also get the warning Both 'colname' and '..colname' exist in calling scope. Please remove the '..colname' variable in calling scope for clarity.
    While I cannot see any use of colname anywhere else. ``

  11. andrewrech commented on Mar 27, 2019

    @andrewrech

    I will add my two cents having also run into these issues...

    @liqg, .. as function makes a lot of sense to me and I would use it in i if available, and it additionally seems to solve the issue astutely raised by @MichaelChirico and others here.

    However, it seems to me that .. and ..() existing is confusing. They would be used in non-overlapping places to non-overlapping effect. Given that .. is liked and used, it does seem that the path forward is implementing .. elsewhere vs. adding ..(). I think additional complexity to variable scoping in data.table would be a burden.

  12. jangorecki commented on Sep 2, 2019

    @jangorecki
    Member

    related: https://stackoverflow.com/questions/57737697/how-to-use-data-table-a-reactive-expression-in-shiny
    trying to use .. not on a symbol but on a call $(input, var)

    DT[, ..input$var]
  13. removed this from the 1.12.4 milestone on Sep 17, 2019
  14. 6 remaining items

  15. dracodoc commented on Apr 2, 2020

    @dracodoc
    Contributor

    Thanks, do you mean something like this?
    dt[, sum(var), env = list(var = "mpg")]

    I may have the variable defined somewhere already, for example I have a function to process a column in dt

    convert_col <- function(dt, col_name) {
      # how should I write this?
      dt[, col_name := xxxx, env = list(col_name = col_name)]
    }
    
    All we need is just to recognize that some variables are from outside of dt scope...
  16. jangorecki commented on Apr 2, 2020

    @jangorecki
    Member

    If you are passing character scalar into col_name, then yes, it will work exactly like this. Of course in this example you could easily wrap variable into ( and it would work as well.

  17. modified the milestones: 1.13.1, 1.13.3 on Oct 17, 2020
  18. modified the milestones: 1.14.3, on Jul 19, 2022
  19. modified the milestones: , 1.15.1 on Oct 29, 2023
  20. modified the milestones: 1.16.0, 1.17.0 on Jul 10, 2024
  21. modified the milestones: 1.17.0, 1.18.0 on Dec 9, 2024
  22. modified the milestones: 1.18.0, 1.19.0 on Nov 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    programmingparameterizing queries: get, mget, eval, env

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions