Skip to content

dt[, (cols) := list(...), by = group] should not silently recycles list #4022

Open

Description

@renkun-ken

Currently, dt[, (cols) := list(...), by = group] seems to silently recycles list(...) when replacing values of cols. If length(list) < length(cols), then list is recyled; if length(list) > length(cols) then redundant elements in list are silently dropped, as demonstrated below:

When by = group is absent, the lengths are checked:

library(data.table)
dt <- data.table(id = 1:10)
xn <- 1:3
xcols <- paste0("x", xn)
dt[, (xcols) := list(10, 20)]
#> Error in `[.data.table`(dt, , `:=`((xcols), list(10, 20))): Supplied 3 columns to be assigned 2 items. Please see NEWS for v1.12.2.

However, if by = group is used, list is recycled:

library(data.table)
dt <- data.table(id = 1:10)
dt[, group := sample(1:2, .N, replace = TRUE)]
xn <- 1:3
xcols <- paste0("x", xn)
dt[, (xcols) := list(10, 20), by = group]
dt
#>     id group x1 x2 x3
#>  1:  1     2 10 20 10
#>  2:  2     2 10 20 10
#>  3:  3     2 10 20 10
#>  4:  4     2 10 20 10
#>  5:  5     2 10 20 10
#>  6:  6     2 10 20 10
#>  7:  7     1 10 20 10
#>  8:  8     2 10 20 10
#>  9:  9     2 10 20 10
#> 10: 10     2 10 20 10
library(data.table)
dt <- data.table(id = 1:10)
dt[, group := sample(1:2, .N, replace = TRUE)]
xn <- 1:3
xcols <- paste0("x", xn)
dt[, (xcols) := list(40, 30, 20, 10), by = group]
dt
#>     id group x1 x2 x3
#>  1:  1     1 40 30 20
#>  2:  2     1 40 30 20
#>  3:  3     2 40 30 20
#>  4:  4     2 40 30 20
#>  5:  5     2 40 30 20
#>  6:  6     1 40 30 20
#>  7:  7     1 40 30 20
#>  8:  8     2 40 30 20
#>  9:  9     2 40 30 20
#> 10: 10     1 40 30 20

Personally, the recycling behavior is almost always unwanted. If it occurs, it is mostly something wrong with my code.

Consider the following example where list(...) is produced by lapply(.SD, ...). If the function is inlined and a bit complicated, one often forgets to write .SDcols.

library(data.table)
set.seed(123)
dt <- data.table(id = 1:10)
dt[, group := sample(1:2, .N, replace = TRUE)]
xn <- 1:3
xcols <- paste0("x", xn)
for (i in xn) {
  dt[, xcols[[i]] := runif(.N)]
}
dt[, (xcols) := lapply(.SD, function(x) {
  x / sd(x)
}), by = group]
dt
#>     id group        x1        x2        x3
#>  1:  1     1 0.2672612 2.7645427 3.2041655
#>  2:  2     1 0.5345225 1.3098014 2.4955128
#>  3:  3     1 0.8017837 1.9576795 2.3071378
#>  4:  4     2 2.3421602 1.5214175 4.9351189
#>  5:  5     1 1.3363062 0.2973764 2.3618854
#>  6:  6     2 3.5132403 2.3907258 3.5168344
#>  7:  7     2 4.0987803 0.6538253 2.7005050
#>  8:  8     2 4.6843204 0.1117471 2.9490603
#>  9:  9     1 2.4053512 0.9474491 1.0415680
#> 10: 10     1 2.6726124 2.7578116 0.5299108

Undesired/incorrect results are silently produced. The following are the correct results with .SDcols added.

library(data.table)
set.seed(123)
dt <- data.table(id = 1:10)
dt[, group := sample(1:2, .N, replace = TRUE)]
xn <- 1:3
xcols <- paste0("x", xn)
for (i in xn) {
  dt[, xcols[[i]] := runif(.N)]
}
dt[, (xcols) := lapply(.SD, function(x) {
  x / sd(x)
}), by = group, .SDcols = xcols]
dt
#>     id group        x1        x2        x3
#>  1:  1     1 2.7645427 3.2041655 2.5018371
#>  2:  2     1 1.3098014 2.4955128 2.3440794
#>  3:  3     1 1.9576795 2.3071378 1.7943807
#>  4:  4     2 1.5214175 4.9351189 2.9399476
#>  5:  5     1 0.2973764 2.3618854 0.0639438
#>  6:  6     2 2.3907258 3.5168344 1.7658739
#>  7:  7     2 0.6538253 2.7005050 2.8031711
#>  8:  8     2 0.1117471 2.9490603 0.7998165
#>  9:  9     1 0.9474491 1.0415680 0.8266013
#> 10: 10     1 2.7578116 0.5299108 0.6017398

I suggest that list(...) recycling should be consistent with the behavior data.table has already adopted with row recycling: only accepting zero, one, or .N elements.

Activity

  1. added this to the 1.12.7 milestone on Nov 2, 2019
  2. Henrik-P commented on Nov 2, 2019

    @Henrik-P
  3. modified the milestones: 1.12.7, 1.12.9 on Dec 8, 2019
  4. jangorecki commented on Apr 5, 2020

    @jangorecki
    Member

    @renkun-ken isn't this one a duplicate of your earlier issue linked by Henrik?

  5. renkun-ken commented on Apr 6, 2020

    @renkun-ken
    MemberAuthor

    @jangorecki The linked issue I posted earlier seems broader but overlaps with this. I'm totally okay if this issue is considered duplicate of that one.

  6. modified the milestones: 1.13.1, 1.13.3 on Oct 17, 2020
  7. modified the milestones: 1.14.3, on Jul 19, 2022
  8. modified the milestones: , 1.15.1 on Oct 29, 2023
  9. modified the milestones: 1.16.0, 1.17.0 on Jul 10, 2024
  10. modified the milestones: 1.17.0, 1.18.0 on Dec 10, 2024
  11. modified the milestones: 1.18.0, 1.19.0 on Nov 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions