我想创建我的data.table列,对于与另一个ID列,如果有相同ID的前一年进入相对于第三日期列。
我非常低效的解决方案:
library(data.table)
set.seed(123)
DT = data.table(
ID = c("b","b","b","a","a","c"),
dates = sample(seq(as.Date('2016/01/01'), as.Date('2019/01/01'), by="day"), 12)
)
setorder(DT, ID, dates)
DT[, Desired_Column:=DT[ID == .BY[[1]] & year(dates) < year(.BY[[2]]), ID[.N]], by=.(ID, dates)]
我的问题:为什么是缓慢的大数据集,这将是一个办法做到这一点更快?
编辑:最初版本没有抓住问题的全部。我很惊讶,该过滤器year( dates ) > min( year( dates ) )
通过工作组,但实际上并非如此。我改变了dates
列,从而使全年2016
的日期是可能的。现在集团a
具有比年初2017
没有进入,这应该使Desired_Column
NA
中的第一项。
这是我希望得到的输出:
ID dates Desired_Column
1: a 2017-05-11 <NA>
2: a 2018-08-24 a
3: a 2018-10-24 a
4: a 2018-11-06 a
5: b 2016-11-11 <NA>
6: b 2017-03-23 b
7: b 2017-07-30 b
8: b 2017-08-23 b
9: b 2018-05-13 b
10: b 2018-08-30 b
11: c 2016-02-19 <NA>
12: c 2017-05-07 c
下面是与非球菌加入一个选项。作为“日期”一栏已经被订购,可以子集first
“年”的“ID”进行分组,并使用在非等距自联接用于创建“Desired_Column”,从而避免了一步拿到min
imum值
DT[, yr := year(dates)]
DT[DT[, .(yr = first(yr)), ID], Desired_Column := ID, on = .(ID, yr > yr)]
DT
# ID dates yr Desired_Column
# 1: a 2017-11-26 2017 <NA>
# 2: a 2018-10-05 2018 a
# 3: a 2018-11-15 2018 a
# 4: a 2018-11-21 2018 a
# 5: b 2017-07-30 2017 <NA>
# 6: b 2017-10-26 2017 <NA>
# 7: b 2018-01-18 2018 b
# 8: b 2018-02-03 2018 b
# 9: b 2018-07-30 2018 b
#10: b 2018-10-09 2018 b
#11: c 2017-02-03 2017 <NA>
#12: c 2017-11-23 2017 <NA>
我的方法
DT[ DT[, .I[ year(dates) > min(year(dates))], by = "ID"]$V1, Desired_Column := ID][]
# ID dates Desired_Column
# 1: a 2017-05-11 <NA>
# 2: a 2018-08-24 a
# 3: a 2018-10-24 a
# 4: a 2018-11-06 a
# 5: b 2016-11-11 <NA>
# 6: b 2017-03-23 b
# 7: b 2017-07-30 b
# 8: b 2017-08-23 b
# 9: b 2018-05-13 b
# 10: b 2018-08-30 b
# 11: c 2016-02-19 <NA>
# 12: c 2017-05-07 c
标杆
microbenchmark::microbenchmark(
my_solution = DT[ DT[, .I[ year( dates ) > min( year( dates ) ) ], by = "ID"]$V1, Desired_Column := ID][],
your_solution = DT[, Desired_Column:=DT[ID == .BY[[1]] & year(dates) < year(.BY[[2]]), ID[.N]], by=.(ID, dates)][],
akrun = {
DT[, yr := year(dates)]
DT[DT[, .(yr = first(yr)), ID], Desired_Column := ID, on = .(ID, yr > yr)]
}
)
# Unit: milliseconds
# expr min lq mean median uq max neval
# my_solution 1.349660 1.470769 1.670500 1.612211 1.836653 2.764091 100
# your_solution 4.317707 4.510213 4.877906 4.656327 4.893572 21.164655 100
# akrun 3.637755 3.812187 4.320189 4.197804 4.675306 10.018512 100
和长度1000的数据集
# Unit: milliseconds
# expr min lq mean median uq max neval
# my_solution 1.635860 1.787998 2.323437 2.038197 2.504854 10.82108 100
# your_solution 342.582218 352.706475 367.424500 359.987257 375.076633 467.85023 100
# akrun 3.749825 4.291949 5.448715 4.949456 5.368815 39.72218 100
和长度1,000,000数据集
# Unit: milliseconds
# expr min lq mean median uq max neval
# my_solution 270.8044 280.4150 324.1195 284.5502 390.1511 393.2282 10
# your_solution - I did not dare to run ;-)
# akrun 166.2049 167.8109 209.5945 178.2247 291.4220 297.0243 10
结论
我的子集化的回答工作最有效的data.tables高达约50,000行,即规模以上非等距加入的解决方案通过@akrun是性能赢家。
这里是一个办法
library(data.table)
library(lubridate)
DT[year(dates)>(min(year(dates))), Desired_Column:=ID, by=.(ID, year(dates))]