按组检查是否以前项存在基于其他列

问题描述 投票:4回答:3

我想创建我的data.table列,对于与另一个ID列,如果有相同ID的前一年进入相对于第三日期列。

我非常低效的解决方案:

library(data.table)
set.seed(123)
DT = data.table(
  ID = c("b","b","b","a","a","c"),
  dates = sample(seq(as.Date('2016/01/01'), as.Date('2019/01/01'), by="day"), 12)
)

setorder(DT, ID, dates)
DT[, Desired_Column:=DT[ID == .BY[[1]] & year(dates) < year(.BY[[2]]), ID[.N]], by=.(ID, dates)]

我的问题:为什么是缓慢的大数据集,这将是一个办法做到这一点更快?

编辑:最初版本没有抓住问题的全部。我很惊讶,该过滤器year( dates ) > min( year( dates ) )通过工作组,但实际上并非如此。我改变了dates列,从而使全年2016的日期是可能的。现在集团a具有比年初2017没有进入,这应该使Desired_Column NA中的第一项。

这是我希望得到的输出:

      ID      dates Desired_Column
 1:  a 2017-05-11           <NA>
 2:  a 2018-08-24              a
 3:  a 2018-10-24              a
 4:  a 2018-11-06              a
 5:  b 2016-11-11           <NA>
 6:  b 2017-03-23              b
 7:  b 2017-07-30              b
 8:  b 2017-08-23              b
 9:  b 2018-05-13              b
10:  b 2018-08-30              b
11:  c 2016-02-19           <NA>
12:  c 2017-05-07              c
r data.table
3个回答
3
投票

下面是与非球菌加入一个选项。作为“日期”一栏已经被订购,可以子集first“年”的“ID”进行分组,并使用在非等距自联接用于创建“Desired_Column”,从而避免了一步拿到minimum值

DT[, yr := year(dates)]
DT[DT[, .(yr = first(yr)), ID],  Desired_Column := ID, on = .(ID, yr > yr)]
DT
#    ID      dates   yr Desired_Column
# 1:  a 2017-11-26 2017           <NA>
# 2:  a 2018-10-05 2018              a
# 3:  a 2018-11-15 2018              a
# 4:  a 2018-11-21 2018              a
# 5:  b 2017-07-30 2017           <NA>
# 6:  b 2017-10-26 2017           <NA>
# 7:  b 2018-01-18 2018              b
# 8:  b 2018-02-03 2018              b
# 9:  b 2018-07-30 2018              b
#10:  b 2018-10-09 2018              b
#11:  c 2017-02-03 2017           <NA>
#12:  c 2017-11-23 2017           <NA>

4
投票

我的方法

DT[ DT[, .I[ year(dates) > min(year(dates))], by = "ID"]$V1, Desired_Column := ID][]

#     ID      dates Desired_Column
#  1:  a 2017-05-11           <NA>
#  2:  a 2018-08-24              a
#  3:  a 2018-10-24              a
#  4:  a 2018-11-06              a
#  5:  b 2016-11-11           <NA>
#  6:  b 2017-03-23              b
#  7:  b 2017-07-30              b
#  8:  b 2017-08-23              b
#  9:  b 2018-05-13              b
# 10:  b 2018-08-30              b
# 11:  c 2016-02-19           <NA>
# 12:  c 2017-05-07              c

标杆

microbenchmark::microbenchmark( 
  my_solution = DT[ DT[, .I[ year( dates ) > min( year( dates ) ) ], by = "ID"]$V1, Desired_Column := ID][],
  your_solution = DT[, Desired_Column:=DT[ID == .BY[[1]] & year(dates) < year(.BY[[2]]), ID[.N]], by=.(ID, dates)][],
  akrun = {
    DT[, yr := year(dates)]
    DT[DT[, .(yr = first(yr)), ID],  Desired_Column := ID, on = .(ID, yr > yr)]
  }
)

# Unit: milliseconds
#          expr      min       lq     mean   median       uq       max neval
#   my_solution 1.349660 1.470769 1.670500 1.612211 1.836653  2.764091   100
# your_solution 4.317707 4.510213 4.877906 4.656327 4.893572 21.164655   100
#         akrun 3.637755 3.812187 4.320189 4.197804 4.675306 10.018512   100

和长度1000的数据集

# Unit: milliseconds
#          expr        min         lq       mean     median         uq       max neval
#   my_solution   1.635860   1.787998   2.323437   2.038197   2.504854  10.82108   100
# your_solution 342.582218 352.706475 367.424500 359.987257 375.076633 467.85023   100
#         akrun   3.749825   4.291949   5.448715   4.949456   5.368815  39.72218   100

和长度1,000,000数据集

# Unit: milliseconds
#          expr      min       lq     mean   median       uq      max neval
#   my_solution 270.8044 280.4150 324.1195 284.5502 390.1511 393.2282    10
# your_solution   - I did not dare to run ;-)
#         akrun 166.2049 167.8109 209.5945 178.2247 291.4220 297.0243    10

结论

我的子集化的回答工作最有效的data.tables高达约50,000行,即规模以上非等距加入的解决方案通过@akrun是性能赢家。


1
投票

这里是一个办法

library(data.table)
library(lubridate)
DT[year(dates)>(min(year(dates))), Desired_Column:=ID, by=.(ID, year(dates))]
© www.soinside.com 2019 - 2024. All rights reserved.