如何将lubridate :: round_date用于sparklyr？

Question

我正在将日期时间截断为分钟，小时等。诸如lubridate :: round_date之类的方法非常有用。但是我不能与sparklyr一起使用吗？

Undefined function: 'round_date'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 46

Answer 1

您不‒见Why can't I use double colon operator with dplyr when the dataset is in sparklyr?。

正确的方法是使用内置的Spark SQL函数。

# For the sake of reproducibility
# https://stackoverflow.com/a/52365367/10465355
spark_session(sc) %>% 
  invoke("conf") %>% 
  invoke("set", "spark.sql.session.timeZone", "UTC")

options(tibble.width = 120)

df <- copy_to(sc, data.frame(
  ts = c("2019-01-08 23:21:15", "2020-02-06 13:14:00")
)) %>% mutate(ts = to_timestamp(ts))

df

# Source: spark<?> [?? x 1]
  ts                 
  <dttm>             
1 2019-01-08 23:21:15
2 2020-02-06 13:14:00

df %>% 
  transmute(
     year = date_trunc("year", ts),
     month = date_trunc("month", ts),
     day = date_trunc("day", ts),
     hour = date_trunc("hour", ts),
     minute = date_trunc("minute", ts)
  )

# Source: spark<?> [?? x 5]
  year                month               day                
  <dttm>              <dttm>              <dttm>             
1 2019-01-01 00:00:00 2019-01-01 00:00:00 2019-01-08 00:00:00
2 2020-01-01 00:00:00 2020-02-01 00:00:00 2020-02-06 00:00:00
  hour                minute             
  <dttm>              <dttm>             
1 2019-01-08 23:00:00 2019-01-08 23:21:00
2 2020-02-06 13:00:00 2020-02-06 13:14:00

不需要其他导入库或第三方库。