鉴于spark studio here提供的闪亮示例
library(sparklyr)
packageVersion("sparklyr")
#‘0.8.4’
sc <- spark_connect(master = "local")
#* Using Spark: 2.3.0
install.packages(c("nycflights13", "Lahman"))
library(dplyr)
flights_tbl <- copy_to(sc, nycflights13::flights, "flights")
src_tbls(sc)
#[1] "flights"
试图循环不同的值,
carriers <- flights_tbl %>% filter(dep_delay >= 1000) %>% dplyr::select(carrier) %>% dplyr::distinct()
> carriers
# Source: lazy query [?? x 1]
# Database: spark_connection
carrier
<chr>
1 MQ
2 HA
3 AA
> typeof(carriers)
[1] "list"
像这样:
for (carrier in carriers){
print(carrier)
}
返回此意外结果:
> for (carrier in carriers){
+ print(carrier)
+ }
spark connection master=local[4] app=sparklyr local=TRUE
From: flights
<Table: flights>
-> filter()
- ~dep_delay >= 1000
-> select()
- ~carrier
-> distinct()
而不是简单地:
MQ
HA
AA
如何循环不同的值?
好吧,显然首先应该“收集”火花结果(参见qazxsw poi)
here