我正在使用 SpraklyR 将 csv 读入 Spark
schema <- structType(structField("TransTime", "array<timestamp>", TRUE),
structField("TransDay", "Date", TRUE))
spark_read_csv(sc, filename, "path", infer_schema = FALSE, schema = schema)
但是得到:
Error: could not find function "structType"
如何使用spark_read_csv指定列类型?
提前致谢。
structType 函数来自 Scala 的 SparkAPI,在 Sparklyr 中要指定数据类型,您必须将其作为列表传递到“column”参数中,假设我们有以下 CSV(data.csv):
name,birthdate,age,height
jader,1994-10-31,22,1.79
maria,1900-03-12,117,1.32
读取对应数据的函数为:
mycsv <- spark_read_csv(sc, "mydate",
path = "data.csv",
memory = TRUE,
infer_schema = FALSE, #attention to this
columns = list(
name = "character",
birthdate = "date", #or character because needs date functions
age = "integer",
height = "double"))
# integer = "INTEGER"
# double = "REAL"
# character = "STRING"
# logical = "INTEGER"
# list = "BLOB"
# date = character = "STRING" # not sure
要操作日期类型,您必须使用 hive 日期函数,而不是 R 函数。
mycsv %>% mutate(birthyear = year(birthdate))
参考:https://spark.rstudio.com/articles/guides-dplyr.html#hive-functions
关于如何执行此操作的示例以及有关 Sparklyr 的免费在线书籍中解释的详细信息https://therinspark.com/data.html
但是 Jader Martins 答案中的命名列表示例更简单