我有一个大型数据集，我想通过仅使用
cmd我可以使用它，但是太慢了：
fread(cmd='findstr /r "^.*,.*,4[0-9]," data.csv')
如何有效地做到这一点，最好是让 findstr 或等效函数仅从第三列开始读取？
这是完整的 MWE。
library(data.table)
library(reprex)
library(rstudioapi)
library(styler)
#> Warning: package 'styler' was built under R version 4.2.3
data <- data.frame(
"A" = c(11, 33, 34, 11),
"B" = c("aaa","bbb","ccc","ddd"),
"C" = c(13,45,46,57),
"D" = c(1,2,3,4)
)
# Writing data to a CSV file
write.csv(data, "data.csv",row.names=FALSE)
#whole file
fread("data.csv")
#> A B C D
#> 1: 11 aaa 13 1
#> 2: 33 bbb 45 2
#> 3: 34 ccc 46 3
#> 4: 11 ddd 57 4
#filter based on first column(ok)
fread(cmd='findstr /r "^3[0-9]" data.csv')
#> V1 V2 V3 V4
#> 1: 33 bbb 45 2
#> 2: 34 ccc 46 3
#filter based on third column but reads the first two columns as well (works as intended, but too slow)
fread(cmd='findstr /r "^.*,.*,4[0-9]," data.csv')
#> V1 V2 V3 V4
#> 1: 33 bbb 45 2
#> 2: 34 ccc 46 3
#filter based on third column (does not work)
#fread(cmd='for /f "tokens=3 delims=," %i in (findstr /r "4[0-9]" data.csv) do echo %i')
# in cmd, this would print the numbers 45,46 as seperate outputs - and not return the filtered csv
#for /f "tokens=3 delims=," %i in ('findstr /r "4[0-9]" data.csv') do echo %i
创建于 2024-02-28，使用 reprex v2.0.2
替代建议：使用
mlr演示：我有一个 2.2Mi 行 CSV，有 57 列（7 个字符串，其余数字）。我想过滤名为
numberlibrary(data.table)
ps::ps_memory_info()["rss"] / 1024^2
# rss
# 75.64062
Sys.which("mlr")
# mlr
# "/usr/bin/mlr"
system.time(obj <- fread(cmd = "mlr --csv filter '$number == 22' largedata.csv"))
# user system elapsed
# 21.526 1.317 13.809
dim(obj)
# [1] 64537 57
object.size(obj) # megabytes
# 29.3095 bytes
ps::ps_memory_info()["rss"] / 1024^2
# rss
# 114.8398
如果我使用比较做同样的事情
read.csv("largedata.csv")
fread("largedata.csv")
arrow::read_csv_arrow("largedata.csv", as_data_frame=TRUE) # all data
arrow::read_csv_arrow("largedata.csv", as_data_frame=FALSE) |>
filter(number == 22) |>
collect()
使用
mlr| Method | Time | Size | RSS |
|:----------------------|:-------|:----------|:----------|
| `read.csv` | 73.767 | 1045.60MB | 2850.20MB |
| `fread` | 3.085 | 1039.80MB | 1339.57MB |
| `arrow` true (=22) | 3.559 | 29.32MB | 427.64MB |
| `arrow` false (=22) | <2 | 29.31MB | 1460.91MB |
| `sqldf` | 27.72 | 52.80MB | 91.75MB |
| `fread(cmd='mlr...')` | 13.8 | 29.31MB | 114.84MB |
他们有二进制文件，甚至适用于 Windows。
（我在 RuiBarradas 的答案中加入了
sqldf以下是使用
arrow我不知道你会得到一个万无一失的方法，这样只使用
freadarrow::read_csv_arrowdplyras.data.table如果您提前知道相关列名称，则相当直接：
arr <- arrow::read_csv_arrow("data.csv", as_data_frame = FALSE)
arr |>
filter(between(C, 40, 49)) |>
collect()
# # A tibble: 2 × 4
# A B C D
# <int> <chr> <int> <int>
# 1 33 bbb 45 2
# 2 34 ccc 46 3
如果你知道这是第三个字段，但仅此而已，我们可以在这里使用一些“使用 dplyr 进行编程”：
fld3 <- as.symbol(names(arr)[3])
arr |>
filter(between(!!fld3, 40, 49)) |>
collect()
# # A tibble: 2 × 4
# A B C D
# <int> <chr> <int> <int>
# 1 33 bbb 45 2
# 2 34 ccc 46 3
并不是所有的R表达式都能执行，但是很多简单的函数和逻辑过滤都可以快速完成。
请注意，虽然
arrarr
# Table
# 4 rows x 4 columns
# $A <int64>
# $B <string>
# $C <int64>
# $D <int64>
不建议任何数据已加载到内存中，只是有关文件的一些“元”。例如，您可以看到它“知道”哪些是字符串，哪些是整数，等等。它保持“惰性”，直到您最终
collect()查看 https://arrow.apache.org/docs/r/articles/dataset.html 了解有关功能的一些演练，以及 https://arrow.apache.org/docs/dev/r /reference/acero.html 获取受支持的 R 函数的详细列表。
sqldfSELECTlibrary(sqldf)
#> Loading required package: gsubfn
#> Loading required package: proto
#> Loading required package: RSQLite
data <- data.frame(
"A" = c(11, 33, 34, 11),
"B" = c("aaa","bbb","ccc","ddd"),
"C" = c(13,45,46,57),
"D" = c(1,2,3,4)
)
# Writing data to a CSV file
write.csv(data, "data.csv",row.names=FALSE)
rm(data)
sql <- "select * from file where C between 40 and 49"
read.csv.sql("data.csv", sql)
#> A B C D
#> 1 33 "bbb" 45 2
#> 2 34 "ccc" 46 3
# or simply
sqldf::read.csv.sql("data.csv", "select * from file where C between 40 and 49")
#> A B C D
#> 1 33 "bbb" 45 2
#> 2 34 "ccc" 46 3
创建于 2024-02-28，使用 reprex v2.0.2