我一直在使用 tm 包来运行一些文本分析。 我的问题是创建一个包含单词及其相关频率的列表
library(tm)
library(RWeka)
txt <- read.csv("HW.csv",header=T)
df <- do.call("rbind", lapply(txt, as.data.frame))
names(df) <- "text"
myCorpus <- Corpus(VectorSource(df$text))
myStopwords <- c(stopwords('english'),"originally", "posted")
myCorpus <- tm_map(myCorpus, removeWords, myStopwords)
#building the TDM
btm <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
myTdm <- TermDocumentMatrix(myCorpus, control = list(tokenize = btm))
我通常使用以下代码来生成频率范围内的单词列表
frq1 <- findFreqTerms(myTdm, lowfreq=50)
有什么方法可以自动执行此操作,以便我们获得包含所有单词及其频率的数据框?
我面临的另一个问题是将术语文档矩阵转换为数据框。当我处理大量数据样本时,我遇到了内存错误。 有没有简单的解决方案?
试试这个
data("crude")
myTdm <- as.matrix(TermDocumentMatrix(crude))
FreqMat <- data.frame(ST = rownames(myTdm),
Freq = rowSums(myTdm),
row.names = NULL)
head(FreqMat, 10)
# ST Freq
# 1 "(it) 1
# 2 "demand 1
# 3 "expansion 1
# 4 "for 1
# 5 "growth 1
# 6 "if 1
# 7 "is 2
# 8 "may 1
# 9 "none 2
# 10 "opec 2
我在 R 中有以下几行可以帮助创建单词频率并将它们放入表格中,它读取 .txt 格式的文本文件并创建单词频率,我希望这可以帮助任何感兴趣的人。
avisos<- scan("anuncio.txt", what="character", sep="\n")
avisos1 <- tolower(avisos)
avisos2 <- strsplit(avisos1, "\\W")
avisos3 <- unlist(avisos2)
freq<-table(avisos3)
freq1<-sort(freq, decreasing=TRUE)
temple.sorted.table<-paste(names(freq1), freq1, sep="\\t")
cat("Word\tFREQ", temple.sorted.table, file="anuncio.txt", sep="\n")
查看 findFreqTerms
的
source,看起来函数
slam::row_sums
在术语文档矩阵上调用时可以发挥作用。例如,尝试一下:
data(crude)
slam::row_sums(TermDocumentMatrix(crude))
根据您的需求,使用一些
tidyverse
函数可能是一个粗略的解决方案,它在如何处理大写、标点符号和停用词方面提供了一定的灵活性:
text_string <- 'I have been using the tm package to run some text analysis. My problem is with creating a list with words and their frequencies associated with the same. I typically use the following code for generating list of words in a frequency range. Is there any way to automate this such that we get a dataframe with all words and their frequency?
The other problem that i face is with converting the term document matrix into a data frame. As i am working on large samples of data, I run into memory errors. Is there a simple solution for this?'
stop_words <- c('a', 'and', 'for', 'the') # just a sample list of words I don't care about
library(tidyverse)
data_frame(text = text_string) %>%
mutate(text = tolower(text)) %>%
mutate(text = str_remove_all(text, '[[:punct:]]')) %>%
mutate(tokens = str_split(text, "\\s+")) %>%
unnest() %>%
count(tokens) %>%
filter(!tokens %in% stop_words) %>%
mutate(freq = n / sum(n)) %>%
arrange(desc(n))
# A tibble: 64 x 3
tokens n freq
<chr> <int> <dbl>
1 i 5 0.0581
2 with 5 0.0581
3 is 4 0.0465
4 words 3 0.0349
5 into 2 0.0233
6 list 2 0.0233
7 of 2 0.0233
8 problem 2 0.0233
9 run 2 0.0233
10 that 2 0.0233
# ... with 54 more rows
a = scan(file='~/Desktop//test.txt',what="list")
a1 = data.frame(lst=a)
count(a1,vars="lst")
似乎可以获取简单的频率。我使用过 scan,因为我有一个 txt 文件,但它也应该适用于 read.csv。
apply(myTdm, 1, sum)
或 rowSums(as.matrix(myTdm))
是否给出了您想要的 ngram 计数?
使用
qdap
包:
text_string <- 'I have been using the tm package to run some text analysis. My problem is with creating a list with words and their frequencies associated with the same. I typically use the following code for generating list of words in a frequency range. Is there any way to automate this such that we get a dataframe with all words and their frequency?
The other problem that i face is with converting the term document matrix into a data frame. As i am working on large samples of data, I run into memory errors. Is there a simple solution for this?'
qdap::freq_terms(text_string, stopwords = c("of", "a", "i", "with", "is"), top = Inf)
# WORD FREQ
# 1 the 5
# 2 words 3
# 3 and 2
# 4 data 2
# 5 for 2
# 6 frequency 2
# 7 into 2
# 8 list 2
# 9 problem 2
# 10 run 2
# ...
默认情况下此功能只会显示前 20 个,因此我们设置
top = Inf
。您可以将任何字符向量传递给 stopwords
参数:c(tm::stopwords(), "other", "stop", "words")
.