句子中每个单词的概率

Question

我有一堆文字，我需要找到文本中每个字母的概率，不包括标点符号和数字。我计算了每个字母的数量并将其形成字典：

import re
def tokenize(string):
    return re.compile('\w+').findall(string)

from collections import Counter
def word_freq(string): 
    text = tokenize(string.lower())
    str1 = ''.join(str(e) for e in text)
return dict(Counter(str1))

但是，我想找到每个字母的概率。谁能告诉我如何更新字典中的值？

Answer 1

从你的开始，你可以计算单词或字母。将它们作为字典归还。他们总结了总数和发生的数量&& print：

import re
def tokenize(string):
    return re.compile('\w+').findall(string)

from collections import Counter

def word_freq(string): 
    text = tokenize(string.lower())
    c = Counter(text)           # count the words
    d = Counter(''.join(text))  # count all letters
    return (dict(c),dict(d))    # return a tuple of counted words and letters

data = "This is a text, it contains  dupes and more dupes and dupes of dupes and lkkk."

words, letters = word_freq(data) # count and get dicts with counts

sumWords = sum(words.values())       # sum total words
sumLetters = sum(letters.values())   # sum total letters

# calc / print probability of word
for w in words:
    print("Probability of '{}': {}".format(w,words[w]/sumWords))

# calc / print probability of letter
for l in letters:
    print("Probability of '{}': {}".format(l,letters[l]/sumLetters))

要使用概率修改您的dict，只需将dicts值替换为计算出的概率：

# update the counts to propabilities:
for w in words: 
    words[w] = words[w]/sumWords

print ( words) 

输出：

# words
Probability of 'this': 0.0625
Probability of 'is': 0.0625
Probability of 'a': 0.0625
Probability of 'text': 0.0625
Probability of 'it': 0.0625
Probability of 'contains': 0.0625
Probability of 'dupes': 0.25
Probability of 'and': 0.1875
Probability of 'more': 0.0625
Probability of 'of': 0.0625
Probability of 'lkkk': 0.0625

# letters
Probability of 't': 0.08333333333333333
Probability of 'h': 0.016666666666666666
Probability of 'i': 0.06666666666666667
# [....] snipped some for brevity
Probability of 'f': 0.016666666666666666
Probability of 'l': 0.016666666666666666
Probability of 'k': 0.05

重新计算words的值后：

{'this': 0.0625, 'is': 0.0625, 'a': 0.0625, 'text': 0.0625, 'it': 0.0625,
 'contains': 0.0625, 'dupes': 0.25, 'and': 0.1875, 'more': 0.0625, 
 'of': 0.0625, 'lkkk': 0.0625}