无法加载glove.6B.300d.txt

Question

我正在尝试使用以下代码加载手套向量

en_model = gensim.models.KeyedVectors.load_word2vec_format(model_path, binary=False)

我意外地收到以下错误。

 File "/home/k/Desktop/Work/Vector explorer/word2vec-explorer/vec_test_loader.py", line 55, in make_model
en_model = KeyedVectors.load_word2vec_format(model_path, binary=is_bin)
 File "/home/k/.local/lib/python3.5/site-packages/gensim/models/keyedvectors.py", line 1119, in load_word2vec_format
limit=limit, datatype=datatype)
 File "/home/k/.local/lib/python3.5/site-packages/gensim/models/utils_any2vec.py", line 175, in _load_word2vec_format
vocab_size, vector_size = (int(x) for x in header.split())  # throws for invalid file format
 File "/home/k/.local/lib/python3.5/site-packages/gensim/models/utils_any2vec.py", line 175, in <genexpr>
vocab_size, vector_size = (int(x) for x in header.split())  # throws for invalid file format

ValueError: invalid literal for int() with base 10: 'the'

有人可以帮忙吗？

Answer 1

Gensim需要更多关于

model_path

的信息，我们必须在第一行附加两个数字，第一个表示我们有多少个单词词汇，第二个表示单词嵌入的维数，如下所示：

101 300
the 1.0 2.1 -1.3 ...
I   1.1 0.2 -0.3 ...
.
.
.

您可以尝试使用如下一行代码：

python -m gensim.scripts.glove2word2vec --input  glove.840B.300d.txt --output glove.840B.300d.w2vformat.txt

或者您可以使用我的代码作为参考：

import gensim
import os
import shutil
import hashlib
from sys import platform

def getFileLineNums(filename):
    f = open(filename, 'r')
    count = 0
    for line in f:
        count += 1
    return count


def prepend_line(infile, outfile, line):
    with open(infile, 'r') as old:
        with open(outfile, 'w') as new:
            new.write(str(line) + "\n")
            shutil.copyfileobj(old, new)

def prepend_slow(infile, outfile, line):
    with open(infile, 'r') as fin:
        with open(outfile, 'w') as fout:
            fout.write(line + "\n")
            for line in fin:
                fout.write(line)

def load(filename):
    num_lines = getFileLineNums(filename)
    gensim_file = 'glove_model.txt'
    gensim_first_line = "{} {}".format(num_lines, 300)
    # Prepends the line.
    if platform == "linux" or platform == "linux2":
        prepend_line(filename, gensim_file, gensim_first_line)
    else:
        prepend_slow(filename, gensim_file, gensim_first_line)

    model = gensim.models.KeyedVectors.load_word2vec_format(gensim_file)
    return model
model = load(your_model_path)

Answer 2

由于我们只是在行前面添加行数和维度计数（可以说，嵌入的长度和宽度），所以我使用了这个快速脚本：

#!/bin/bash

# Path to the GloVe file
GLOVE_FILE="path/to/glove.840B.300d.txt"

# Calculate the number of lines (words) in the GloVe file
NUM_LINES=$(wc -l < "$GLOVE_FILE")

# Embedding dimension (should match the dimension in the GloVe file)
EMBEDDING_DIM=300
    
# Extract the filename without the extension
FILENAME=$(basename "$GLOVE_FILE" .txt)

# Construct the new filename with "_w2vformat_bash" before the extension
NEW_FILE="${FILENAME}_w2vformat_bash.txt"

# Create a new file with the first line containing the number of lines and embedding dimension
echo "$NUM_LINES $EMBEDDING_DIM" | cat - "$GLOVE_FILE" > "$NEW_FILE"