Sklearn中的T-SNE给出NaN或Inf数据错误

Question

我试图在sklearn上运行t-sne来减少尺寸数据。

首先，我有一个tfidf矩阵。这是相同的代码。

def tf_vectorizer（docs）：“”“提取每个文档的术语频率向量

"""

from sklearn.feature_extraction.text import TfidfVectorizer
print("Extracting tfidf features for clustering...\n")
tf_vec = TfidfVectorizer(max_df=0.95, min_df=2,norm='l2').fit(docs)

print("Tf-idf features extracted!!\n")
return tf_vec

然后我使用TruncatedSVD将尺寸从11K减小到500.这就是

def reduce_dimensions(tfidf_data,n_components):
    """ This function will reduce the dimension of the dataset"""


    from sklearn.decomposition import TruncatedSVD
    svd=TruncatedSVD(n_components=n_components,random_state=42)

    svd_reduced_data=svd.fit_transform(tfidf_data)
    svd_reduced_data=svd_reduced_data.astype('float')
    #print("Explained Variance of all components {}".format(svd.explained_variance_ratio_))
    print("Total variance explained {}".format(svd.explained_variance_ratio_.sum()))

    return svd_reduced_data

我检查了svd_reduced_data的输出，检查是否有NaN或Inf。

np.isnan(svd_reduced_data).sum()
0

因此，这表明此数据中没有缺失值。现在我将500维的数据传递给tsne，将其减少到2维，如下所示：

  from sklearn.manifold import TSNE
   tsne=TSNE(n_components=n_components,n_iter=300,random_state=42)
   tsne_reduced_data=tsne.fit_transform(svd_reduced_data)

我收到这个错误：

/Users/anaconda/envs/dl/lib/python3.5/site-packages/scipy/linalg/misc.py in norm(a, ord, axis, keepdims)
    127     """
    128     # Differs from numpy only in non-finite handling and the use of blas.
--> 129     a = np.asarray_chkfinite(a)
    130 
    131     # Only use optimized norms if axis and keepdims are not specified.

/Users/anaconda/envs/dl/lib/python3.5/site-packages/numpy/lib/function_base.py in asarray_chkfinite(a, dtype, order)
   1231     if a.dtype.char in typecodes['AllFloat'] and not np.isfinite(a).all():
   1232         raise ValueError(
-> 1233             "array must not contain infs or NaNs")
   1234     return a
   1235 

ValueError: array must not contain infs or NaNs

当基础数据没有NaN时，不确定为什么会出现此错误。有帮助吗？

Answer 1

您还需要检查np.isinf()以确保没有无限的功能。