使用python应用程序进行多处理，以减少目前36小时的运行时间

Question

我目前正在开发一个数据挖掘项目，该项目正在创建一个18000x18000的相似性矩阵

以下是构建矩阵的两种方法

def CreateSimilarityMatrix(dbSubsetData, distancePairsList):

    global matrix
    matrix = [ [0.0 for y in range(dbSubsetData.shape[0])] for x in range(dbSubsetData.shape[0])]
    for i in range(len(dbSubsetData)): #record1

        SimilarityArray = []
        start = time.time()
        for j in range(i+1, len(dbSubsetData)): #record2
            Similarity = GetDistanceBetweenTwoRecords(dbSubsetData, i, j, distancePairsList)

            #The similarities are all very small numbers which might be why the preference value needs to be so precise. 
            #Let's multiply the value by a scalar 10 to give the values more range. 
            matrix[i][j] = Similarity * 10.0
            matrix[j][i] = Similarity * 10.0

        end = time.time()
    return matrix

def GetDistanceBetweenTwoRecords(dbSubsetData, i, j, distancePairsList):

    Record1 = dbSubsetData.iloc[i]
    Record2 = dbSubsetData.iloc[j]
    columns = dbSubsetData.columns
    distancer    = 0.0
    distancec    = 0.0

    for i in range(len(Record1)):

        columnName = columns[i]
        Record1Value = Record1[i]
        Record2Value = Record2[i]

        if(Record1Value != Record2Value):
            ob = distancePairsList[distancePairsDict[columnName]-1]

            if(ob.attributeType == "String"):
                strValue = Record1Value+":"+Record2Value
                strValue2 = Record2Value+":"+Record1Value

                if strValue in ob.distancePairs:
                    val = ((ob.distancePairs[strValue])**2) 
                    val = val * -1
                    distancec = distancec + val
                elif strValue2 in ob.distancePairs:
                    val = ((ob.distancePairs[strValue2])**2)
                    val = val * -1
                    distancec = distancec + val

            elif(ob.attributeType == "Number"):
                val = ((Record1Value - Record2Value)*ob.getSignificance())**2 
                val = val * -1
                distancer = distancer + val

    distance = distancer + distancec
    return distance

每次迭代循环18000x19次（每行18000次，每个属性19次）。迭代总数为（18000x18000x19）/ 2，因为它是对称的，因此我只需要做矩阵的一半。这将花费大约36个小时来完成，这是一个我显然想要削减的时间框架。

我认为多处理就是诀窍。由于每一行都是独立生成数字并将它们拟合到矩阵中，因此我可以使用CreateSimilarityMatrix运行多进程。所以我在创建我的流程的函数中创建了这个

matrix = [ [0.0 for y in range(SubsetDBNormalizedAttributes.shape[0])] for x in range(SubsetDBNormalizedAttributes.shape[0])] 
if __name__ == '__main__':    
    procs = []
    for i in range(4):
        proc = Process(target=CreateSimilarityMatrix, args=(SubsetDBNormalizedAttributes, distancePairsList, i, 4))
        procs.append(proc)
        proc.start()
        proc.join()

创建相似度矩阵现在更改为

def CreateSimilarityMatrix(dbSubsetData, distancePairsList, counter=0, iteration=1):

    global Matrix

    for i in range(counter, len(dbSubsetData), iteration): #record1

        SimilarityArray = []
        start = time.time()
        for j in range(i+1, len(dbSubsetData)): #record2
            Similarity = GetDistanceBetweenTwoRecords(dbSubsetData, i, j, distancePairsList)
            #print("Similarity Between Records",i,":",j," is ", Similarity)

            #The similarities are all very small numbers which might be why the preference value needs to be so precise. 
            #Let's multiply the value by a scalar 10 to give the values more range. 
            Matrix[i][j] = Similarity * 10.0
            Matrix[j][i] = Similarity * 10.0

        end = time.time()
        print("Iteration",i,"took",end-start,"(s)")

目前这是s-l-o-w。这真的很慢。启动一个进程需要几分钟，然后启动下一个进程需要几分钟。我以为这些应该同时运行？我的流程申请是否不正确？

Answer 1

如果您正在使用CPython，那么有一种称为全局解释器锁（GIL）的东西，它使得实际多线程变得困难，同时使事情变得更快，并且可以反而大大减慢速度。

如果你正在处理矩阵，使用numpy，这肯定比普通的Python快很多。