我想计算同一文件的多个散列,并通过多重处理节省时间。
据我所知,从ssd读取文件相对较快,但哈希计算的速度几乎慢了4倍。如果我想计算2个不同的哈希值(md5和sha),则要慢8倍。我想能够在不同的处理器内核上并行计算不同的哈希值(最多4个,具体取决于设置),但不了解如何解决GIL。
这是我当前的代码(hash.py
):
import hashlib
from io import DEFAULT_BUFFER_SIZE
file = 'test/file.mov' #50MG file
def hash_md5(file):
md5 = hashlib.md5()
with open(file, mode='rb') as fl:
chunk = fl.read(DEFAULT_BUFFER_SIZE)
while chunk:
md5.update(chunk)
chunk = fl.read(DEFAULT_BUFFER_SIZE)
return md5.hexdigest()
def hash_sha(file):
sha = hashlib.sha1()
with open(file, mode='rb') as fl:
chunk = fl.read(DEFAULT_BUFFER_SIZE)
while chunk:
sha.update(chunk)
chunk = fl.read(DEFAULT_BUFFER_SIZE)
return sha.hexdigest()
def hash_md5_sha(file):
md5 = hashlib.md5()
sha = hashlib.sha1()
with open(file, mode='rb') as fl:
chunk = fl.read(DEFAULT_BUFFER_SIZE)
while chunk:
md5.update(chunk)
sha.update(chunk)
chunk = fl.read(DEFAULT_BUFFER_SIZE)
return md5.hexdigest(), sha.hexdigest()
def read_file(file):
with open(file, mode='rb') as fl:
chunk = fl.read(DEFAULT_BUFFER_SIZE)
while chunk:
chunk = fl.read(DEFAULT_BUFFER_SIZE)
return
我做了一些测试,这是结果:
from hash import *
from timeit import timeit
timeit(stmt='read_file(file)',globals=globals(),number = 100)
1.6323043460000122
>>> timeit(stmt='hash_md5(file)',globals=globals(),number = 100)
8.137973076999998
>>> timeit(stmt='hash_sha(file)',globals=globals(),number = 100)
7.1260356809999905
>>> timeit(stmt='hash_md5_sha(file)',globals=globals(),number = 100)
13.740918666999988
此结果应该是一个函数,主脚本将遍历文件列表,并应检查不同文件的不同哈希值(从1到4)。有什么想法可以实现吗?
如评论中的某人所述,您可以使用concurrent.futures
。我做过一些基准测试,最有效的方法是使用ProcessPoolExecutor
。这是一个例子:
executor = ProcessPoolExecutor(4)
executor.map(hash_function, files)
executor.shutdown()
[如果您想看一下我的基准,可以找到它们here和结果:
Total using read_file: 10.121980099997018
Total using hash_md5_sha: 40.49621040000693
Total (multi-thread) using read_file: 6.246223400000417
Total (multi-thread) using hash_md5_sha: 19.588415799999893
Total (multi-core) using read_file: 4.099713300000076
Total (multi-core) using hash_md5_sha: 14.448464199999762
我使用了40个300 MiB的文件进行测试。