使用 python 异步上传文件到 azure blob 遇到困难

Question

我有一个图像目录，我想将其上传到 azure blob。 我使用以下Python代码：

    from multiprocessing import Pool
    def upload_file_to_blob(file_path, dest):
         file_name = os.path.basename(file_path)
         blob_client = BlobClient.from_connection_string(connection_string, container_name, os.path.join(dest, file_name))
         with open(file_path, "rb") as data:        
             blob_client.upload_blob(data, overwrite=True)
      
    with Pool(processes=10) as pool:
         for file_ind, file_path in enumerate(file_list):
             pool.apply_async(upload_file_to_blob, args=(file_path, file_dest))
         pool.close()
         pool.join()
    print(f"done")

看起来所有文件确实都已上传，但程序卡在完成处，并且最终打印未打印到终端。我做错了什么？

Answer 1

使用下面的代码，我可以将多个文件从文件夹上传到 Azure Blob 存储。

import os
from multiprocessing import Pool
from azure.storage.blob import BlobClient

# Replace these placeholders with your Azure Blob Storage connection information
connection_string = ""
container_name = " "
local_file_path = " "  # Replace with your local file path
file_dest = "your_destination_folder"

def upload_file_to_blob(file_path, dest):
    file_name = os.path.basename(file_path)
    blob_client = BlobClient.from_connection_string(connection_string, container_name, os.path.join(dest, file_name))
    with open(file_path, "rb") as data:
        blob_client.upload_blob(data, overwrite=True)

if __name__ == "__main__":
    with Pool(processes=1) as pool:  # Using 1 process for a single file
        result = pool.apply_async(upload_file_to_blob, args=(local_file_path, file_dest))
        result.get()  # Wait for the task to complete

    print("done")

输出：

• 与

  processes=10

• 有关更多详细信息，请参阅此SO。