[并行处理paramiko.SFTPClient.get请求时如何防止我的应用挂起？

Question

我正在尝试通过SFTP并行化从服务器检索文件并上传到AWS。我正在使用python多线程，上载部分工作正常，但是，我注意到paramiko.SFTPClient的get操作使程序始终挂在最后。实际上，所有文件都已撤回并上载，但程序没有退出。我尝试了类似文章中的许多内容，但没有任何效果，以下是我的伪代码，欢迎您提供任何帮助：

def create_sftp_connection(host, port, username, password):
    transport = paramiko.Transport((host, port))
    transport.connect(username, password)
    sftp_client = paramiko.SFTPClient.from_transport(transport)

def get_and_upload_file(s3, sftp_client, file, local_full_path, destination_bucket, cloud_path):
     sftp_client.get(file, local_full_path)
     upload_file_to_s3(s3, local_full_path, destination_bucket, cloud_path)

def transfer_files(sftp_client, remote_path, local_path, destination_bucket):
     all_files = get_files_to_transfer(sftp_client, remote_path)
     s3 = init_s3()
     threads = list()

     for file in all_files:
         ....
         thread = threading.Thread(target=get_and_upload_file, args=(s3, sftp_client, file, local_full_path, destination_bucket, cloud_path))
         thread.daemon = True
         threads.append(thread)
         thread.start()

      for thread in threads:
           thread.join()


if __name__ == "__main__":
     sftp_client = create_sftp_connection(host, port, username, password)
     transfer_files(sftp_client, remote_path, local_path, destination_bucket)

Note：我还试图等待线程停止使用：

for thread in threads:
    while thread.is_alive():
          thread.join(timeout=0.1)

Answer 1

您需要为每个线程创建一个单独的连接（Transport）。