在毫秒内优化 Python 图像下载 30MB 图像

Question

Stack Overflow 社区您好，

我目前正在编写一个 Python 脚本来有效下载 30MB 大小的图像。我一直在使用requests库，但是下载时间没有达到我的预期。下面是我的代码的简化版本： 导入请求

def download_image(url, save_folder):
    # Your download logic here

if __name__ == "__main__":
    image_urls = ["https://example.com/image1.jpg", "https://example.com/image2.jpg"]
    save_folder = "downloaded_images"

    for url in image_urls:
        download_image(url, save_folder)

我的目标是在毫秒时间内实现更快的下载。我知道网络条件和服务器响应时间是限制因素，但我正在寻找脚本本身的优化。是否有更有效的库或方法来实现更快的下载？

具体来说，我感兴趣的是：

对我当前代码的任何改进。 以更快的图像下载速度而闻名的替代库或方法。 在 Python 中优化图像下载过程的一般技巧或最佳实践。 我愿意接受社区的任何见解或建议。预先感谢您的帮助！

Answer 1

我相信最好的方法是并行下载图像，就像这样

import asyncio
import aiohttp
import time
from urllib.parse import urlparse
import os

image_urls = ["https://example.com/image1.jpg", "https://example.com/image2.jpg"]
save_folder = "downloaded_images"

async def get(url, session, save_folder):
    try:
        async with session.get(url=url) as response:
            resp = await response.read()
            parsed = urlparse(url)
            destination = os.path.join(save_folder, os.path.basename(parsed.path))
            with open(destination, "w") as f:
                f.write(resp)
            print("Successfully got url {} with resp of length {}.".format(url, len(resp)))
    except Exception as e:
        print("Unable to get url {} due to {}.".format(url, e.__class__))


async def main(urls, save_folder):
    async with aiohttp.ClientSession() as session:
        ret = await asyncio.gather(*[get(url, session, save_folder) for url in urls])
    print("Finalized all. Return is a list of len {} outputs.".format(len(ret)))


start = time.time()
asyncio.run(main(image_urls, save_folder))
end = time.time()

print("Took {} seconds to pull {} images.".format(end - start, len(image_urls)))