如何强制dask worker不将数据写入磁盘？

Question

我试图强迫工人不要将数据写入磁盘，但可能我做错了。

我修改了文件〜/ .config / dask / distributed.yaml，如下所示：

distributed:
  worker:
    # Fractions of worker memory at which we take action to avoid memory blowup
    # Set any of the lower three values to False to turn off the behavior entirely
    memory:
      target: 1.00  # target fraction to stay below
      spill: 1.00  # fraction at which we spill to disk
      pause: 1.00  # fraction at which we pause worker threads
      terminate: 1.00  # fraction at which we terminate the worker

但他们只是不断写入磁盘临时结果。我还试图使用虚构的内存限制设置（100GB），但他们仍然在任务结束时写在磁盘上。我如何强迫他们保持记忆？我错过了什么吗？

Answer 1

你正在做什么应该工作正常。我很惊讶它不适合你。

官方建议使用false值，如下所述：https://docs.dask.org/en/latest/setup/hpc.html#no-local-storage

distributed:
  worker:
    memory:
      target: false  # don't spill to disk
      spill: false  # don't spill to disk
      pause: 0.80  # pause execution at 80% memory use
      terminate: 0.95  # restart the worker at 95% use