Colab 如何增加数据加载器中的 num_workers

Question

当我尝试在

num_workers

中指定

dataloader

时，出现此错误。如果

colab

不允许这样做或者我需要做些什么，有人可以告诉我吗？

`---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in _try_get_data(self, timeout)
   1132         try:
-> 1133             data = self._data_queue.get(timeout=timeout)
   1134             return (True, data)

10 frames
RuntimeError: DataLoader worker (pid 13603) is killed by signal: Segmentation fault. 

上述异常是导致以下异常的直接原因：

RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py in _try_get_data(self, timeout)
   1144             if len(failed_workers) > 0:
   1145                 pids_str = ', '.join(str(w.pid) for w in failed_workers)
-> 1146                 raise RuntimeError(f'DataLoader worker (pid(s) {pids_str}) exited unexpectedly') from e
   1147             if isinstance(e, queue.Empty):
   1148                 return (False, None)

RuntimeError: DataLoader worker (pid(s) 13603) exited unexpectedly` 

`  # Create data loaders with increased timeout
  train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True,num_workers=2)
  val_loader = DataLoader(val_dataset, batch_size=batch_size,num_workers=2)
  test_loader = DataLoader(test_dataset, batch_size=batch_size,num_workers=2`

我想通过增加工作人员数量来更快地加载数据集。

Answer 1

colab

会有所帮助：

pin_memory=True