我怎样才能批量消费一个可迭代对象（同样大小的块）？

Question

我经常在Python中使用

batch()

。自 ES6 具有迭代器和生成器函数以来，在 JavaScript 中是否有一些替代方案？

Answer 1

我不得不为自己写一个，我在这里分享，让我和其他人可以在这里轻松找到：

// subsequently yield iterators of given `size`
// these have to be fully consumed
function* batches(iterable, size) {
  const it = iterable[Symbol.iterator]();
  while (true) {
    // this is for the case when batch ends at the end of iterable
    // (we don't want to yield empty batch)
    let {value, done} = it.next();
    if (done) return value;

    yield function*() {
      yield value;
      for (let curr = 1; curr < size; curr++) {
        ({value, done} = it.next());
        if (done) return;

        yield value;
      }
    }();
    if (done) return value;
  }
}

它产生生成器，而不是例如

Array

s。在再次调用

next()

之前，您必须完全消耗每批。

Answer 2

来这里是为了看看其他人的建议。这是我在看这篇文章之前最初用 TypeScript 编写的版本。

async function* batch<T>(iterable: AsyncIterableIterator<T>, batchSize: number) {
  let items: T[] = [];
  for await (const item of iterable) {
    items.push(item);
    if (items.length >= batchSize) {
      yield items;
      items = []
    }
  }
  if (items.length !== 0) {
    yield items;
  }
}

这允许您按如下所示分批使用可迭代对象。

async function doYourThing<T>(iterable: AsyncIterableIterator<T>) {
  const itemsPerBatch = 5
  const batchedIterable = batch<T>(iterable, itemsPerBatch)
  for await (const items of batchedIterable) {
    await someOperation(items)
  }
}

就我而言，这让我可以更轻松地在 Mongo 中使用 bulkOps，如下所示。

import { MongoClient, ObjectID } from 'mongodb';
import { batch } from './batch';

const config = {
  mongoUri: 'mongodb://localhost:27017/test?replicaSet=rs0',
};

interface Doc {
  readonly _id: ObjectID;
  readonly test: number;
}

async function main() {
  const client = await MongoClient.connect(config.mongoUri);
  const db = client.db('test');
  const coll = db.collection<Doc>('test');
  await coll.deleteMany({});
  console.log('Deleted test docs');

  const testDocs = new Array(4).fill(null).map(() => ({ test: 1 }));
  await coll.insertMany(testDocs);
  console.log('Inserted test docs');

  const cursor = coll.find().batchSize(5);
  for await (const docs of batch<Doc>(cursor as any, 5)) {
    const bulkOp = coll.initializeUnorderedBulkOp();
    docs.forEach((doc) => {
      bulkOp.find({ _id: doc._id }).updateOne({ test: 2 });
    });
    console.log('Updating', docs.length, 'test docs');
    await bulkOp.execute();
  }
  console.log('Updated test docs');
}

main()
  .catch(console.error)
  .then(() => process.exit());

Answer 3

这是 Typescript 中一个相对简洁的示例：

function* batchIterable<T>(iter: Iterable<T>, batchSize: number): Iterable<Iterable<T>> {
    const iterator = iter[Symbol.iterator]()
    let done = false
    while (!done) {
        const batch: T[] = []
        while (batch.length < batchSize) {
            const res = iterator.next()
            if (res.done) {
                done = true
                break
            } else {
                batch.push(res.value)
            }
        }
        if (batch.length > 0) {
            yield batch
        }
    }
}

适用于任何可迭代对象，包括数组：

> Array.from(batchIterable([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 3))
[ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ], [ 10 ] ]

还有发电机：

function* genNums() { 
    yield 1; 
    yield 2; 
    yield 3; 
    yield 4;
}

> Array.from(batchIterable(genNums(), 3))
[ [ 1, 2, 3 ], [ 4 ] ]

但是，对于从产生的值中返回一个单独值的生成器，情况并非如此：

function* genNums() { 
    yield 1; 
    yield 2; 
    yield 3; 
    yield 4;

    return 5;
}

> Array.from(batchIterable(genNums(), 3))
[ [ 1, 2, 3 ], [ 4 ] ]  // return-ed value 5 not included