在javascript mvc项目中批量插入mongodb。

Question

我需要为别人开发的一个项目（与我的硕士学位有关）做出贡献，但我无法联系到实际的开发人员。我是MongoDB的新手。在项目中，他通过insertmany函数将数据插入MongoDB中。但是，现在我试图插入的数据比较大，并给了我下面的错误。我指定了相关的代码范围。我如何才能进行bulkinsert？我试着将数据拼接成更小的数组，但是我得到了回调已经被调用的错误。我应该bulkinsert还是拼接数据，我应该用哪种方式？

谢谢。

错误。

2020-05-02T19:50:20+0200 <info> admin.js:521 () Error occured..on bulk pool saving... { MongoError: BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"
    at Function.MongoError.create (C:\TopicBinder2-master\node_modules\mongodb-core\lib\error.js:31:11)
    at C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\pool.js:497:72
    at authenticateStragglers (C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\pool.js:443:16)
    at Connection.messageHandler (C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\pool.js:477:5)
    at Socket.<anonymous> (C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\connection.js:331:22)
    at Socket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
  name: 'MongoError',
  message:
   'BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"',
  operationTime:
   Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1588441812 },
  ok: 0,
  errmsg:
   'BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"',
  code: 10334,
  codeName: 'Location10334',
  '$clusterTime':
   { clusterTime:
      Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1588441812 },
     signature: { hash: [Binary], keyId: [Long] } } }
2020-05-02T19:50:20+0200 <error> admin.js:400 () Error creating pool { message: 'Error occured on bulk saving!' }

var Pools = (module.exports = mongoose.model("pools", poolSchema));
module.exports.createPoolItems = function (poolItems, callback) {
  populateUniqueId(poolItems);

  Pools.collection.insertMany(poolItems, { ordered: false }, callback);
};

function populateUniqueId(poolItems) {
  if (poolItems.length > 0) {
    poolItems.forEach((element) => {
      element.unique_id =
        element.project + "_" + element.topic_id + "_" + element.document_id;
    });
  }
}

Answer 1

这个错误与你的文档超过16MB有关。而这是MongoDB的限制。你看更多细节。MongoDB的限制和阈值

...
 errmsg:
   'BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"',
...

你可以从下面的链接中找到数据模型的例子和模式。

MongoDB数据模型