Elasticsearch批量API发布请求中的NewLine错误

Question

我正在尝试使用elasticsearch批量api将多个记录插入索引。我的JSON看起来像这样：request json

我在文档末尾插入了新行（\\n），但仍然得到newline error。

    Error: {
        "error": {
            "root_cause": [
                {
                    "type": "illegal_argument_exception",
                    "reason": "The bulk request must be terminated by a newline [\n]"
                }
            ],
            "type": "illegal_argument_exception",
            "reason": "The bulk request must be terminated by a newline [\n]"
        },
        "status": 400
    }

Answer 1

您的json有时是nd-json（用新行分隔）JSON，但现在看起来一团糟，因此我们必须事先进行一些清理。

初始化：

const {
    Client
} = require("@elastic/elasticsearch");

const client = new Client({
    node: 'http://localhost:9200'
});

const INDEX_NAME = 'index_name';

将可能的ndjson转换为可消耗的数组或对象：

const docs_as_body_params = JSON.parse(
    '[' +
    `{"index":{}} {"tags":["ab","cd"],"question":"test this","answer":"answer first"} {"index":{}} {"tags":["de","fg"],"question":"test second","answer":"answer second"}`.split(
        /(\s?{"index":{}} )/g
    )
    // filter out empty strings
    .filter(match => match.length)
    // take every odd member (skipping `{"index":{}}`)
    .filter((_, index) => index % 2 !== 0) +
    ']'
);

构造主体

const bulk_body = [];
docs_as_body_params.map((doc) => {
    bulk_body.push({
        index: {
            _index: INDEX_NAME,
            _id: doc.id || null
        }
    });
    bulk_body.push(doc);
});

执行批量索引：

client.bulk({
        body: bulk_body
    },
    (err, resp) => {
        if (err || resp.errors) {
            console.err(err || resp.errors)
        }
        console.info(resp.body.items);
    }
);