elasticsearch:使用多于一种类型的mongodb的数据索引问题

问题描述 投票:0回答:1

elasticsearch和mongodb的新手,我在批量索引中将数据从mongodb导入es时遇到了问题。要使用mongo-connector和elastic2-doc-manager同步数据。我能够成功地从一个集合中获取数据。但是,涉及到mongodb中django2_tables创建的元数据集合。 Elasticsearch出错了。顺便说一句,它较早就可以正常工作,不确定为什么以及如何。也许我缺少或忘记了一些基本配置。任何帮助将不胜感激。

谢谢,附加来自elasticsearch和mongo-connector的日志。来自elasticsearch的日志:

[2020-01-24T12:55:04,898][DEBUG][o.e.a.b.TransportShardBulkAction] [Qasims-MBP.hub] [djangodb01][0] failed to execute bulk item (index) index {[djangodb01][auth_permission][5e2a32540c31b2edc41dd9e6], source[{"id": 32, "name": "Can view project", "content_type_id": 8, "codename": "view_project"}]}
java.lang.IllegalArgumentException: Rejecting mapping update to [djangodb01] as the final mapping would have more than 1 type: [spiderCollection1, auth_permission]
    at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:272) ~[elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:238) ~[elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.5.1.jar:7.5.1]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.5.1.jar:7.5.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:830) [?:?]

来自mongo-connector的日志:

2020-01-24 12:55:04,909 [ERROR] mongo_connector.doc_managers.elastic2_doc_manager:505 - Bulk request failed with exception
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 497, in send_buffered_operations
    successes, errors = bulk(self.elastic, action_buffer)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/elasticsearch/helpers/__init__.py", line 257, in bulk
    for ok, item in streaming_bulk(client, actions, **kwargs):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/elasticsearch/helpers/__init__.py", line 188, in streaming_bulk
    for data, (ok, info) in zip(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/elasticsearch/helpers/__init__.py", line 137, in _process_bulk_chunk
    raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)
elasticsearch.helpers.BulkIndexError: ('40 document(s) failed to index.', [{'index': {'_index': 'djangodb01', '_type': 'django_content_type', '_id': '5e2a32540c31b2edc41dd9bf', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Rejecting mapping update to [djangodb01] as the final mapping would have more than 1 type: [spiderCollection1, django_content_type]'}, 'data': {'id': 1, 'app_label': 'admin', 'model': 'logentry'}}}, {'index': {'_index': 'djangodb01', '_type': 'auth_permission', '_id': '5e2a32540c31b2edc41dd9c0', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Rejecting mapping update to [djangodb01] as the final mapping would have more than 1 type: [spiderCollection1, auth_permission]'}, 'data': {'id': 1, 'name': 'Can add log entry', 'content_type_id': 1, 'codename': 'add_logentry'}}}, {'index': {'_index': 'djangodb01', '_type': 'auth_permission', '_id': '5e2a32540c31b2edc41dd9c1', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Rejecting mapping update to [djangodb01] as the final mapping would have more than 1 type: [spiderCollection1, auth_permission]'}, 'data': {'id': 2, 'name': 'Can change log entry', 'content_type_id': 1, 'codename': 'change_logentry'}}},
mongodb elasticsearch
1个回答
1
投票

在最新版本中,elasticsearch每个索引仅支持一种映射类型。详细信息Here

您面临的问题仍未解决:https://github.com/yougov/mongo-connector/issues/770

Here is a potential workaround。如果这样做不起作用,请考虑将ES降级为5.X版本,该版本允许多个映射。从mongo导入后,您将能够升级到版本6.X。这是some background

© www.soinside.com 2019 - 2024. All rights reserved.