Elasticsearch Spring boot findAll 结果窗口太大，from + 大小必须小于或等于：[10000] 但为 [331576]

Question

我是elasticsearch的新手，我正在使用spring data elasticsearch（https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#reference）。

这是我的pom.xml

<properties>
        <spring-data-elasticsearch.version>3.2.6</spring-data-elasticsearch.version>

...
<!-- Elasticsearch -->
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-elasticsearch</artifactId>
            <version>${spring-data-elasticsearch.version}.RELEASE</version>
        </dependency>
    </properties>

我有代码

Iterable<Data> dataList = this.dataRepository.findAll();

我的数据存储库就是这样

public interface DataRepository extends ElasticsearchRepository<myData, String> {
}

我收到错误

"Result window is too large, from + size must be less than or equal to: [10000] but was [331576]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"dfs","grouped":true,"failed_shards":[{"shard":0,"index":"hardwarezone_index","node":"Psv3GnjpQ52aNB52QfvWWw","reason":{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [331576]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}]},"status":500}

我需要检索所有数据以对其进行一些处理。我该怎么做？

Answer 1

这是弹性搜索的限制（并且有充分的理由）。您不能（也不应该）在单个搜索操作中加载巨大的数据集。 Elasticsearch 滚动 API 正是为此目的而设计的。

spring-data-elasticsearch 支持滚动 API 透明。您所要做的就是将存储库方法的返回类型从

Iterable<Data>

更改为

Stream<Data>

，spring-data-elasticsearch 将开始在后台使用滚动 API。这记录在第 11.2 节中。使用 Scroll For Big Result Set spring-data-elasticsearch 文档。

顺便说一句，虽然这会减少存储和持久层的负载，但如果您将此数据转换为集合而不是在控制器和 HttpConverter 层使用流，则可能会导致服务和控制器层需要大量内存春天之内。然而，这是一个单独的主题。如果您计划将其用于生产用例，我想与您分享。