Elasticsearch 设置自定义索引和写入问题

Question

Elasticsearch 通过 filebeats shipper 默认设置获取日志。所有自定义索引设置都在

/etc/filebeats/filebeats.yml

文件中配置。这是我的配置文件：

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["host-ip:9200"]
  protocol: "https"
  index: "samba-%{[agent.hostname]}-%{[agent.version]}-%{+dd.MM.yyyy}"
  # Authentication credentials - either API key or username/password.
  username: "elastic"
  password: "password"
  ssl:
    enabled: true
    certificate_authorities:
      - |
        -----BEGIN CERTIFICATE-----
       XXX
        -----END CERTIFICATE-----

setup.template:
  name: "samba"
  pattern: "samba-%{[agent.version]}"
  overwrite: true

setup.ilm.enabled: false

运行filebeat setup命令时，抛出

"no matching index template found for data stream [samba]"

异常，虽然这个自定义索引模板是在ELK上创建的。启动 filebeat 服务后，所有日志都收集在默认索引（.ds-filebeat-8.6.2-2023.03.09-000001）上。

更新： 简而言之，这是 api 调用输出：

{
    "index_templates": [
      {
        "name": "samba",
        "index_template": {
          "index_patterns": [
            "samba-8.6.2"
          ],
          "template": {
            "settings": {
              "index": {
                "mapping": {
                  "total_fields": {
                    "limit": "10000"
                  }
                },
                "refresh_interval": "5s",
                "number_of_shards": "1",
                "max_docvalue_fields_search": "200",
                "query": {
                  "default_field": [
                    // other fileds.
                    "fields.*"
                  ]
                }
              }
            },
            "mappings": {
              "_meta": {
                "beat": "filebeat",
                "version": "8.6.2"
              }
              // about 30.000 line is removed by use vscode ide.
            }
          },
          "composed_of": [],
          "priority": 150,
          "data_stream": {
            "hidden": false,
            "allow_custom_routing": false
          }
        }
      }
    ]
  }

Answer 1

长话短说；

错误说

no matching index template found for data stream [samba]

，实际上你的模式是

samba-%{[agent.version]}

解决方案

将图案更改为

samba-*

所以你的文件应该看起来像

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["host-ip:9200"]
  protocol: "https"
  index: "samba-%{[agent.hostname]}-%{[agent.version]}-%{+dd.MM.yyyy}"
  # Authentication credentials - either API key or username/password.
  username: "elastic"
  password: "password"
  ssl:
    enabled: true
    certificate_authorities:
      - |
        -----BEGIN CERTIFICATE-----
       XXX
        -----END CERTIFICATE-----

setup.template:
  name: "samba"
  pattern: "samba*"
  overwrite: true

setup.ilm.enabled: false