需要 Elasticsearch Logstash 过滤器方面的帮助

Question

我想从另一个索引丰富主机名，因为在某些情况下，我的主索引中缺少该值。因此，我使用 Elasticsearch Logstash 过滤器来查询主机名{如附件所示}。

但是，当我使用命令 {manually} 测试管道时，出现如下所述的错误 -

[ERROR] 2023-04-24 10:02:58.784 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<Elasticsearch::Transport::Transport::Errors::Unauthorized: [401] >, :backtrace=>["/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/transport/base.rb:341:in `perform_request'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/transport/http/manticore.rb:91:in `perform_request'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/client.rb:197:in `perform_request'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-7.17.1/lib/elasticsearch.rb:41:in `method_missing'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-api-7.17.1/lib/elasticsearch/api/actions/ping.rb:38:in `ping'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:330:in `test_connection!'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:118:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:233:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:232:in `register_plugins'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:599:in `maybe_setup_out_plugins'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:245:in `start_workers'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:190:in `run'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["/data/logstash/pipelines/sendEmailAlerts_updated.conf"], :thread=>"#<Thread:0x1d8d4a8@/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:130 run>"}
[INFO ] 2023-04-24 10:02:58.785 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[ERROR] 2023-04-24 10:02:58.793 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

注意：其他管道工作正常，没有任何错误；因此看起来 Logstash 配置文件没问题。

logstash 配置文件

input {
        elasticsearch
        {
        hosts => "localhost:9200"
        user => "reader"
        password => "*******************"
        index => "*-testalert"
        query => '{ "query": {
                             "bool": {
                                      "must": [{"terms": { "kibana.alert.severity": [ "high", "critical"] }} ],
                                      "filter": [ {"range": {"@timestamp": { "gte": "now-2d"}}}]
                                     }
                             }
                  }'
        schedule => "/5 * * * *"
        size => 500
        scroll => "5m"
        docinfo => true
        docinfo_target => "[@metadata][doc]"
        codec => "json"
        }
    }



filter {
if [host][hostname] != ""
{
mutate {
        add_field => {
            "alertHostName" => "%{[host][hostname]}"
            "alertReason" => "%{kibana.alert.reason}"
            "alertSeverity" => "%{kibana.alert.severity}"
            "alertTime" => "%{kibana.alert.original_time}"

        }
    }
}
if [host][hostname] == ""
{
elasticsearch {
              hosts => "localhost:9200"
              index => ".fleet-agents"
              query => "{[local_metadata][host][id]}:%{[host][id]}"
              fields => {
                       "[local_metadata][host][id]" => "host_name"
                      }
              }

mutate {
        add_field => {
            "alertHostName" => "%{[host_name]}"
            "alertReason" => "%{kibana.alert.reason}"
            "alertSeverity" => "%{kibana.alert.severity}"
            "alertTime" => "%{kibana.alert.original_time}"

        }
    }
}
}

output {
stdout {
 codec => "json"
}
}

Answer 1

在第一行日志中，您可以看到此错误：

未经授权：[401]

所以看起来你只是在查询

elasticsearch

索引的

.fleet-agents

过滤器中缺少一些身份验证。

您可能需要添加

user => "reader"

和适当的

password

与输入中的相同。