在本周早些时候进行更新后，我注意到日志不再进入 Kibana。缩小范围后，我认为错误出在我的 logstash 服务器上。运行约 30 秒后，两者都有相同的错误。
[2023-05-11T12:10:07,387][ERROR][logstash.javapipeline ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"(NoMethodError) undefined method `shutdown_requested' for #<LogStash::JavaPipeline:0x60319724>", :exception=>Java::OrgJrubyExceptions::NoMethodError, :backtrace=>["usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_15_dot_2_minus_java.lib.logstash.plugin_mixins.elasticsearch.common.pipeline_shutdown_requested?(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.15.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:380)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_15_dot_2_minus_java.lib.logstash.outputs.elasticsearch.wait_for_successful_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.15.2-java/lib/logstash/outputs/elasticsearch.rb:440)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_15_dot_2_minus_java.lib.logstash.outputs.elasticsearch.multi_receive(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.15.2-java/lib/logstash/outputs/elasticsearch.rb:390)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.multi_receive(org/logstash/config/ir/compiler/OutputStrategyExt.java:143)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300)"], :thread=>"#<Thread:0x42c34b77 sleep>"}
在我运行 filebeat 的日志服务器上：
May 11 06:12:41 logs01.mgmt filebeat[18266]: 2023-05-11T06:12:41.005-0600 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to failover(backoff(async(tcp://10.0.0.226:5000)),backoff(async(tcp://10.0.0.227:5000))): dial tcp 10.0.0.226:5000: connect: connection refused
May 11 06:12:41 logs01.mgmt filebeat[18266]: 2023-05-11T06:12:41.005-0600 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to failover(backoff(async(tcp://10.0.0.226:5000)),backoff(async(tcp://10.0.0.227:5000))) with 26 reconnect attempt(s)
Filebeat 配置：
output.logstash:
# The Logstash hosts
hosts: ["10.0.0.226:5000","10.0.0.227:5000"]
Logstash 配置：
input {
beats {
port => 5000
type => syslog
}
}
我检查了 logstash 机器上的 netstat，它甚至没有在端口 5000 上侦听。看起来服务器启动然后几乎立即因管道工作者错误而崩溃。
我已经检查过，两端都没有运行防火墙。 SELinux 在两者上都被禁用。我已经尝试以 root 身份重新启动 logstash 和 filebeat 似乎都没有改变任何东西。