我正在尝试 将我的 Logstash Docker 容器连接到我的 Elasticsearch Docker 容器。
首先我将向 Elasticsearch 展示我正在使用的配置：
我正在使用单独的 docker-compose 文件运行这两项服务，以便能够手动等待一项服务准备就绪等（只是暂时）
这是我的 Elasticsearch
docker-compose.yamlservices:
elasticsearch:
image: elasticsearch:7.17.5
volumes:
- ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,z
ports:
- 9200:9200 #API calls over HTTP - Maps the container port to a host port -> <host port>:<container port>
- 9300:9300 #Custom binary protocol used for communications between nodes in a cluster
environment:
# Maximum amount of memory the JVm will allocate to a process.
ES_JAVA_OPTS: -Xms512m -Xmx512m
# Bootstrap password.
# Used to initialize the keystore during the initial startup of
# Elasticsearch. Ignored on subsequent runs.
ELASTIC_PASSWORD: elasticsearch
映射到容器内部的
elasticsearch.yml---
## Default Elasticsearch configuration from Elasticsearch base image.
## https://github.com/elastic/elasticsearch/blob/master/distribution/docker/src/docker/config/elasticsearch.yml
cluster.name: "docker-cluster"
network.host: 0.0.0.0
## X-Pack settings
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html
xpack.license.self_generated.type: basic
xpack.security.enabled: true
## Custom
# Use single node discovery in order to disable production mode and avoid bootstrap checks.
# see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
discovery.type: single-node
容器运行起来！
curl http://elastic:elasticsearch@localhost:9200{
"name" : "a1d7acf92aa1",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "PstzFCUqRXKQx9WeD0-6WA",
"version" : {
"number" : "7.17.5",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "8d61b4f7ddf931f219e3745f295ed2bbc50c8e84",
"build_date" : "2022-06-23T21:57:28.736740635Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
我注意到该服务处于 状态“黄色”！
curl -u elastic http://localhost:9200/_cluster/health\?prettyEnter host password for user 'elastic':
{
"cluster_name" : "docker-cluster",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 4,
"active_shards" : 4,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 80.0
}
我不确定这是否与logstash无法正确连接有关。
现在我的Logstash
docker-compose.yamlversion: "3.9"
services:
logstash:
image: logstash:7.17.5
volumes:
- ./pipeline:/usr/share/logstash/pipeline:ro,Z #Mount logstash pipeline config into container
ports:
- "5044:5044"
- "50000:50000/tcp"
- "50000:50000/udp"
- "9600:9600"
environment:
# Maximum amount of memory the JVm will allocate to a process.
ES_JAVA_OPTS: -Xms512m -Xmx512m
LOGSTASH_INTERNAL_PASSWORD: logstash
networks:
default:
name: elasticsearch_default
external: true
Elasticsearch镜像的网络默认为
elasticsearch_default映射到容器内部的
logstash.confinput {
beats {
port => 5044
}
tcp {
port => 50000
}
}
## Add your filters / logstash plugins configuration here
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "elastic"
password => "elasticsearch"
}
}
现在，当我启动 Logstash 容器时，我得到以下输出：
ecreating logstash_logstash_1 ... done
Attaching to logstash_logstash_1
logstash_1 | 2022/07/25 09:22:01 Setting 'xpack.monitoring.elasticsearch.ssl.verification_mode' from environment.
logstash_1 | Using bundled JDK: /usr/share/logstash/jdk
logstash_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash_1 | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1 | [2022-07-25T09:22:10,864][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
logstash_1 | [2022-07-25T09:22:10,870][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.17.5", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
logstash_1 | [2022-07-25T09:22:10,871][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/]
logstash_1 | [2022-07-25T09:22:10,885][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash_1 | [2022-07-25T09:22:10,890][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash_1 | [2022-07-25T09:22:11,056][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"5dd7cac8-7d3f-4d24-8672-9c7bf445f893", :path=>"/usr/share/logstash/data/uuid"}
logstash_1 | [2022-07-25T09:22:11,565][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
logstash_1 | [2022-07-25T09:22:11,567][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and may be removed in a future release.
logstash_1 | Please configure Metricbeat to monitor Logstash. Documentation can be found at:
logstash_1 | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash_1 | [2022-07-25T09:22:11,791][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2022-07-25T09:22:11,841][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2022-07-25T09:22:12,018][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash_1 | [2022-07-25T09:22:12,116][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
logstash_1 | [2022-07-25T09:22:12,144][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/_xpack'"}
logstash_1 | [2022-07-25T09:22:12,161][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
logstash_1 | [2022-07-25T09:22:12,320][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
logstash_1 | [2022-07-25T09:22:13,078][INFO ][org.reflections.Reflections] Reflections took 53 ms to scan 1 urls, producing 119 keys and 419 values
logstash_1 | [2022-07-25T09:22:13,501][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2022-07-25T09:22:13,511][WARN ][deprecation.logstash.inputs.beats] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2022-07-25T09:22:13,553][WARN ][deprecation.logstash.codecs.line] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2022-07-25T09:22:13,558][WARN ][deprecation.logstash.inputs.tcp] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2022-07-25T09:22:13,575][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2022-07-25T09:22:13,591][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1 | [2022-07-25T09:22:13,629][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
logstash_1 | [2022-07-25T09:22:13,647][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}}
logstash_1 | [2022-07-25T09:22:13,714][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch:9200/"}
logstash_1 | [2022-07-25T09:22:13,727][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.17.5) {:es_version=>7}
logstash_1 | [2022-07-25T09:22:13,728][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
logstash_1 | [2022-07-25T09:22:13,773][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
logstash_1 | [2022-07-25T09:22:13,774][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
logstash_1 | [2022-07-25T09:22:13,796][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
logstash_1 | [2022-07-25T09:22:13,817][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x765f0a97 run>"}
logstash_1 | [2022-07-25T09:22:14,340][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.52}
logstash_1 | [2022-07-25T09:22:14,364][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
logstash_1 | [2022-07-25T09:22:14,467][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1 | [2022-07-25T09:22:14,473][INFO ][logstash.inputs.tcp ][main][feac9b54ea188d2b2c8c296a8162a4fbc47d86d4390fe1ecf5e3d888b981444c] Starting tcp input listener {:address=>"0.0.0.0:50000", :ssl_enable=>false}
logstash_1 | [2022-07-25T09:22:14,495][INFO ][org.logstash.beats.Server][main][ab49b76e319e88922a61b48e46cf6f5e5fb830d4db168e3ce59a32926a48ae0d] Starting server on port: 5044
logstash_1 | [2022-07-25T09:22:14,527][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1 | [2022-07-25T09:22:42,162][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/_xpack'"}
logstash_1 | [2022-07-25T09:22:42,437][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}
^CGracefully stopping... (press Ctrl+C again to force)
Stopping logstash_logstash_1 ... done
我想这与许可有关？我已经做了很多研究，但还没有发现任何真正对我有帮助的东西。我只希望 Logstash 能够使用基本用户/密码登录进行身份验证。
有什么想法吗？致以诚挚的问候。
安装elasticsearch时，会为传输层和HTTP层生成证书和密钥。
如果您遵循此文档：https://www.elastic.co/guide/en/elasticsearch/reference/8.5/docker.html
那里有一个 bash 命令。它从 ca 文件创建 .crt 和 .key 文件。自带elasticsearch，用于节点间通信
那么您应该按照说明使用此部件：
xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es01/es01.key
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/es01/es01.key
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
这意味着节点间连接现在是安全的。 对于logstash，您应该使用
output {
elasticsearch {
...
ssl => true
cacert => '/path/to/cert.crt'
}
}