logstash opensearch oss 输出插件与 opensearch 集成

问题描述 投票：0回答：1

我正在尝试进行此logstash 和opensearch 设置。这是我正在使用的 docker-compose 文件的详细信息。

opensearch 和 dasboard 服务 docker-compose 文件。

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:2.3.0
    #image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    #image: opensearchproject/opensearch:latest
    image: opensearchproject/opensearch:2.3.0
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.3.0
    #image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

logstash oss

version: '2.1'
services:
logstash:
  #image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.3
  image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2
  ports:
    - "5044:5044"
  volumes:
    - $PWD/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
  networks:
    - opensearch-net
networks:
opensearch-net:

这是logstash.conf 文件。

input { 
  stdin { } 
}

filter {

}

output {
  opensearch {
    hosts => ["https://opensearch_fqdn:9200/"]
    index => "testindexing"
    user => "admin"
    password => "admin"
    ssl => true
    ssl_certificate_verification => false
   }
}

我期望使用发送到标准输入终端的数据创建测试索引。但我收到以下错误。

错误

logstash_1  | [2022-10-25T07:44:39,946][ERROR][logstash.outputs.opensearch][main] Failed to install template {:message=>"Failed to load default template for OpenSearch v2 with ECS disabled; caused by: #<ArgumentError: Template file '/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/templates/ecs-disabled/2x.json' could not be found>", :exception=>RuntimeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:33:in `load_default_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:21:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:412:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:247:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:224:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}

根据以下问题详细信息 - opensearch-project/opensearch-devops#85，我尝试使用 opensearchproject/logstash-oss-with-opensearch-output-plugin:8.4.0 作为logstash oss输出插件，但logstash终止没有太多信息。

logstash_1  | [2022-10-25T12:03:11,405][INFO ][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["https://opensearch_fqdn:9200/"]}
logstash_1  | [2022-10-25T12:03:11,435][WARN ][logstash.outputs.opensearch][main] ** WARNING ** Detected UNSAFE options in opensearch output configuration!
logstash_1  | ** WARNING ** You have enabled encryption but DISABLED certificate verification.
logstash_1  | ** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
logstash_1  | [2022-10-25T12:03:11,691][INFO ][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://admin:xxxxxx@opensearch_fqdn:9200/]}}
logstash_1  | [2022-10-25T12:03:11,903][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://admin:xxxxxx@opensearch_fqdn:9200/"}
logstash_1  | [2022-10-25T12:03:11,956][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.3.0) {:version=>2}
logstash_1  | [2022-10-25T12:03:12,039][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:v8}
logstash_1  | [2022-10-25T12:03:12,058][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x50914657 run>"}
logstash_1  | [2022-10-25T12:03:12,571][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.51}
logstash_1  | [2022-10-25T12:03:12,622][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1  | [2022-10-25T12:03:12,692][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1  | [2022-10-25T12:03:13,008][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
logstash_1  | [2022-10-25T12:03:13,262][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
logstash_1  | [2022-10-25T12:03:13,360][INFO ][logstash.runner          ] Logstash shut down.

任何我在这里缺少的建议。

logstash opensearch
1个回答
0
投票

你可以尝试 cp /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/templates/ecs-disabled /1x.json /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/templates/ecs-disabled/2x .json （与你的路径）

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.