如何在Kubernetes集群上集成opensearch和logstash?

问题描述 投票:0回答:1

我有一个 kubernetes 集群,我已经在上面部署了一个 opensearch 集群和 opensearch 仪表板作为我的目标,因为我也找不到太多关于它的文档。任何帮助表示赞赏....提前致谢!

也使用 Helm 和 logstash 部署了 opensearch 但无法集成它们

在这里更新!!!

进行了一些更改以简化部署并更好地控制功能,

我这次是在测试部署和服务文件,我会在下面添加文件

打开搜索部署文件


---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: logging
  name: opensearch
  labels:
    component: opensearch
spec:
  selector:
    matchLabels:
      component: opensearch
  replicas: 1
  serviceName: opensearch
  template:
    metadata:
      labels:
        component: opensearch
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox
          imagePullPolicy: IfNotPresent
          command:
            - sysctl
            - -w
            - vm.max_map_count=262144
          securityContext:
            privileged: true
      containers:
        - name: opensearch
          securityContext:
            capabilities:
              add:
                - IPC_LOCK
          image: opensearchproject/opensearch
          env:
            - name: KUBERNETES_CA_CERTIFICATE_FILE
              value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: "cluster.name"
              value: "opensearch-cluster"
            - name: "network.host"
              value: "0.0.0.0"
            - name: "discovery.seed_hosts"
              value: "[]"
            - name: discovery.type
              value: single-node              
            - name: OPENSEARCH_JAVA_OPTS
              value: -Xmx512M -Xms512M
            - name: "plugins.security.disabled"
              value: "false" 
          ports:
            - containerPort: 9200
              name: http
              protocol: TCP
            - containerPort: 9300
              name: transport
              protocol: TCP
          volumeMounts:
            - name: os-mount
              mountPath: /data
      volumes:
        - name: os-mount
          persistentVolumeClaim:
            claimName: nfs-pvc-os-logging


Opensearch svc 文件

---
apiVersion: v1
kind: Service
metadata:
  name: opensearch
  namespace: logging
  labels:
    service: opensearch
spec:
  type: ClusterIP
  selector:
    component: opensearch
  ports:
  - port: 9200
    targetPort: 9200

Opensearch 仪表板部署

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-dash
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: open-dash
  template:
    metadata:
      labels:
        app: open-dash
    spec:
#      securityContext:
#      runAsUser: 0
            
      containers:
        - name: opensearch-dashboard
          image: opensearchproject/opensearch-dashboards:latest
          ports:
            - containerPort: 80
          env:
#            - name: ELASTICSEARCH_URL
#              value: https://opensearch.logging:9200
#            - name: "SERVER_HOST"
#              value: "localhost"
#            - name: "opensearch.hosts" 
#              value: https://opensearch.logging:9200
            - name: OPENSEARCH_HOSTS
              value:  '["https://opensearch.logging:9200"]'  


Opensearch 仪表板 svc

---
apiVersion: v1
kind: Service
metadata:
  name: opensearch
  namespace: logging
  labels:
    service: opensearch
spec:
  type: ClusterIP
  selector:
    component: opensearch
  ports:
  - port: 9200
    targetPort: 9200


通过上述配置,我可以打开 Dashboard UI,但在 Dashboard pod 日志中我可以看到 400 条代码日志,任何人都可以尝试重现此问题,另外我需要将 logstash 与此堆栈集成。

{“类型”:“响应”,“@timestamp”:“2023-02-20T05:05:34Z”,“标签”:[],“pid”:1,“方法”:“head”,“statusCode ":400,"req":{"url":"/app/home","method":"head","headers":{"connection":"Keep-Alive","content-type":" application/json","host":"3.108.199.0:30406","user-agent":"Manticore 0.9.1","accept-encoding":"gzip,deflate","securitytenant":"user "},"remoteAddress":"10.244.1.1","userAgent":"Manticore 0.9.1"},"res":{"statusCode":400,"responseTime":2,"contentLength":9},"消息”:“HEAD /app/home 400 2ms - 9.0B

部署 logstash pod 时,我得到一个错误

[警告] 2023-02-20 05:13:52.212 [Ruby-0-Thread-9:/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0。 1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - 试图恢复与死 OpenSearch 实例的连接,但出现错误 {:url=>"http://logstash:xxxxxx@opensearch .logging:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://logstash:[email protected]:9200 /][Manticore::ClientProtocolException] opensearch.logging:9200 响应失败”}

有人可以帮我解决这个难题吗

@Benla 已根据您对图像的建议进行了更改,现在我在 logstash 中收到以下日志

[2023-02-20T05:18:43,028][信息][logstash.agent] 成功启动 Logstash API 端点 {:port=>9600, :ssl_enabled=>false} [2023-02-20T05:18:43,147][INFO][org.reflections.Reflections] 反射花费 70 毫秒扫描 1 个 url,产生 127 个键和 444 个值 [2023-02-20T05:18:43,351][INFO][logstash.javapipeline] 管道

main
配置有
pipeline.ecs_compatibility: v8
设置。除非明确配置,否则此管道中的所有插件都将默认为
ecs_compatibility => v8
。 [2023-02-20T05:18:43,370][INFO][logstash.javapipeline][main] 启动管道 {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"= >125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread= >“#”} [2023-02-20T05:18:43,811][INFO][logstash.javapipeline][main] 管道 Java 执行初始化时间 {"seconds"=>0.44} [2023-02-20T05:18:43,816][INFO][logstash.inputs.beats][main] 启动输入侦听器 {:address=>"0.0.0.0:5044"} [2023-02-20T05:18:43,821][INFO][logstash.javapipeline][main] 管道启动 {"pipeline.id"=>"main"} [2023-02-20T05:18:43,835][INFO][logstash.agent] 管道运行 {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2023-02-20T05:18:43,869][INFO][org.logstash.beats.Server][main][0710cad67e8f47667bc7612580d5b91f691dd8262a4187d9eca8cf87229d04aa] 在端口上启动服务器:5044

kubernetes logstash opensearch opensearch-dashboards
1个回答
0
投票

我认为部署

logstash
<->
opensearch
管道与在
K8S
中部署<->logstash
elasticsearch没有太大区别,
我遇到的唯一区别是:

  1. docker 或其他容器运行时镜像应该是
    opensearchproject/logstash-oss-with-opensearch-output-plugin:latest
    而不是
    logstash
    .
  2. logstash
    配置文件中的输出插件应该是
    opensearch
    而不是
    elasticsearch
    .

除了K8S教程中的每一个

Logstash
<->
ElasticSearch
你都可以去的以外,还有很多

祝你好运!

© www.soinside.com 2019 - 2024. All rights reserved.