如何将运行在GCP之上的Kubernetes上的pod中的日志发送到elasticsearch / logstash?

问题描述 投票:6回答:4

我在Google-Container-Engine中运行我系统的新模块。我想将stdout和stderr从他们(在pods中运行)带到我的集中式logstash。有没有一种简单的方法可以将日志从pod转发到外部日志服务,例如logstash或elasticsearch?

kubernetes google-cloud-platform fluent google-kubernetes-engine
4个回答
5
投票

我决定直接登录elasticsearch,这是一个可以在elasticsearch.c.my-project.internal访问的外部虚拟机(我在Google-Cloud-Platform上)。这很容易:

  1. 使用名称设置ExternalService:elasticsearch指向elasticsearch实例: apiVersion: v1 kind: Service metadata: name: elasticsearch-logging namespace: kube-system labels: k8s-app: elasticsearch kubernetes.io/name: "elasticsearch" spec: type: ExternalName externalName: elasticsearch.c.my-project.internal ports: - port: 9200 targetPort: 9200
  2. 将一个流畅的弹性搜索部署为DeamonSet。 fluentd-elasticsearch将自动连接到名为elasticsearch-logging的服务(基于fluentd-elasticsearch deployment defintionapiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: tier: monitoring app: fluentd-logging k8s-app: fluentd-logging spec: template: metadata: labels: name: fluentd-elasticsearch spec: containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.19 volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers 使用kubectl logs fluentd-elasticsearch-...检查您是否能够连接到elasticsearach实例。
  3. 现在,您可以访问kibana并查看日志。

3
投票

你可以在stack-driverpub-sub的日志中创建一个接收器,然后使用logstash-input-google_pubsub插件 - 使用logstash-input-google_pubsub image将所有日志导出为弹性,请参阅source code

将日志导出到pub-sub

  1. 在pubsub中创建主题和订阅按照说明here
  2. 在日志查看器page中单击create export,确保过滤到应用程序的日志(GKE容器 - >集群名称,应用程序名称),输入接收器名称,选择Cloud Pubsub作为接收服务,现在在接收器中选择您的主题目的地。

从现在开始和之后的日志都会导出到pub-sub

配置logstash管道

这是pubsub-elastic.conf文件:

input {
    google_pubsub {
        project_id => "my-gcloud-project-id"
        topic => "elastic-pubsub-test"
        subscription => "elastic-pubsub-test"
        json_key_file => "/etc/logstash/gcloud-service-account-key.json"
    }
}


output {
    elasticsearch {
        hosts => "https://example.us-east-1.aws.found.io:9243"
        user => "elastic"
        password => "mypassword"
    }
}

这是我的Docker文件:

FROM sphereio/logstash-input-google_pubsub


# Logstash config
COPY gcloud-service-account-key.json /etc/logstash/gcloud-service-account-key.json
COPY config /etc/logstash/conf.d
COPY logstash.yml /etc/logstash/logstash.yml

现在你应该建立图像并运行

如果在kubernetes上运行,请使用以下命令:

这是deployment.yaml

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: logstash-input-google-pubsub spec: replicas: 1 strategy: type: RollingUpdate template: metadata: labels: app: logstash-input-google-pubsub spec: containers: - name: logstash-input-google-pubsub image: us.gcr.io/my-gcloud-project-id/logstash-input-google_pubsub:1.0.0

构建您的图像并推送到注册表

docker build --rm -t us.gcr.io/my-gcloud-project-id/logstash-input-google_pubsub:1.0.0 . 
gcloud docker -- push us.gcr.io/my-gcloud-project-id/logstash-input-google_pubsub:1.0.0

现在创建实例kubectl create -f deployment.yaml

完成!


2
投票

因为elasticsearch 6.00你可以使用filebeats

blog

Download Filebeat DaemonSet manifest

curl -L -O https://raw.githubusercontent.com/elastic/beats/6.0/deploy/kubernetes/filebeat-kubernetes.yaml

Update Elasticsearch connection details

- name: ELASTICSEARCH_HOST
 value: elasticsearch
- name: ELASTICSEARCH_PORT
 value: "9200"
- name: ELASTICSEARCH_USERNAME
 value: elastic
- name: ELASTICSEARCH_PASSWORD
 value: changeme

Deploy it to Kubernetes

kubectl create -f filebeat-kubernetes.yaml

0
投票

您可以尝试安装以下kubernetes插件:https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

我自己没试过,但我也在寻找合适的伐木方法。 GCE日志记录在某种程度上仅限于我的观点。

© www.soinside.com 2019 - 2024. All rights reserved.