数量指标未在 longhorn 管理器的 /metrics 端点上公开
Longhorn version:1.1.2 or 1.1.1
Kubernetes version: 1.19.9-gke.1900
Node config
OS type and version: Ubuntu with Docker
Disk type : Standard persistent disk 100GB
Underlying Infrastructure : (GKE)
我有一个带有 ubuntu 和 gke 版本 1.19.9-gke.1900 的标准 GKE 集群
我已经使用 kubectl 安装了 longhorn
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.1.1/deploy/longhorn.yaml
我之前尝试过1.1.2，也遇到了同样的问题。 如果我登录到实例管理器 pod 并在 /metrics 端点上运行curl
kubectl -n longhorn-system exec -it longhorn-manager-9d797 -- curl longhorn-manager-9d797:9500/metrics
我得到了这个舞会输出
# HELP longhorn_disk_capacity_bytes The storage capacity of this disk
# TYPE longhorn_disk_capacity_bytes gauge
longhorn_disk_capacity_bytes{disk="default-disk-4cd3831f07717134",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 1.0388023296e+11
# HELP longhorn_disk_reservation_bytes The reserved storage for other applications and system on this disk
# TYPE longhorn_disk_reservation_bytes gauge
longhorn_disk_reservation_bytes{disk="default-disk-4cd3831f07717134",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 3.1164069888e+10
# HELP longhorn_disk_usage_bytes The used storage of this disk
# TYPE longhorn_disk_usage_bytes gauge
longhorn_disk_usage_bytes{disk="default-disk-4cd3831f07717134",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 5.855387648e+09
# HELP longhorn_instance_manager_cpu_requests_millicpu Requested CPU resources in kubernetes of this Longhorn instance manager
# TYPE longhorn_instance_manager_cpu_requests_millicpu gauge
longhorn_instance_manager_cpu_requests_millicpu{instance_manager="instance-manager-e-523d6b01",instance_manager_type="engine",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 113
longhorn_instance_manager_cpu_requests_millicpu{instance_manager="instance-manager-r-9d8f7ae9",instance_manager_type="replica",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 113
# HELP longhorn_instance_manager_cpu_usage_millicpu The cpu usage of this longhorn instance manager
# TYPE longhorn_instance_manager_cpu_usage_millicpu gauge
longhorn_instance_manager_cpu_usage_millicpu{instance_manager="instance-manager-e-523d6b01",instance_manager_type="engine",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 4
longhorn_instance_manager_cpu_usage_millicpu{instance_manager="instance-manager-r-9d8f7ae9",instance_manager_type="replica",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 4
# HELP longhorn_instance_manager_memory_requests_bytes Requested memory in Kubernetes of this longhorn instance manager
# TYPE longhorn_instance_manager_memory_requests_bytes gauge
longhorn_instance_manager_memory_requests_bytes{instance_manager="instance-manager-e-523d6b01",instance_manager_type="engine",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 0
longhorn_instance_manager_memory_requests_bytes{instance_manager="instance-manager-r-9d8f7ae9",instance_manager_type="replica",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 0
# HELP longhorn_instance_manager_memory_usage_bytes The memory usage of this longhorn instance manager
# TYPE longhorn_instance_manager_memory_usage_bytes gauge
longhorn_instance_manager_memory_usage_bytes{instance_manager="instance-manager-e-523d6b01",instance_manager_type="engine",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 7.29088e+06
longhorn_instance_manager_memory_usage_bytes{instance_manager="instance-manager-r-9d8f7ae9",instance_manager_type="replica",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 1.480704e+07
# HELP longhorn_manager_cpu_usage_millicpu The cpu usage of this longhorn manager
# TYPE longhorn_manager_cpu_usage_millicpu gauge
longhorn_manager_cpu_usage_millicpu{manager="longhorn-manager-9d797",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 13
# HELP longhorn_manager_memory_usage_bytes The memory usage of this longhorn manager
# TYPE longhorn_manager_memory_usage_bytes gauge
longhorn_manager_memory_usage_bytes{manager="longhorn-manager-9d797",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 2.9876224e+07
# HELP longhorn_node_count_total Total number of nodes
# TYPE longhorn_node_count_total gauge
longhorn_node_count_total 3
# HELP longhorn_node_cpu_capacity_millicpu The maximum allocatable cpu on this node
# TYPE longhorn_node_cpu_capacity_millicpu gauge
longhorn_node_cpu_capacity_millicpu{node="gke-longhorn-2-default-pool-277a6687-tjgl"} 940
# HELP longhorn_node_cpu_usage_millicpu The cpu usage on this node
# TYPE longhorn_node_cpu_usage_millicpu gauge
longhorn_node_cpu_usage_millicpu{node="gke-longhorn-2-default-pool-277a6687-tjgl"} 256
# HELP longhorn_node_memory_capacity_bytes The maximum allocatable memory on this node
# TYPE longhorn_node_memory_capacity_bytes gauge
longhorn_node_memory_capacity_bytes{node="gke-longhorn-2-default-pool-277a6687-tjgl"} 2.950684672e+09
# HELP longhorn_node_memory_usage_bytes The memory usage on this node
# TYPE longhorn_node_memory_usage_bytes gauge
longhorn_node_memory_usage_bytes{node="gke-longhorn-2-default-pool-277a6687-tjgl"} 1.22036224e+09
# HELP longhorn_node_status Status of this node
# TYPE longhorn_node_status gauge
longhorn_node_status{condition="allowScheduling",condition_reason="",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 1
longhorn_node_status{condition="mountpropagation",condition_reason="",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 1
longhorn_node_status{condition="ready",condition_reason="",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 1
longhorn_node_status{condition="schedulable",condition_reason="",node="gke-longhorn-2-default-pool-277a6687-tjgl"} 1
# HELP longhorn_node_storage_capacity_bytes The storage capacity of this node
# TYPE longhorn_node_storage_capacity_bytes gauge
longhorn_node_storage_capacity_bytes{node="gke-longhorn-2-default-pool-277a6687-tjgl"} 1.0388023296e+11
# HELP longhorn_node_storage_reservation_bytes The reserved storage for other applications and system on this node
# TYPE longhorn_node_storage_reservation_bytes gauge
longhorn_node_storage_reservation_bytes{node="gke-longhorn-2-default-pool-277a6687-tjgl"} 3.1164069888e+10
# HELP longhorn_node_storage_usage_bytes The used storage of this node
# TYPE longhorn_node_storage_usage_bytes gauge
longhorn_node_storage_usage_bytes{node="gke-longhorn-2-default-pool-277a6687-tjgl"} 5.855387648e+09
我已经使用 PV 创建了一个示例 mysql pod，我可以看到它由 longhorn 进行配置和管理，并在集群上的所有 3 个节点上都有副本。但是我没有看到这些指标 https://longhorn.io/docs/1.1.0/monitoring/metrics/#volume
我在这里缺少什么？任何帮助表示赞赏
我能够弄清楚这一点。显然，这些指标仅从一个管理器实例公开，而不是全部。
对于通过谷歌发现此问题的任何人：
每个 longhorn-manager pod 仅公开有关在同一节点上运行的卷的卷指标。因此，您需要配置您的 prometheus scrape_configs，以便扫描所有 longhorn-manager pod。
普罗米修斯操作员应该处理这个问题，但是对于手动抓取，您可以使用类似的东西
- job_name: 'longhorn'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_name, __meta_kubernetes_pod_container_port_number]
action: keep
regex: 'longhorn-manager;9500'