如何通过 OTLP 使用 OpenTelemetry Collector 将跟踪发送到 Elastic APM 服务器

Question

开放遥测收集器版本：0.82.0

Elastic APM 服务器版本：7.12.1

HAProxy 版本：2.0.31-0ubuntu0.2

APM Server 的服务端口为8080，处于非安全模式。 我在多个 APM 服务器前面配置了一个 HAProxy LB。

奇怪的是，当我将exporters.otlp.endpoint设置为APM服务器实例IP时，可以正常收集跟踪信息，但是当我将其设置为HAProxy的IP时，我收到*错误。

• 错误

Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: 2023-09-05T14:17:00.601+0900        error        exporterhelper/queued_retry.go:391        Exporting failed. The error is not retryable. Dropping data.        {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "Permanent error: error exporting items, request to http://haproxy-apm-lb:8080/v1/traces responded with HTTP Status Code 404", "dropped_items": 8}
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]:         go.opentelemetry.io/collector/[email protected]/exporterhelper/queued_retry.go:391
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]:         go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:126
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]:         go.opentelemetry.io/collector/[email protected]/exporterhelper/queued_retry.go:195
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]:         go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/bounded_memory_queue.go:47

• 配置

[OTELCOL-服务器-001]

# /etc/otelcol/config.yaml - not working, something wrong
...
exporters:
  otlphttp:
    endpoint: "http://haproxy-apm-lb:8080"
    tls:
      insecure: true
    headers:
      Content-Type: "application/json"

  logging:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging,otlphttp]   
...

# /etc/otelcol/config.yaml - working properly
...
exporters:
  otlp:
    endpoint: "http://elastic-apm-server-001:8080"
    tls:
      insecure: true
    headers:
      Content-Type: "application/json"

  logging:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging,otlp]   
...

# /etc/otelcol/otelcol.conf
OTELCOL_OPTIONS="--config=/etc/otelcol/config.yaml"
OTEL_EXPORTER_OTLP_HEADERS="Content-Type=application/json"
OTEL_EXPORTER_OTLP_PROTOCOL="http/json"

[HAPROXY-APM-LB]

# /etc/haproxy/haproxy.cfg
...
frontend http-in
        bind *:8080
        default_backend http_main

        acl acl_main path_beg /v1

        http-response set-header Content-Type application/json if acl_main
        use_backend http_main if acl_main

backend http_main
        mode http
        option httpchk GET /
        http-request add-header Content-Type application/json
        http-check expect status 200
        server elastic-apm-server-001         10.10.10.11:8080 check port 8080
        server elastic-apm-server-002         10.10.10.12:8080 check port 8080
...

我认为这是因为当导出器的请求通过 HAProxy 时标头会发生变化。

因此我尝试通过 HAProxy 前端和后端配置中的 set-header 和 add-header 设置来设置 Content-Type，但无济于事。

Answer 1

# /etc/otelcol/config.yaml - not working, something wrong
...
exporters:
  otlphttp:
...

otlphttp

配置 OTLP/HTTP 导出器。 v8.3.0 中的 Elastic APM Server 添加了对 OTLP/HTTP 的支持 - 请参阅 https://www.elastic.co/guide/en/apm/guide/current/release-notes-8.3.html#_added_7

# /etc/otelcol/config.yaml - working properly
...
exporters:
  otlp:
...

otlp

配置 OTLP/gRPC 导出器。

APM Server v7.12.1 支持 OTLP/gRPC，但不支持 OTLP/HTTP。