Logstash 配置从 S3 存储桶获取 ALB 日志

Question

我正在尝试配置 Logstash 以从 S3 存储桶获取 ALB 日志。在我的应用程序负载均衡器中，我添加了多个目标组。现在我的要求是从 ALB 日志中为每个 TargetGroup 创建单独的索引。

因此，我创建了logstash 配置并添加了ALB 日志消息的过滤器。在过滤条件中，我提到了目标组名称。如果此名称匹配，那么我将为特定日志消息添加新字段。

基于新字段，我添加了为输出配置中的每个目标组创建新索引的条件。

添加此类配置后，我从 Logstash 收到“elasticsearch - 格式错误的索引，插值后仍包含占位符”消息，并且索引也未创建。

无需添加这些过滤器，我就可以在 Logstash 中获取 ALB 日志。

我已附上 Logstash 配置和警告消息。任何人都可以帮助我解决这个问题或纠正我，如果我错了？

提前致谢..！

Logstash 配置：

input {
  s3 {
    access_key_id => "credentials"
    secret_access_key => "credentials"
    bucket => "bucket_name"
    region => "region_name"
    prefix => "ALB-logs/AWSLogs/5298/elasticloadbalancing/region_name/"
  }
}


filter {
  # Parse log lines with a grok filter
  grok {
    match => {
      "message" => '%{DATA:timestamp} %{WORD:elb_name}/%{DATA} %{IPORHOST:client_ip}:%{NUMBER:client_port} %{IPORHOST:backend_ip}:%{NUMBER:backend_port} %{NUMBER:request_processing_time} %{NUMBER:backend_processing_time} %{NUMBER:response_processing_time} %{NUMBER:elb_status_code} %{NUMBER:backend_status_code} %{NUMBER:received_bytes} %{NUMBER:sent_bytes} "%{WORD:http_method} %{DATA:request_uri} HTTP/%{NUMBER:http_version}" "%{DATA:user_agent}" %{DATA:ssl_cipher} %{DATA:ssl_protocol} %{DATA:target_group_arn} "%{DATA:trace_id}" "%{DATA:host}" "%{DATA:ssl_certificate_arn}" %{NUMBER:ssl_cipher_bits} %{DATA:timestamp} "%{DATA:action}" "%{DATA:waf_response_code}" "%{DATA:waf_message}" "%{DATA:backend_description}" "%{DATA:elb_response_code}" "%{DATA:elb_response_description}" "%{DATA:elb_target_ip}" "%{DATA:elb_target_port}"'
    }
  }
  
  # Add a field to indicate the target group
  if [target_group_arn] =~ /app1/ {
    mutate {
      add_field => { "target_group" => "app1" }
    }
  } else if [target_group_arn] =~ /frontend/ {
    mutate {
      add_field => { "target_group" => "frontend" }
    }
  } else if [target_group_arn] =~ /Backend/ {
    mutate {
      add_field => { "target_group" => "Backend" }
    }
  } else if [target_group_arn] =~ /ORM/ {
    mutate {
      add_field => { "target_group" => "ORM" }
    }
  } else if [target_group_arn] =~ /OASC/ {
    mutate {
      add_field => { "target_group" => "OASC" }
    }
  } else if [target_group_arn] =~ /security/ {
    mutate {
      add_field => { "target_group" => "security" }
    }
  } else if [target_group_arn] =~ /TOB/ {
    mutate {
      add_field => { "target_group" => "TOB" }
    }
  }
  # Add more conditions for other target groups as needed...

}

output {
  # Output to Elasticsearch with separate indices based on target group
  if ([target_group] in ["app1"]) {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-app-%{+YYYY.MM.dd}"
      # Additional configuration for Elasticsearch output
    }
  } else if ([target_group] in ["frontend", "Backend"]) {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-prod-%{+YYYY.MM.dd}"
      # Additional configuration for Elasticsearch output
    }
  } else if ([target_group] in ["ORM", "OASC"]) {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-orm-%{+YYYY.MM.dd}"
      # Additional configuration for Elasticsearch output
    }
  } else if ([target_group] in ["security"]) {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-security-%{+YYYY.MM.dd}"
      # Additional configuration for Elasticsearch output
    }
  }
  # Add more conditions for other target groups as needed...

  else {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "alb-default-%{+yyyy.MM.dd}"
      manage_template => false
    }
  }
  
  stdout { codec => rubydebug }
}

来自 Logstash 的警告消息：

[WARN ] 2024-02-29 18:35:06.186 [[main]>worker0] elasticsearch - Badly formatted index, after interpolation still contains placeholder: [%{[@metadata][beat]}-2024.02.29]; event: `{"@timestamp"=>2024-02-29T13:01:18.463720661Z, "@metadata"=>{"s3"=>{"key"=>"ALB-logs/AWSLogs/5298/elasticloadbalancing/region_name/2024/02/03/5298_elasticloadbalancing_region_name_app.Application-LB.6729648b993f37bb_20240203T0430Z_13.126.185.122_4xljf1i6.log.gz"}}, "@version"=>"1", "message"=>"https 2024-02-03T04:25:53.992906Z app/Application-LB/6729648b993f37bb 115.97.253.187:43816 192.168.11.215:8080 0.001 0.001 0.000 404 404 622 1921 \"GET https://demo.example.net:443/mytag_js.js HTTP/1.1\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\" TLS_AES_128_GCM_SHA256 TLSv1.3 arn:aws:elasticloadbalancing:region_name:5298:targetgroup/app1/50e7f17a37ab3cdd \"Root=1-65bdc051-72c953f82263264c\" \"demo.example.net\" \"arn:aws:acm:region_name:5298:certificate/14eda-6242-43dc-e-a5fc75a55b2f\" 0 2024-02-03T04:25:53.990000Z \"waf,forward\" \"-\" \"-\" \"192.168.11.215:8080\" \"404\" \"-\" \"-\"", "event"=>{"original"=>"https 2024-02-03T04:25:53.992906Z app/Application-LB/67296993f37bb 115.97.253.187:43816 192.168.11.215:8080 0.001 0.001 0.000 404 404 622 1921 \"GET https://demo.example.net:443/mytag_js.js HTTP/1.1\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\" TLS_AES_128_GCM_SHA256 TLSv1.3 arn:aws:elasticloadbalancing:region_name:5298:targetgroup/app1/50e7f17a37ab3cdd \"Root=1-65bdc051-72c953f82ad59fbd4263264c\" \"demo.example.net\" \"arn:aws:acm:region_name:5298:certificate/14ed3a7a-6242-4c-9d5e-a55a55b2f\" 0 2024-02-03T04:25:53.990000Z \"waf,forward\" \"-\" \"-\" \"192.168.11.215:8080\" \"404\" \"-\" \"-\""}, "tags"=>["_grokparsefailure"]}`

Answer 1

要修复 Logstash 配置中的“格式错误的索引”警告：

验证target_group_arn解析：确保grok过滤器正确提取target_group_arn字段。 _grokparsefailure 标签表明这里存在问题。 添加默认 target_group：在过滤器中包含 else 条件来处理与任何 target_group_arn 条件不匹配的日志，并为其分配默认 target_group 值。 检查区分大小写：确保您的过滤条件与 target_group_arn 值的大小写完全匹配，正如它们在日志中显示的那样。 确保回退索引：确保您的输出部分包含不匹配任何指定条件的日志的回退索引。 启用调试日志记录：提高日志级别以调试更详细的错误消息，从而帮助识别问题。 语法检查：仔细检查您的配置是否有任何语法错误或错误配置。 实施这些步骤应该有助于解决 Logstash 中的索引问题。