我目前正在构建一个数据管道，该管道使用在由 Terraform 管理的 ecs 集群中运行的向量代理来摄取数据。此外，docker 映像从 s3 存储桶读取其配置文件。
因此，现在矢量代理在 ecs 服务中完美运行，该服务将数据从 SNS 主题拉入 s3 存储桶。
这是它的地形代码：
resource "aws_ecs_task_definition" "s3_task_def" {
family = "vector-s3-task"
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
cpu = "256"
memory = "512"
execution_role_arn = aws_iam_role.logging_execution_role.arn
task_role_arn = aws_iam_role.logging_role.arn
container_definitions = jsonencode([
{
"name": "infosec-vector-container",
"image": "${aws_ecr_repository.repository.repository_url}:latest",
"essential": true,
"portMappings": [
{
"containerPort": 8686,
"hostPort": 8686
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": aws_cloudwatch_log_group.log_group.name,
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": aws_cloudwatch_log_stream.log_stream.name
}
},
"environment": [
{
"name": "VECTOR_FILE",
"value": var.vector
}
]
}
])
}
# creates an ECS service within the ECS cluster for s3 bucket
resource "aws_ecs_service" "service" {
name = "vector-s3-service"
cluster = aws_ecs_cluster.cluster.id
task_definition = aws_ecs_task_definition.s3_task_def.arn
desired_count = var.desired_count
launch_type = "FARGATE"
network_configuration {
security_groups = [aws_security_group.sg.id]
subnets = [aws_subnet.private_subnet.id]
}
}
但是，当我添加通过 API 提取数据的 ecs 服务时，我开始在 ecs 日志中接收奇怪的日志，并且从 API 提取的日志不会发送到 s3 存储桶。
这是 API ecs 服务 terraform 代码：
resource "aws_ecs_task_definition" "api_task_def" {
family = "vector-api-task"
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
cpu = "512"
memory = "1024"
execution_role_arn = aws_iam_role.logging_execution_role.arn
task_role_arn = aws_iam_role.logging_role.arn
container_definitions = jsonencode([
{
"name": "infosec-vector-container",
"image": "${aws_ecr_repository.repository.repository_url}:latest",
"essential": true,
"portMappings": [
{
"containerPort": 8686,
"hostPort": 8686
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": aws_cloudwatch_log_group.log_group.name,
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": aws_cloudwatch_log_stream.log_stream.name
}
},
"environment": [
{
"name": "VECTOR_FILE",
"value": var.vector_api
},
{
"name": "SLACK_KEY",
"value": data.aws_secretsmanager_secret_version.slack_secret.secret_string
}
]
}
])
}
# creates an ECS service within the ECS cluster for s3api polling
resource "aws_ecs_service" "api_service" {
name = "vector-api-service"
cluster = aws_ecs_cluster.cluster.id
task_definition = aws_ecs_task_definition.api_task_def.arn
desired_count = 1
launch_type = "FARGATE"
network_configuration {
security_groups = [aws_security_group.sg.id]
subnets = [aws_subnet.private_subnet.id]
}
}
这也是我的 api 轮询配置的 vector.toml 文件
###################################################################################################################
### SLACK AUDIT ###
###################################################################################################################
[sources.slack_audit]
type = "http_client"
endpoint = "https://api.slack.com/audit/v1/logs"
method = "GET"
scrape_interval_secs = 900
auth.strategy = "bearer"
auth.token = "${SLACK_KEY}"
[sources.slack_audit.headers]
Accept = ["application/json"]
[transforms.slack_audit_output]
type = "remap"
inputs = ["slack_audit"]
source = '''
.source = "slack_audit"
.vtime = now()
.data = parse_json!(.message)
del(.message)
'''
###################################################################################################################
### SLACK DATA SOURCE ###
###################################################################################################################
[sinks.infosec_log_prod]
type = "aws_s3"
inputs = ["*_output"]
bucket = "EXAMPLE"
key_prefix = "application={{ source }}/env=prod/year=%Y/month=%m/day=%d/"
region = "us-west-2"
compression = "gzip"
filename_extension = "json"
encoding.codec = "json"
encoding.timestamp_format = "rfc3339"
这些是我在 ecs 服务/任务日志中收到的日志，我不明白为什么。
{"appname":"shaneIxD","facility":"local2","hostname":"random.org","message":"A bug was encountered but not in Vector, which doesn't have bugs","msgid":"ID258","procid":240,"severity":"warning","timestamp":"2023-07-20T21:39:06.581Z","version":1}
{"appname":"meln1ks","facility":"syslog","hostname":"make.com","message":"We're gonna need a bigger boat","msgid":"ID452","procid":4132,"severity":"info","timestamp":"2023-07-20T21:39:07.580Z","version":1}
{"appname":"devankoshal","facility":"local5","hostname":"we.de","message":"Great Scott! We're never gonna reach 88 mph with the flux capacitor in its current state!","msgid":"ID517","procid":8273,"severity":"notice","timestamp":"2023-07-20T21:39:08.580Z","version":1}
{"appname":"ahmadajmi","facility":"user","hostname":"names.com","message":"A bug was encountered but not in Vector, which doesn't have bugs","msgid":"ID236","procid":6192,"severity":"notice","timestamp":"2023-07-20T21:39:09.580Z","version":2}
{"appname":"devankoshal","facility":"local7","hostname":"make.us","message":"There's a breach in the warp core, captain","msgid":"ID172","procid":5465,"severity":"info","timestamp":"2023-07-20T21:39:10.580Z","version":1}
{"appname":"devankoshal","facility":"news","hostname":"make.de","message":"A bug was encountered but not in Vector, which doesn't have bugs","msgid":"ID916","procid":5888,"severity":"emerg","timestamp":"2023-07-20T21:39:11.580Z","version":1}
每次重新启动服务时，我都会得到完全相同的日志。当我删除 API ECS 服务时，我的 S3 ECS 服务工作正常。有人遇到过这种情况吗？
我已经测试了配置文件，但没有将数据发送到 s3 存储桶并发送到我的终端，它按预期工作。
我尝试删除 s3 ecs 服务以查看这是否是问题所在，但我仍然收到相同的错误。
我尝试删除 API s3 ecs 服务，一切正常。
事实证明，当我应该创建另一个 cloudwatch 服务时，我为每个服务使用了相同的 cloudwatch 日志