Amazon Kinesis：在同步Kinesis碎片和租约时捕获到异常

Question

我正在尝试使Snowplow在AWS上工作。当我尝试在实例上运行流丰富的服务时，出现此异常：

[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Syncing Kinesis shard info
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask - Caught exception while sync'ing Kinesis shards and leases
[cw-metrics-publisher] WARN com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable - Could not publish 4 datums to CloudWatch

我不认为错误归因于Cloud Watch：

同步Kinesis分片和租约时捕获到异常

Answer 1

如上面的评论中所述，当您缺少Kinesis Client Library (KCL)所需的AWS资源权限时，将出现此错误。这可以是DynamoDB，CloudWatch或Kinesis。对于Snowplow的Stream Enrich组件，您需要以下权限：

• 读取输入运动学流的权限（收集器良好）
• 写入输出运动学流的许可（充实和充实不良）
• 列出运动流的权限
• 对DynamoDB状态表的读取/写入/创建权限（表名是流丰富application.conf中的“ appName”值）
• PutMetricData到Cloudwatch

满足这些需求的IAM策略的模板版本如下：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:ListShards"
      ],
      "Resource": [
        "${collector_stream_out_good}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
          "kinesis:ListStreams"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": [
        "${enricher_stream_out_good}",
        "${enricher_stream_out_bad}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:CreateTable",
        "dynamodb:DescribeTable",
        "dynamodb:Scan",
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": [
        "${enricher_state_table}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Resource": "*"
    }
  ]
}

我已经写了一个blog post，其中涵盖了Stream Enrich和其他Snowplow组件的必需IAM权限，因为exact必需权限的文档在Snowplow文档中很少/不存在。

希望有帮助！