我希望有人能帮助我，因为我对 kinesis firehose 和 firehose agent 比较陌生。
我已经为我的内部 debian 服务器和 ec2 debian 实例编译了 kinesis-agent (在一个测试的 aws 账户中)。在一个单独的 aws 帐户中，我创建了 Kinesis Stream，并把它指向 AWS elasticsearch 域 (Monitoring AWS ACCOUNT)。
我在AWS监控账户中创建了一个用户(kinesistestagent)，这个用户可以访问kinesis firehose流，并添加了正确的STS角色(我被这个卡住了好几天，因为它甚至不会认证)，这样kinesis代理就可以通过firehose流认证并发送数据。
我的 kinesis 的 agent.json 文件是这样的。我试着把它缩小到只想把数据传到firehose和elasticsearch里......。
{
"checkpointFile": "/opt/aws-kinesis-agent/run/checkpoints",
"cloudwatch.endpoint": "https://monitoring.eu-west-2.amazonaws.com",
"cloudwatch.emitMetrics": "false",
"firehose.endpoint": "https://firehose.eu-west-2.amazonaws.com",
"assumeRoleExternalId" :"arn:aws:firehose:eu-west-2:117215238277453:deliverystream/TEST-Firehose-EKK",
"awsAccessKeyId": "AKIRADXQWUX45KCM2IKB",
"awsSecretAccessKey": "bpq7KdidkfkeodmadeuppaccessZg4BL",
"flows": [
{
"filePattern": "/data/log/server.log",
"initialPosition": "END_OF_FILE",
"deliveryStream": "TEST-Firehose-EKK"
}
]
}
由于我的linux实例不是Amazon AMI，我已经明确使用了 "awsAccessKeyId "和 "awsSecretAccessKey "的授权值。
我从日志中得到的确切错误是，认证工作正常，但请求中的安全令牌无效？
2020-03-26 23:00:00.088+0000 (sender-2318) com.amazon.kinesis.streaming.agent.tailing.AsyncPublisher [ERROR] AsyncPublisher[fh:TEST-Firehose-EKK:/data/log/server.log]:RecordBuffer(id=2,records=179,bytes=36161) Retriable send error (com.amazonaws.services.kinesisfirehose.model.AmazonKinesisFirehoseException: The security token included in the request is invalid. (Service: AmazonKinesisFirehose; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: db4f7c53-e4a0-17e0-8db8-4a638f74b5fb)). Will retry.
整个事务在kinesis日志中的样子是这样的。
2020-03-26 22:59:59.574+0000 (FileTailer[fh:TEST-Firehose-EKK:/data/log/server.log].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:TEST-Firehose-EKK:/data/log/server.log]: Tailer Progress: Tailer has parsed 179 records (997399 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2020-03-26 22:59:59.581+0000 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 179 records parsed (997399 bytes), and 0 records sent successfully to destinations. Uptime: 23790134ms
2020-03-26 23:00:00.058+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] connecting to firehose.eu-west-2.amazonaws.com/52.94.49.83:443
2020-03-26 23:00:00.059+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Connecting socket to firehose.eu-west-2.amazonaws.com/52.94.49.83:443 with timeout 10000
2020-03-26 23:00:00.060+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Enabled protocols: [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1]
2020-03-26 23:00:00.060+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Enabled cipher suites:[TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2020-03-26 23:00:00.061+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] socket.getSupportedProtocols(): [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello], socket.getEnabledProtocols(): [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1]
2020-03-26 23:00:00.061+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1, TLSv1.3]
2020-03-26 23:00:00.061+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Starting handshake
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] Secure session established
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] negotiated protocol: TLSv1.2
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] peer principal: CN=firehose.eu-west-2.amazonaws.com
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] peer alternative names: [*.firehose.eu-west-2.vpce.amazonaws.com, firehose.eu-west-2.amazonaws.com]
2020-03-26 23:00:00.070+0000 (sender-2318) com.amazonaws.http.conn.ssl.SdkTLSSocketFactory [DEBUG] issuer principal: CN=Amazon, OU=Server CA 1B, O=Amazon, C=US
2020-03-26 23:00:00.088+0000 (sender-2318) com.amazon.kinesis.streaming.agent.tailing.AsyncPublisher [ERROR] AsyncPublisher[fh:TEST-Firehose-fEKK:/data/log/server.log]:RecordBuffer(id=2,records=179,bytes=36161) Retriable send error (com.amazonaws.services.kinesisfirehose.model.AmazonKinesisFirehoseException: The security token included in the request is invalid. (Service: AmazonKinesisFirehose; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: db4f7c53-e4a0-17e0-8db8-4a638f74b5fb)). Will retry.
有没有人以前遇到过这个错误，或者有没有人设法让AWS kinesis代理在prem服务器上工作？
非常感谢你抽出时间来阅读我的问题，任何帮助或建议都将是非常感激的。
欢呼
终于解决了我的问题...
由于我创建了Kinesis用户，可以访问流与AWSAccessKeyID和AWSSECRET，我实际上没有承担任何角色。通过采取这一行出来的一切工作。
我不得不使用一个或另一个，而不是两者。
我希望有人能帮助我，因为我对kinesis firehose和firehose代理相当陌生。
我已经为我的内部 debian 服务器和 ec2 debian 实例编译了 kinesis-agent (在一个测试的 aws 帐户中)。在一个单独的 aws 帐户中，我创建了 Kinesis Stream，并把它指向 AWS elasticsearch 域 (Monitoring AWS ACCOUNT)。
我在AWS监控账户中创建了一个用户(kinesistestagent)，这个用户可以访问kinesis firehose流，并添加了正确的STS角色(我被这个卡住了好几天，因为它甚至不会认证)，这样kinesis代理就可以通过firehose流认证并发送数据。
我的 kinesis 的 agent.json 文件是这样的。我试着把它简化成只想把数据传到firehose和elasticsearch里......
{
"checkpointFile": "/opt/aws-kinesis-agent/run/checkpoints",
"cloudwatch.endpoint": "https://monitoring.eu-west-2.amazonaws.com",
"cloudwatch.emitMetrics": "false",
"firehose.endpoint": "https://firehose.eu-west-2.amazonaws.com",
"awsAccessKeyId": "AKIRADXQWUSX45KCM2IKB",
"awsSecretAccessKey": "bpq7KdidfkfkemadeuppaccessZg4BL",
"flows": [
{
"filePattern": "/data/log/server.log",
"initialPosition": "END_OF_FILE",
"deliveryStream": "TEST-Firehose-EKK"
}
]
}