我的本地设置包括本地apache-flink(通过brew安装)和运行Kinesis服务的localstack。
我的docker-compose有
localstack:
image: localstack/localstack:0.10.7
environment:
- SERVICES=kinesis
ports:
- "4568:4568"
和我的Kinesis使用者:
kinesisConsumerConfig.setProperty(ConsumerConfigConstants.AWS_ACCESS_KEY_ID, "123");
kinesisConsumerConfig.setProperty(ConsumerConfigConstants.AWS_SECRET_ACCESS_KEY, "123");
kinesisConsumerConfig.setProperty(ConsumerConfigConstants.AWS_ENDPOINT, "http://localhost:4568");
但是当我运行flink程序时,出现此错误:
原因:org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.model.AmazonKinesisException:null(服务:AmazonKinesis;状态代码:502;错误代码:null;请求ID:null)
仅在使用localstack时发生。如果我通过aws帐户连接到我的Kinesis流,则可以正常使用。
如果使用Java,则可以使用jar
库来模拟某些亚马逊组件:
首先,您需要在pom.xml
中添加以下组件,以便能够在测试期间直接初始化本地堆栈:
<dependency>
<groupId>cloud.localstack</groupId>
<artifactId>localstack-utils</artifactId>
<version>0.2.0</version>
<scope>test</scope>
</dependency>
然后,如果需要使用kinesis
和dynamo
,则需要指定以下库,因为从aws
提供的最新库与最新版本的localstack不兼容:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.11.642</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>1.8.10</version>
<scope>test</scope>
</dependency>
现在您可以使用以下注释来使用docker实例化堆栈,如果系统中不存在这些图像,则会自动将其拉出。因此,无需运行任何docker / docker-compose映像。
@LocalstackDockerProperties(services = {"kinesis", "dynamodb"})
@ExtendWith(LocalstackDockerExtension.class)
@Slf4j
public class TestPipelineComplete {
public static final String AWS_ACCESS_KEY_ID = "foo";
public static final String AWS_SECRET_ACCESS_KEY = "bar";
static {
System.setProperty("AWS_ACCESS_KEY_ID", AWS_ACCESS_KEY_ID);
System.setProperty("AWS_SECRET_ACCESS_KEY", AWS_SECRET_ACCESS_KEY);
// These two lines are fundamental
cloud.localstack.TestUtils.setEnv("aws.cborEnabled", "false");
cloud.localstack.TestUtils.setEnv("AWS_CBOR_DISABLE", "true");
}
}
现在,如果需要初始化DynamoDB
客户端,则可以使用以下行:
final AmazonDynamoDB clientDynamoDB = cloud.localstack.TestUtils.getClientDynamoDB();
现在,如果需要初始化Kinesis
客户端,则可以使用以下行:
final AmazonKinesis kinesisClient = cloud.localstack.TestUtils.getClientKinesis();
如果需要从运动学中读取数据(测试目的),可以使用以下代码片段作为模板(https://gist.github.com/alessiosavi/4ea88d73d6853de695843631207b7bc6):
package org.example;
import com.amazonaws.services.kinesis.AmazonKinesis;
import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder;
import com.amazonaws.services.kinesis.model.*;
import java.nio.charset.StandardCharsets;
import java.util.List;
public class App {
private static final String streamName = "API_NAME" + "_kineis-notification-stream";
private static final AmazonKinesis client = AmazonKinesisClientBuilder.defaultClient();
public static void main(String[] args) {
printKinesisRecords(getRecordsFromKinesis(client));
}
private static List<Record> getRecordsFromKinesis(AmazonKinesis kClient) {
final ListShardsRequest listShardsRequest = new ListShardsRequest().withStreamName(streamName).withMaxResults(1);
Shard shard = kClient.listShards(listShardsRequest).getShards().get(0);
GetShardIteratorRequest getShardIteratorRequest = new GetShardIteratorRequest();
getShardIteratorRequest.setStreamName(streamName);
getShardIteratorRequest.setShardId(shard.getShardId());
getShardIteratorRequest.setShardIteratorType("TRIM_HORIZON");
final GetShardIteratorResult getShardIteratorResult = kClient.getShardIterator(getShardIteratorRequest);
String shardIterator = getShardIteratorResult.getShardIterator();
// Create a new getRecordsRequest with an existing shardIterator
// Set the maximum records to return to 1
GetRecordsRequest getRecordsRequest = new GetRecordsRequest();
getRecordsRequest.setShardIterator(shardIterator);
getRecordsRequest.setLimit(10);
final GetRecordsResult result = kClient.getRecords(getRecordsRequest);
// Put the result into record list. The result can be empty.
return result.getRecords();
}
private static void printKinesisRecords(List<Record> records) {
for (Record record : records) {
System.err.println("RECORD: " + StandardCharsets.UTF_8.decode(record.getData()).toString());
}
}
}
结果我们需要通过ENV var禁用cbor和证书检查,并在同一控制台中启动flink
export AWS_CBOR_DISABLE=1
DISABLE_CERT_CHECKING_JAVA_OPTS="-Dorg.apache.flink.kinesis.shaded.com.amazonaws.sdk.disableCertChecking"
export FLINK_ENV_JAVA_OPTS=${DISABLE_CERT_CHECKING_JAVA_OPTS}
/usr/local/Cellar/apache-flink/1.9.1/libexec/bin/start-cluster.sh