我正在使用https://doc.akka.io/docs/alpakka-kafka/current/consumer.html来使用来自kafka的数据,如下所示:
implicit val system: ActorSystem = ActorSystem("SAPEVENTBUS")
implicit val materializer: Materializer = ActorMaterializer()
val config = system.settings.config.getConfig("akka.kafka.consumer")
val consumerSettings =
ConsumerSettings(config, new StringDeserializer, new StringDeserializer)
.withBootstrapServers("localhost:9092")
.withGroupId("SAP-BUS")
.withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
.withProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true")
.withProperty(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "5000")
val kafkaConsumer = Consumer
.plainSource(
consumerSettings,
Subscriptions.topics("SAPEVENTBUS"))
.toMat(Sink.foreach(println))(Keep.both)
.mapMaterializedValue(DrainingControl.apply)
接下来,我将通过akka http websocket client将收到的结果转发给网络服务器
如何构建websocket客户端:
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
import system.dispatcher
// print each incoming strict text message
val printSink: Sink[Message, Future[Done]] =
Sink.foreach {
case message: TextMessage.Strict =>
println(message.text)
}
val helloSource: Source[Message, NotUsed] =
Source.single(TextMessage("hello world!"))
// the Future[Done] is the materialized value of Sink.foreach
// and it is completed when the stream completes
val flow: Flow[Message, Message, Future[Done]] =
Flow.fromSinkAndSourceMat(printSink, helloSource)(Keep.left)
// upgradeResponse is a Future[WebSocketUpgradeResponse] that
// completes or fails when the connection succeeds or fails
// and closed is a Future[Done] representing the stream completion from above
val (upgradeResponse, closed) =
Http().singleWebSocketRequest(WebSocketRequest("ws://echo.websocket.org"), flow)
val connected = upgradeResponse.map { upgrade =>
// just like a regular http request we can access response status which is available via upgrade.response.status
// status code 101 (Switching Protocols) indicates that server support WebSockets
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
Done
} else {
throw new RuntimeException(s"Connection failed: ${upgrade.response.status}")
}
}
我有两个问题:
如何建立这样的图表?
如果您计划在没有响应处理的情况下将所有Kafka消息推送到Web套接字,则应编写Web套接字的消息处理程序in a true bi-directional scenario where input and output aren’t logically connected:
//Kafka reading logic
val kafkaSource: Source[ConsumerRecord[String, String], Consumer.Control] = Consumer
.plainSource(consumerSettings, Subscriptions.topics("SAPEVENTBUS"))
//kafka message serialization logic
val kafkaRecordToMessageTransform: Flow[ConsumerRecord[String, String], Message, NotUsed] =
Flow[ConsumerRecord[String, String]].map[Message](consumerRecord => {
TextMessage.Strict(s"${consumerRecord.key} - ${consumerRecord.value}")
})
//web socket's messages sending logic
val webSocketWriteLogic: Source[Message, Consumer.Control] =
kafkaSource.via(kafkaRecordToMessageTransform)
//web socket's messages receiving logic
val webSocketReadLogic: Sink[Message, NotUsed] = Flow[Message].mapAsync[String](1)({
case textMessage: TextMessage =>
textMessage.toStrict(collectTimeout).map(_.text)
case binaryMessage: BinaryMessage =>
binaryMessage.toStrict(collectTimeout).map(_.data.toString())
}).to(Sink.foreach[String](messageText => println(s"received $messageText")))
//web socket's logic
val webSocketLogic: Flow[Message, Message, Consumer.Control] =
Flow.fromSinkAndSourceMat(webSocketReadLogic, webSocketWriteLogic)(Keep.right)
您可以根据partition stage的某些条件将流消息广播到多个接收器中。此外,你可以检查this explanation。