我正在尝试可视化结构化流中的流查询。我该怎么办?我应该使用仪表板还是其他工具?
我在网上找不到任何类似的东西。
DF = spark \
.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", bootstrapServers)\
.option("subscribe", topics)\
.load()\
.selectExpr("CAST(value AS STRING)")
...
query1 = prediction.writeStream.outputMode("update").format('console').start()
query1.awaitTermination()
尝试类似的事情-queryName线索:
Scala
// Have all the aggregates in an in-memory table
aggDF
.writeStream
.queryName("aggregates") // this query name will be the table name
.outputMode("complete")
.format("memory")
.start()
spark.sql("select * from aggregates").show()
pyspark
# Have all the aggregates in an in-memory table. The query name will be the table name
aggDF \
.writeStream \
.queryName("aggregates") \
.outputMode("complete") \
.format("memory") \
.start()
spark.sql("select * from aggregates").show() # interactively query in-memory table