customer - c_id,c_name,c_address product - p_id,p_name,price supplier - s_id,s_name,s_address orders - o_id,c_id,p_id,quantity,time
SELECT o.o_id,
c.c_id,
c.c_name,
p.p_id,
p.p_name,
p.price * o.quantity AS amount
FROM customer c
JOIN orders o ON o.c_id = c.c_id
JOIN product p ON p.p_id = o.p_id;
我想执行上面的查询,而不是在pyspark中将3个表作为单独的数据帧提取并在数据帧上执行连接。
您可以使用表的查询,如下所述
df = spark.read.jdbc(
"url", "(query) as table",
properties={"user":"username", "password":"password"})
在你的情况下,它将是:
df = spark.read.jdbc("url", """
(
SELECT o.o_id,
c.c_id,
c.c_name,
p.p_id,
p.p_name,
p.price * o.quantity AS amount
FROM customer c
JOIN orders o ON o.c_id = c.c_id
JOIN product p ON p.p_id = o.p_id
) as table""", properties={"user":"username", "password":"password"})