将多个数组合并到映射中

Question

我有一些数据（来自全表的样本），看起来像这样：

| prov_id | hotel_id | m_id | apis_xml | company_id | yyyy_mm_dd |
|---------|----------|------|----------|------------|------------|
| 945     | 78888    | 3910 | [5]      | 998        | 2020-05-20 |
| 1475    | 78888    | 6676 | [1,2,4]  | 37         | 2020-05-20 |
| 1475    | 78888    | 6670 | [1,2,4]  | 37         | 2020-05-20 |
| 945     | 78888    | 2617 | [5]      | 998        | 2020-05-20 |

我想找到每个酒店的最低 apis_xml值，并将关联的prov_id设置为“ primary_prov”。我可以通过合并并排序在第一行之前升序排列的数组来做到这一点。我对此有一个查询：

SELECT
    yyyy_mm_dd,
    hotel_id,
    prov_id as primary_prov
FROM(
    SELECT
        yyyy_mm_dd,
        hotel_id,
        prov_id,
        apis_xml,
        ROW_NUMBER() OVER(PARTITION BY yyyy_mm_dd, hotel_id ORDER BY apis_xml) rn
    FROM(
        SELECT
            t.yyyy_mm_dd,
            t.hotel_id,
            t.prov_id,
            t.apis_xml,
            CAST(e.apis_xml AS INT) AS api
        FROM
            my_table t
        LATERAL VIEW EXPLODE(apis_xml) e AS apis_xml
    )s
)s
WHERE rn=1

上面的查询用于获取映射到最低apis_xml值的primary_prov。但是，我想有一个额外的列，其中包含其他xml值以供参考。也许地图不是这里的最佳用例，但这就是我想到的。密钥将是prov_id，而值将是apis_xml值。输出看起来像这样，因为我需要将其保留为每个hotel_id的一行：

| hotel_id | primary_prov | detailed_prov            | yyyy_mm_dd |
|----------|--------------|--------------------------|------------|
| 78888    | 1475         | {1475: [1,2,4], 945: [5] | 2020-05-20 |

Answer 1

您可以使用Brickhouse CollectUDAF构建map。阅读instructions how to build JAR，添加jar，创建函数并使用它：

add jar /path/to/jar/brickhouse-0.7.1.jar;
create temporary function collect as 'brickhouse.udf.collect.CollectUDAF';

SELECT yyyy_mm_dd,
       hotel_id,
       primary_prov,
       collect(prov_id, prov_id_api ) as detailed_prov
  FROM
(
SELECT
    yyyy_mm_dd,
    hotel_id,
    max(case when rn=1 then prov_id else null end) over(partition by yyyy_mm_dd, hotel_id) as primary_prov, 
    collect_set(api) over (partition by yyyy_mm_dd, hotel_id, prov_id) prov_id_api, --re-assemple array to include all elements from multiple initial arrays if there are different arrays per prov_id
    prov_id
FROM(
    SELECT
        yyyy_mm_dd,
        hotel_id,
        prov_id,
        apis_xml,
        api,
        ROW_NUMBER() OVER(PARTITION BY yyyy_mm_dd, hotel_id ORDER BY api) rn
    FROM(
        SELECT
            t.yyyy_mm_dd,
            t.hotel_id,
            t.prov_id,
            t.apis_xml,
            CAST(e.apis_xml_element AS INT) AS api
        FROM
            my_table t
        LATERAL VIEW EXPLODE(apis_xml) e AS apis_xml_element
    )s
)s
)s
group by yyyy_mm_dd,
         hotel_id,
         primary_prov