BigQuery：如何从 STRUCT 数组中过滤掉空值

Question

当我的查询在结构数组中包含空值时，我无法在 BigQuery 中使用 SQL。这是我收到的错误：

Array cannot have a null element; error in writing field permissions.p.accts; error in writing field permissions.p; error in writing field permissions

我已经看过 ARRAY_AGG 函数的

Google 文档

，建议使用

IGNORE NULLS

选项，但据我所知，这仅适用于标量值数组，不适用于对象数组（STRUCT 数组） ）。我尝试了多种不同的方法来使用 ARRAY_AGG 函数，但无法让它在以下场景中工作。因此，我正在寻找一种解决方案，即使存在空值，也可以让我的查询正常工作（请参阅下面的第二个示例）。

这是一个运行良好的示例查询：

with example as (
  select JSON '{"permissions":{"p":[{"accts":["abc"],"perms":["def"]}]}}' as json_data
)

select STRUCT( 
          ARRAY(SELECT AS STRUCT 
                  JSON_VALUE_ARRAY(permission,'$.accts') as accts,
                  JSON_VALUE_ARRAY(permission,'$.perms') as perms
                FROM UNNEST(
                            JSON_QUERY_ARRAY(example.json_data, '$.permissions.p')
                          ) as permission
                ) as p 
             ) as permissions
from example
;

现在这是一个具有空值的相同查询的示例，它会抛出上面提到的错误。此查询中唯一的区别是

"accts":["abc"]

已更改为

"accts":[null]

。

with example as (
  select JSON '{"permissions":{"p":[{"accts":[null],"perms":["def"]}]}}' as json_data
)

select STRUCT( 
          ARRAY(SELECT AS STRUCT 
                  JSON_VALUE_ARRAY(permission,'$.accts') as accts,
                  JSON_VALUE_ARRAY(permission,'$.perms') as perms
                FROM UNNEST(
                            JSON_QUERY_ARRAY(example.json_data, '$.permissions.p')
                          ) as permission
                ) as p 
             ) as permissions
from example
;

非常感谢您的帮助！

Answer 1

解析JSON并获得数组后，需要对该数组进行取消嵌套和过滤以消除空条目。这可以在用户函数 UDF 或 subSelect 中完成。

Create temp function JSON_VALUE_ARRAY_no_NULL(ARR any type, str string)
as 
 (( SELECT X from UNNEST(JSON_VALUE_ARRAY(ARR ,str )) X WHERE X IS NOT NULL ))
;
with example as (
  select JSON '{"permissions":{"p":[{"accts":["abc"],"perms":["def"]}]}}' as json_data
  union all select JSON '{"permissions":{"p":[{"accts":[null],"perms":["def"]}]}}' 
)

select STRUCT( 
          ARRAY(SELECT AS STRUCT 
                  (( SELECT X from UNNEST(JSON_VALUE_ARRAY(permission ,'$.accts' )) X WHERE X IS NOT NULL )) as accts,
                  JSON_VALUE_ARRAY_no_NULL(permission ,'$.accts' ) as accts3,
                  JSON_VALUE_ARRAY_no_NULL(permission,'$.perms') as perms
                FROM UNNEST(
                            JSON_QUERY_ARRAY(example.json_data, '$.permissions.p')
                          ) as permission
                ) as p 
             ) as permissions
from example