Bigquery查询结果到具有气流的数据框

Question

我正在尝试从bigquery查询数据，并使用Airflow将其写入数据框。但是它给出了file not found（服务帐户密钥）或file name is too long或eof line read错误。

我也尝试过钩子，但是我不能将密钥文件作为json放，因为它说太长了。

关于如何实现的任何提示？

def get_data_from_GBQ():

global customer_data
ofo_cred = Variable.get("ofo_cred")
logging.info(ofo_cred)
logging.info("Variable is here")
customer_data_query = """ SELECT FirstName, LastName, Organisation FROM `bigquery-bi.ofo.Customers` LIMIT 2 """
logging.info("test")

# Creating a connection to the google bigquery
client = bigquery.Client.from_service_account_json(ofo_cred)
logging.info("after client")
customer_data = client.query(customer_data_query).to_dataframe()
logging.info("after client")
print(customer_data)

dag = DAG(
'odoo_gbq_connection',
default_args=default_args,
description='A connection between ',
schedule_interval=timedelta(days=1),)

错误是：

FileNotFoundError: [Errno 2] No such file or directory: '{\r\n  "type": "service_account",\r\n  "project_id":...

Answer 1

bigquery.Client.from_service_account_json函数需要服务帐户文件的文件名，您需要提供该文件的内容，因此它会尝试查找路径以{\r\n "type": "servi...开头但以FileNotFound失败的文件。

可能的解决方法：

client = bigquery.Client.from_service_account_json(path_to_ofo_cred)

https://googleapis.dev/python/google-api-core/latest/auth.html#service-accounts