用于Apache Spark的BigQuery连接器 - 更新分区表

Question

我在Google DataProc上的Scala中写了一个Spark工作，它每天执行并处理每个标记有事务时间的记录。记录按年 - 月组合分组，每组写入GCS的单独月度镶木地板文件（例如2018-07-file.parquet，2018-08-file.parquet等）。请注意，这些文件可以追溯到大约5年，并形成一个非常大的数据集（~1TB）。

我想将这些文件写入BigQuery，并让作业仅更新当前运行中已更改的月度记录。为简单起见，我想删除任何月份的现有记录和更新的记录，然后只需加载月度镶木地板文件中的数据。

我试图使用BigQuery Connector for DataProc，但它似乎只是support updating of an entire table而不是一个由日期字段过滤的一批记录，例如。

做这个的最好方式是什么？我尝试将完整的BigQuery库JAR包含到我的项目中，并使用数据操作查询删除现有的月度记录，如下所示：

def writeDataset(sparkContext: SparkContext, monthYear: String, ds: Dataset[TargetOrder]) = {
    val dtMonthYear = FeedWriter.parquetDateFormat.parse(monthYear)
    val bigquery: BigQuery = BigQueryOptions.getDefaultInstance.getService
    val queryConfig: QueryJobConfiguration =
      QueryJobConfiguration.newBuilder("DELETE FROM `" + getBQTableName(monthYear) + "` " +
        "WHERE header.trans_time BETWEEN PARSE_DATETIME('" + FeedWriter.parquetDateFormat.toPattern + "', '" + monthYear + "') " +
        "AND PARSE_DATETIME('" + FeedWriter.parquetDateFormat.toPattern + "', '" + DateUtils.addMonths(dtMonthYear, 1) + "') ")
    .setUseLegacySql(false)
    .build();

    val jobId: JobId = JobId.of(UUID.randomUUID().toString());
    val queryJob: Job = bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build()).waitFor()
}

但我得到以下错误（我假设不允许在DataProc作业中包含完整的BQ客户端JAR，或者它可能只是不能与BQ连接器一起使用）：

java.lang.NoSuchMethodError: com.google.api.services.bigquery.model.JobReference.setLocation(Ljava/lang/String;)Lcom/google/api/services/bigquery/model/JobReference;
  at com.google.cloud.bigquery.JobId.toPb(JobId.java:114)
  at com.google.cloud.bigquery.JobInfo.toPb(JobInfo.java:370)
  at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:198)
  at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:187)
  at ca.mycompany.myproject.output.BigQueryWriter$.writeDataset(BigQueryWriter.scala:39)

Answer 1

我发现在DataProc作业中包含完整的客户端JAR似乎不起作用（因此为什么他们为BQ和其他服务创建了单独的连接器扩展）所以相反，我最终让我的Dataproc作业向Pub / Sub队列提交消息指示哪个月度镶木地板文件已更新。然后，我创建了一个云函数来监视发布/订阅队列，并生成一个BigQuery作业，只加载已更改的每月文件。

我能够通过使用表分区（例如MyTable $ 20180101）从BQ表中删除月记录，并将所有月记录分组到同一天（目前，BQ仅支持按DAY而不是按月分区表，所以我必须为每个记录创建一个单独的字段，例如，2018-01-xx中的所有记录都设置为2018-01-01。

Dataproc中Scala代码的示例，用于写入Pub / Sub队列：

import java.text.SimpleDateFormat
import java.util.{Date, TimeZone, UUID}

import ca.my.company.config.ConfigOptions
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport
import com.google.api.client.json.jackson2.JacksonFactory
import com.google.api.services.pubsub.Pubsub
import com.google.api.services.pubsub.model.{PublishRequest, PubsubMessage}
import com.google.cloud.hadoop.util.RetryHttpInitializer
import org.apache.spark.streaming.pubsub.SparkGCPCredentials

import scala.collection.mutable

case class MyPubSubMessage (jobId: UUID, processedDate: Date, fileDate: Date,  updatedFilePath: String)

object PubSubWriter {
  private val PUBSUB_APP_NAME: String = "MyPubSubWriter"
  private val messages: mutable.ListBuffer[PubsubMessage] = mutable.ListBuffer()
  private val publishRequest = new PublishRequest()
  private lazy val projectId: String = ConfigOptions().pubsubConfig.projectId
  private lazy val topicId: String = ConfigOptions().pubsubConfig.topicId

  private lazy val client = new Pubsub.Builder(
    GoogleNetHttpTransport.newTrustedTransport(),
    JacksonFactory.getDefaultInstance(),
    new RetryHttpInitializer(
      SparkGCPCredentials.builder.build().provider,
      PUBSUB_APP_NAME
    ))
    .setApplicationName(PUBSUB_APP_NAME)
    .build()

  def queueMessage(message: TlogPubSubMessage) {
    if (message == null) return
    val targetFileDateFormat = new SimpleDateFormat("yyyyMMdd")
    val isoDateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'")
    isoDateFormat.setTimeZone(TimeZone.getTimeZone("UTC"))

    import scala.collection.JavaConversions._
    val pubSubMessage = new PubsubMessage()
      .setAttributes(Map("msgType" -> "t-log-notification", "jobId" -> message.jobId.toString, "processedDate" -> isoDateFormat.format(message.processedDate), "fileDate" -> targetFileDateFormat.format(message.fileDate)))

    messages.synchronized {
      messages.append(pubSubMessage.encodeData(message.updatedFilePath.getBytes))
    }
  }

  def publishMessages(): Unit = {
    import scala.collection.JavaConversions._
    publishRequest.setMessages(messages)
    client.projects().topics()
      .publish(s"projects/$projectId/topics/$topicId", publishRequest)
      .execute()

    println(s"Update notifications: successfully sent ${messages.length} message(s) for topic '${topicId}' to Pub/Sub")
  }
}

我的Python云函数示例从队列中消耗并生成BQ加载作业：

def update_bigquery(data, context):
    import base64
    from datetime import datetime
    from dateutil import parser
    from google.cloud import bigquery
    from google.cloud.bigquery.table import TimePartitioning
    from google.api_core.exceptions import GoogleAPICallError

    dataset_id = 'mydatasetname'
    table_id_base = 'mytablename'

    # The data field looks like this:
    # {'@type': 'type.googleapis.com/google.pubsub.v1.PubsubMessage', 'attributes': {'fileDate': '20171201',
    # 'jobId': '69f6307e-28a1-40fc-bb6d-572c0bea9346', 'msgType': 't-log-notification',
    # 'processedDate': '2018-09-08T02:51:54Z'}, 'data': 'Z3M6Ly9nY3MtbGRsLWRzLWRhdGE...=='}

    # Retrieve file path (filter out SUCCESS file in the folder path) and build the partition name
    attributes = data['attributes']
    file_path = base64.b64decode(data['data']).decode('utf-8') + "/part*"
    partition_name = attributes['fileDate']
    table_partition = table_id_base + "$" + partition_name

    # Instantiate BQ client
    client = bigquery.Client()

    # Get reference to dataset and table
    dataset_ref = client.dataset(dataset_id)
    table_ref = dataset_ref.table(table_partition)

    try:
        # This only deletes the table partition and not the entire table
        client.delete_table(table_ref)  # API request
        print('Table {}:{} deleted.'.format(dataset_id, table_partition))

    except GoogleAPICallError as e:
        print('Error deleting table ' + table_partition + ": " + str(e))

    # Create BigQuery loading job
    job_config = bigquery.LoadJobConfig()
    job_config.source_format = bigquery.SourceFormat.PARQUET
    job_config.time_partitioning = TimePartitioning(field='bigQueryPartition')

    try :
        load_job = client.load_table_from_uri(
            file_path,
            dataset_ref.table(table_partition),
            job_config=job_config)  # API request

        print('Starting job {}'.format(load_job.job_id))

        # This can be commented-out to allow the job to run purely asynchronously
        # though if it fails, I'm not sure how I could be notified
        # For now, I will set this function to the max timeout (9 mins) and see if the BQ load job can consistently complete in time
        load_job.result()  # Waits for table load to complete.
        print('Job finished.')

    except GoogleAPICallError as e:
        print("Error running BQ load job: " + str(e))
        raise e

    return 'Success'

Answer 2

bigquery4s怎么样？

它是BQ Java客户端的Scala包装器。我有同样的问题，它对我有用。