如何知道批云预测何时结束

Question

我正在做一个接收一些数据的应用程序，对其进行处理，然后创建一个AI批处理作业。预测完成后，我应该将它们全部合并起来并与以前的文件合并。批量预测写入存储桶，我们称其为gs://predictions。

[目前，我有一个云功能，每当写入gs://predictions时都会触发。但是，批量预测作业将数据流式传输到文件中，并且在要进行大量预测时会多次更新该文件。这意味着当我只想在作业完成时调用它时，我的云函数就会被触发很多次。

为了克服这个问题，现在调用云函数，然后检查作业是否完成。如果是，则处理文件；如果没有，让它滑动。当然，这带来了很多不必要的处理（和不必要的代码！）：-（

什么真正在这里对我有帮助：批处理作业完成后能否以某种方式写入Pub / Sub？甚至更好的是，它可以使用webhook以便在完成时本身调用我的云函数吗？我试图调查文档，但找不到任何东西。

还有其他建议的解决方案吗？

Answer 1

您可以create a log sink into PubSub并在此自定义过滤器上过滤日志：

resource.type="ml_job"
textPayload="Job completed successfully."

然后，当批处理作业完成时，将打印日志跟踪并将消息发布到PubSub主题中。

Answer 2

请参阅文档Google Cloud Training job docs您可以使用GCP提供的Rest API，请参考job AI Platform jobs docs

{
  "jobId": string,
  "createTime": string,
  "startTime": string,
  "endTime": string,
  "state": enum (State),  # This Field is what will tell you status of job
  "errorMessage": string,
  "labels": {
    string: string,
    ...
  },
  "etag": string,

  // Union field input can be only one of the following:
  "trainingInput": {
    object (TrainingInput)
  },
  "predictionInput": {
    object (PredictionInput)
  }
  // End of list of possible types for union field input.

  // Union field output can be only one of the following:
  "trainingOutput": {
    object (TrainingOutput)
  },
  "predictionOutput": {
    object (PredictionOutput)
  }
  // End of list of possible types for union field output.
}

状态字段是枚举

Enums
STATE_UNSPECIFIED   The job state is unspecified.
QUEUED              The job has been just created and processing has not yet begun.
PREPARING           The service is preparing to run the job.
RUNNING             The job is in progress.
SUCCEEDED           The job completed successfully.
FAILED              The job failed. errorMessage should contain the details of the failure.
CANCELLING          The job is being cancelled. errorMessage should describe the reason for the cancellation.
CANCELLED           The job has been cancelled. errorMessage should describe the reason for the cancellation.

注意：所有这些都可以通过python或任何其他选择的语言来完成。Using Python GCP Client Libraries