使用“clusters.create”API初始化集群时指定连接器版本

Question

我在Python中使用clusters.create API在Dataproc中创建集群。

{
 "projectId": "my-project-id",
 "clusterName": "example-cluster",
 "config": {
"configBucket": "",
"gceClusterConfig": {
  "subnetworkUri": "default",
  "zoneUri": "us-central1-b"
},
"masterConfig": {
  "numInstances": 1,
  "machineTypeUri": "n1-standard-4",
  "diskConfig": {
    "bootDiskSizeGb": 500,
    "numLocalSsds": 0
  }
},
"workerConfig": {
  "numInstances": 2,
  "machineTypeUri": "n1-standard-4",
  "diskConfig": {
    "bootDiskSizeGb": 500,
    "numLocalSsds": 0
  }
},
"initializationActions": [
  {
    "executableFile": "gs://cloud-example-bucket/my-init-action.sh"
   }
  ]
 }
 }

在gcloud shell中，连接器版本和可执行文件指定为：

gcloud dataproc clusters create <CLUSTER_NAME> \
    --initialization-actions gs://dataproc-initialization- 
    actions/connectors/connectors.sh \
    --metadata 'gcs-connector-version=1.7.0' \
    --metadata 'bigquery-connector-version=0.11.0'

如何将连接器版本（在元数据下）传递给API。

在不传递版本的情况下运行我的脚本会出现以下错误：

ERROR: None of connector versions are specified'
ERROR: None of connector versions are specified
+ exit 1

Answer 1

元数据字段可以在config / gceClusterConfig下指定，如下所示：

'config': {
        'gceClusterConfig': {
             "metadata": {
                 "bigquery-connector-version": "0.12.1",
                 "gcs-connector-version": "1.8.1"
                         }
                       }
          }