我正在使用 AWS CloudFormation 模板来部署 AWS Glue 作业,如下所示:
...
"GluePythonScriptRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "glue.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
]
}
},
"GluePythonScriptJob": {
"Type": "AWS::Glue::Job",
"DependsOn": "GluePythonScriptRole",
"Properties": {
"Name": {
"Fn::Sub": "add-partition-projection-to-all-tables-${UniqueID}"
},
"Role": {
"Fn::GetAtt": [
"GluePythonScriptRole",
"Arn"
]
},
"ExecutionProperty": {
"MaxConcurrentRuns": 1
},
"Command": {
"Name": "pythonshell",
"ScriptLocation": "s3://xyz/add-partition-projection-to-all-tables.py"
},
"DefaultArguments": {
"--databasename": {
"Fn::Sub": "dbparquet${UniqueID}"
}
},
"MaxRetries": 0,
"Timeout": 2880
}
},
...
我已经验证了 Glue Job 和相关的 Python 脚本在手动设置时可以按预期工作。但是当我的 Glue Job 通过 CFT 创建时,我收到如下错误消息:
Failed to properly render job config field for: language. Please verify its value.
我的CFT好像少了点什么?
我最终按如下方式更新了 CFT 以使其正常工作,我认为需要显式添加 Python 版本:
"GluePythonScriptJob": {
"Type": "AWS::Glue::Job",
"DependsOn": "GluePythonScriptRole",
"Properties": {
"Name": {
"Fn::Sub": "add-partition-projection-to-all-tables-${UniqueID}"
},
"Role": {
"Fn::GetAtt": [
"GluePythonScriptRole",
"Arn"
]
},
"ExecutionProperty": {
"MaxConcurrentRuns": 1
},
"Command": {
"Name": "pythonshell",
"PythonVersion": "3.9",
"ScriptLocation": "s3://xyz/add-partition-projection-to-all-tables.py"
},
"DefaultArguments": {
"--databasename": {
"Fn::Sub": "dbparquet${UniqueID}"
},
"--job-language": "python",
"--additional-python-modules": "boto3"
},
"MaxRetries": 0,
"Timeout": 2880
}
}