我的环境创建了一个如下所示的变量:
SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}
编辑Ed Morton:根据the OPs comment below,这是他试图在上面描述的样本输入:
$ SM_TRAINING_ENV='{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}'
$ echo "$SM_TRAINING_ENV"
{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}
如何创建一个等于SM_TRAINING_ENV["hyperparameters"]["model_dir"]
值的新bash变量?
为了完整起见,我正在尝试像echo ${SM_TRAINING_ENV} | jq .
这样简单的事情,并且不断尝试我所做的一切。
编辑:我被告知这个值不是一个合适的json,所以重写问题。我认为环境将它设置为python字典的值,所以jq
似乎不可用。删除了json
标签。也许这是awk
的工作?
看起来我可以匹配我想要的值,如果我假设结构不会改变正则表达式模式s3.*?model
,但不知道如何将正则表达式模式设置为新变量。
首先,您需要引用JSON值,以便双引号将包含在值中。
SM_TRAINING_ENV='{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}'
然后,您可以使用jq
实用程序来提取所需的值。
new_var=$(echo "$SM_TRAINING_ENV" | jq '.hyperparameters.model_dir')
这不是真正的索引,但如果订单总是相同的话,它会起作用:
NEW_VAR=$(echo $SM_TRAINING_ENV | egrep -o s3.*?model | head -1)
更喜欢不依赖于订单的东西。