我已经在AWS基础事实中完成了标签工作，并开始研究用于对象检测的笔记本模板。
我有2个清单，其中有293个标记图像用于列车中的鸟类和验证集如下：
{"source-ref":"s3://XXXXXXX/Train/Blackbird_1.JPG","Bird-Label-Train":{"workerId":XXXXXXXX,"imageSource":{"s3Uri":"s3://XXXXXXX/Train/Blackbird_1.JPG"},"boxesInfo":{"annotatedResult":{"boundingBoxes":[{"width":1612,"top":841,"label":"Blackbird","left":1276,"height":757}],"inputImageProperties":{"width":3872,"height":2592}}}},"Bird-Label-Train-metadata":{"type":"groundtruth/custom","job-name":"bird-label-train","human-annotated":"yes","creation-date":"2019-01-16T17:28:23+0000"}}
以下是我用于笔记本实例的参数：
training_params = \
{
"AlgorithmSpecification": {
"TrainingImage": training_image, # NB. This is one of the named constants defined in the first cell.
"TrainingInputMode": "Pipe"
},
"RoleArn": role,
"OutputDataConfig": {
"S3OutputPath": s3_output_path
},
"ResourceConfig": {
"InstanceCount": 1,
"InstanceType": "ml.p3.2xlarge",
"VolumeSizeInGB": 5
},
"TrainingJobName": job_name,
"HyperParameters": { # NB. These hyperparameters are at the user's discretion and are beyond the scope of this demo.
"base_network": "resnet-50",
"use_pretrained_model": "1",
"num_classes": "1",
"mini_batch_size": "16",
"epochs": "5",
"learning_rate": "0.001",
"lr_scheduler_step": "3,6",
"lr_scheduler_factor": "0.1",
"optimizer": "rmsprop",
"momentum": "0.9",
"weight_decay": "0.0005",
"overlap_threshold": "0.5",
"nms_threshold": "0.45",
"image_shape": "300",
"label_width": "350",
"num_training_samples": str(num_training_samples)
},
"StoppingCondition": {
"MaxRuntimeInSeconds": 86400
},
"InputDataConfig": [
{
"ChannelName": "train",
"DataSource": {
"S3DataSource": {
"S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
"S3Uri": s3_train_data_path,
"S3DataDistributionType": "FullyReplicated",
"AttributeNames": ["source-ref","Bird-Label-Train"] # NB. This must correspond to the JSON field names in your augmented manifest.
}
},
"ContentType": "image/jpeg",
"RecordWrapperType": "None",
"CompressionType": "None"
},
{
"ChannelName": "validation",
"DataSource": {
"S3DataSource": {
"S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
"S3Uri": s3_validation_data_path,
"S3DataDistributionType": "FullyReplicated",
"AttributeNames": ["source-ref","Bird-Label"] # NB. This must correspond to the JSON field names in your augmented manifest.
}
},
"ContentType": "image/jpeg",
"RecordWrapperType": "None",
"CompressionType": "None"
}
]
我最终会在运行ml.p3.2xlarge实例后打印出来：
InProgress Starting
InProgress Starting
InProgress Starting
InProgress Training
Failed Failed
后跟此错误消息：'ClientError：未指定列车通道。'
有没有人有任何想法，我怎么能运行没有错误？任何帮助都是非常有用的！
成功运行：下面是使用的参数，以及成功运行的增强清单JSON对象。
training_params = \
{
"AlgorithmSpecification": {
"TrainingImage": training_image, # NB. This is one of the named constants defined in the first cell.
"TrainingInputMode": "Pipe"
},
"RoleArn": role,
"OutputDataConfig": {
"S3OutputPath": s3_output_path
},
"ResourceConfig": {
"InstanceCount": 1,
"InstanceType": "ml.p3.2xlarge",
"VolumeSizeInGB": 50
},
"TrainingJobName": job_name,
"HyperParameters": { # NB. These hyperparameters are at the user's discretion and are beyond the scope of this demo.
"base_network": "resnet-50",
"use_pretrained_model": "1",
"num_classes": "3",
"mini_batch_size": "1",
"epochs": "5",
"learning_rate": "0.001",
"lr_scheduler_step": "3,6",
"lr_scheduler_factor": "0.1",
"optimizer": "rmsprop",
"momentum": "0.9",
"weight_decay": "0.0005",
"overlap_threshold": "0.5",
"nms_threshold": "0.45",
"image_shape": "300",
"label_width": "350",
"num_training_samples": str(num_training_samples)
},
"StoppingCondition": {
"MaxRuntimeInSeconds": 86400
},
"InputDataConfig": [
{
"ChannelName": "train",
"DataSource": {
"S3DataSource": {
"S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
"S3Uri": s3_train_data_path,
"S3DataDistributionType": "FullyReplicated",
"AttributeNames": attribute_names # NB. This must correspond to the JSON field names in your **TRAIN** augmented manifest.
}
},
"ContentType": "application/x-recordio",
"RecordWrapperType": "RecordIO",
"CompressionType": "None"
},
{
"ChannelName": "validation",
"DataSource": {
"S3DataSource": {
"S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
"S3Uri": s3_validation_data_path,
"S3DataDistributionType": "FullyReplicated",
"AttributeNames": ["source-ref","ValidateBird"] # NB. This must correspond to the JSON field names in your **VALIDATION** augmented manifest.
}
},
"ContentType": "application/x-recordio",
"RecordWrapperType": "RecordIO",
"CompressionType": "None"
}
]
}
训练增强的清单在训练作业运行期间生成的文件
Line 1
{"source-ref":"s3://XXXXX/Train/Blackbird_1.JPG","TrainBird":{"annotations":[{"class_id":0,"width":1613,"top":840,"height":766,"left":1293}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"TrainBird-metadata":{"job-name":"labeling-job/trainbird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:21:29.829003","type":"groundtruth/object-detection"}}
Line 2
{"source-ref":"s3://xxxxx/Train/Blackbird_2.JPG","TrainBird":{"annotations":[{"class_id":0,"width":897,"top":665,"height":1601,"left":1598}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"TrainBird-metadata":{"job-name":"labeling-job/trainbird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:22:34.502274","type":"groundtruth/object-detection"}}
Line 3
{"source-ref":"s3://XXXXX/Train/Blackbird_3.JPG","TrainBird":{"annotations":[{"class_id":0,"width":1040,"top":509,"height":1695,"left":1548}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"TrainBird-metadata":{"job-name":"labeling-job/trainbird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:20:26.660164","type":"groundtruth/object-detection"}}
然后我解压缩model.tar文件以获取以下文件：hyperparams.JSON，model_algo_1-0000.params和model_algo_1-symbol
hyperparams.JSON看起来像这样：
{"label_width": "350", "early_stopping_min_epochs": "10", "epochs": "5", "overlap_threshold": "0.5", "lr_scheduler_factor": "0.1", "_num_kv_servers": "auto", "weight_decay": "0.0005", "mini_batch_size": "1", "use_pretrained_model": "1", "freeze_layer_pattern": "", "lr_scheduler_step": "3,6", "early_stopping": "False", "early_stopping_patience": "5", "momentum": "0.9", "num_training_samples": "11", "optimizer": "rmsprop", "_tuning_objective_metric": "", "early_stopping_tolerance": "0.0", "learning_rate": "0.001", "kv_store": "device", "nms_threshold": "0.45", "num_classes": "1", "base_network": "resnet-50", "nms_topk": "400", "_kvstore": "device", "image_shape": "300"}
您的列车和验证渠道中的'AttributeNames'参数需要是['source-ref'，'您的标签']
遗憾的是，AugmentedManifestFile内容类型不支持使用image/jpeg的管道模式。为了能够使用此功能，您需要将RecordWrapperType指定为RecordIO，将ContentType指定为application/x-recordio。
再次感谢你的帮助。所有这些都有助于我进一步发展。收到AWS论坛页面上的回复后，我终于开始工作了。
我知道我的JSON与增强的清单培训指南略有不同。回到基础后，我创建了另一个标签作业，但使用了“边界框”类型而不是“自定义 - 边界框模板”。我的输出符合预期。这没有错误！
由于我的目的是拥有多个标签，我能够编辑输出清单的文件和映射，这也很有效！
即
{"source-ref":"s3://xxxxx/Blackbird_15.JPG","ValidateBird":{"annotations":[{"class_id":0,"width":2023,"top":665,"height":1421,"left":1312}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"ValidateBird-metadata":{"job-name":"labeling-job/validatebird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:23:51.174131","type":"groundtruth/object-detection"}}
{"source-ref":"s3://xxxx/Pigeon_19.JPG","ValidateBird":{"annotations":[{"class_id":2,"width":784,"top":634,"height":1657,"left":1306}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"ValidateBird-metadata":{"job-name":"labeling-job/validatebird","class-map":{"2":"Pigeon"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:23:51.074809","type":"groundtruth/object-detection"}}
通过标签作业，原始映射为0：'Bird'表示所有图像。