据我所知,Ansible的S3模块,它只能立刻得到一个对象。
我的问题是,如果我想一次从S3存储桶下载/获取整个存储桶或多个对象,该怎么办?有什么黑客攻击吗?
我能够像这样实现它:
- name: get s3_bucket_items
s3:
mode=list
bucket=MY_BUCKET
prefix=MY_PREFIX/
register: s3_bucket_items
- name: download s3_bucket_items
s3:
mode=get
bucket=MY_BUCKET
object={{ item }}
dest=/tmp/
with_items: s3_bucket_items.s3_keys
笔记:
{{ item }}
值将具有前缀。ansible S3模块目前没有以递归方式将桶同步到磁盘的内置方法。
理论上,您可以尝试使用a来收集要下载的密钥
- name: register keys for syncronization
s3:
mode: list
bucket: hosts
object: /data/*
register: s3_bucket_items
- name: sync s3 bucket to disk
s3:
mode=get
bucket=hosts
object={{ item }}
dest=/etc/data/conf/
with_items: s3_bucket_items.s3_keys
虽然我经常看到这个解决方案,它似乎不适用于当前的ansible / boto版本,因为嵌套S3'目录'的错误(有关更多信息,请参阅this bug report),以及ansible S3模块没有创建密钥的子目录。我相信在同步非常大的存储桶时,使用此方法可能会遇到一些内存问题。
我还想补充一点,你很可能不想使用编码到你的剧本中的凭证 - 我建议你改用IAM EC2 instance profiles,它更加安全和舒适。
对我有用的解决方案是:
- name: Sync directory from S3 to disk
command: "s3cmd sync -q --no-preserve s3://hosts/{{ item }}/ /etc/data/conf/"
with_items:
- data
它将能够:
- name: Get s3 objects
s3:
bucket: your-s3-bucket
prefix: your-object-directory-path
mode: list
register: s3_object_list
- name: Create download directory
file:
path: "/your/destination/directory/path/{{ item | dirname }}"
state: directory
with_items:
- "{{ s3_object_list.s3_keys }}"
- name: Download s3 objects
s3:
bucket: your-s3-bucket
object: "{{ item }}"
mode: get
dest: "/your/destination/directory/path/{{ item }}"
with_items:
- "{{ s3_object_list.s3_keys }}"
以下代码将列出帐户中每个S3存储桶中的每个文件。它作为包含AWS密钥的group_vars / localhost / vault.yml的角色运行。
我还没有发现为什么第二个更直接的方法II不起作用,但也许有人可以启发我们。
- name: List S3 Buckets
aws_s3_bucket_facts:
aws_access_key: "{{ aws_access_key_id }}"
aws_secret_key: "{{ aws_secret_access_key }}"
# region: "eu-west-2"
register: s3_buckets
#- debug: var=s3_buckets
- name: Iterate buckets
set_fact:
app_item: "{{ item.name }}"
with_items: "{{ s3_buckets.ansible_facts.buckets }}"
register: app_result
#- debug: var=app_result.results #.item.name <= does not work??
- name: Create Fact List
set_fact:
s3_bucketlist: "{{ app_result.results | map(attribute='item.name') | list }}"
#- debug: var=s3_bucketlist
- name: List S3 Bucket files - Method I - works
local_action:
module: aws_s3
bucket: "{{ item }}"
aws_access_key: "{{ aws_access_key_id }}"
aws_secret_key: "{{ aws_secret_access_key }}"
mode: list
with_items:
- "{{ s3_bucketlist }}"
register: s3_list_I
#- debug: var=s3_list_I
- name: List S3 Bucket files - Method II - does not work
aws_s3:
aws_access_key: "{{ aws_access_key_id }}"
aws_secret_key: "{{ aws_secret_access_key }}"
bucket: "{{ item }}"
mode: list
with_items: "{{ s3_bucketlist }}"
register: s3_list_II
也许你可以改变你的“with_items”,然后应该工作
- name: get list to download aws_s3: region: "{{ region }}" bucket: "{{ item }}" mode: list with_items: "{{ s3_bucketlist }}" register: s3_bucket_items
但也许很快就是:
- name: Sync directory from S3 to disk command: "aws --region {{ region }} s3 sync s3://{{ bucket }}/ /tmp/test"
非安全的解决方案,但最终得到它在运行假定角色与S3桶访问,或AWS_ACCESS_KEY_ID
和AWS_SECRET_ACCESS_KEY
环境变量的实例
---
- name: download fs s3 bucket
command: aws s3 sync s3://{{ s3_backup_bucket }} {{ dst_path }}