Placement_pools on Rados-GW

The purpose of this test is to map a RadosGw Bucket to a specific Ceph pool. For exemple, if using a fast pool with ssd and a low pool for archive…

1 2	standard_bucket datas --> .rgw.buckets (default pool) specific_bucket datas --> .rgw.buckets.custom

First, we create a pool .rgw.buckets.custom, with, for example, some specific parameters (different size and different ruleset in crushmap) :

1 2 3 4 5 6 7 8

$ ceph osd pool create .rgw.buckets.custom 64 64 pool '.rgw.buckets.custom' created $ ceph osd pool set .rgw.buckets.custom size 2 set pool 59 size to 2 $ ceph osd pool set .rgw.buckets.custom crush_ruleset 6 set pool 59 crush_ruleset to 6

Then, we need to configure a specific placement_targets in region map and zone. For next step, you need to have a running config of rados-gw…

1 2	$ radosgw-admin region get > region.conf.json $ vim region.conf.json # Add an entry in placement_targets

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

{ "name": "default", "api_name": "", "is_master": "true", "endpoints": [], "master_zone": "", "zones": [ { "name": "default", "endpoints": [], "log_meta": "false", "log_data": "false"}], "placement_targets": [ { "name": "default-placement", "tags": []}, { "name": "custom-placement", "tags": []}], "default_placement": "default-placement"}

1 2	$ radosgw-admin region set < region.conf.json ....

1 2	$ radosgw-admin zone get > zone.conf.json $ vim zone.conf.json # Add an entry in placement_pools with key "custom-placement"

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

{ "domain_root": ".rgw", "control_pool": ".rgw.control", "gc_pool": ".rgw.gc", "log_pool": ".log", "intent_log_pool": ".intent-log", "usage_log_pool": ".usage", "user_keys_pool": ".users", "user_email_pool": ".users.email", "user_swift_pool": ".users.swift", "user_uid_pool": ".users.uid", "system_key": { "access_key": "", "secret_key": ""}, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": ".rgw.buckets.index", "data_pool": ".rgw.buckets", "data_extra_pool": ".rgw.buckets.extra"}}, { "key": "custom-placement", "val": { "index_pool": ".rgw.buckets.index", "data_pool": ".rgw.buckets.custom", "data_extra_pool": ".rgw.buckets.extra"}}]}

1 2 3

$ radosgw-admin zone set <zone.conf.json 2014-11-25 18:03:23.894153 7f728c0f2780 0 couldn't find old data placement pools config, setting up new ones for the zone .....

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

$ radosgw-admin regionmap update { "regions": [ { "key": "default", "val": { "name": "default", "api_name": "", "is_master": "true", "endpoints": [], "master_zone": "", "zones": [ { "name": "default", "endpoints": [], "log_meta": "false", "log_data": "false"}], "placement_targets": [ { "name": "custom-placement", "tags": []}, { "name": "default-placement", "tags": []}], "default_placement": "default-placement"}}], "master_region": "default", "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1}, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1}} $ /etc/init.d/radosgw reload Reloading ...

To configure s3cmd for RadosGW you can have a look here : http://lollyrock.com/articles/s3cmd-with-radosgw/

Now we can test bucket creation :

1 2 3 4 5 6 7 8 9

$ s3cmd mb s3://custombucket --bucket-location=custom-placement Bucket 'custombucket' created $ touch "file_on_custom_pool" $ s3cmd put file_on_custom_pool s3://custombucket WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. file_on_custom_pool -> s3://custombucket/file_on_custom_pool [1 of 1] 0 of 0 0% in 0s 0.00 B/s done

Verify :

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

$ radosgw-admin bucket stats --bucket=custombucket { "bucket": "custombucket", "pool": ".rgw.buckets.custom", "index_pool": ".rgw.buckets.index", "id": "default.240909.1", "marker": "default.240909.1", "owner": "testuser", "ver": 1, "master_ver": 0, "mtime": 1417016078, "max_marker": "", "usage": {}, "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1}}

Pool var is set on “.rgw.buckets.custom”.

1 2	$ rados -p .rgw.buckets.custom ls default.241071.1_file_on_custom_pool

It’s here !

Data placement pool is define in this order :

1. from the request (“bucket location”)
2. from user (“default_placement” : see with radosgw-admin metadata get user:<uid>)
3. from region map (“default_placement”)