[ceph-users] rados bench leaves objects in tiered pool

Gregory Farnum gfarnum at redhat.com
Tue Nov 3 07:02:22 PST 2015


When you have a caching pool in writeback mode, updates to objects
(including deletes) are handled by writeback rather than writethrough.
Since there's no other activity against these pools, there is nothing
prompting the cache pool to flush updates out to the backing pool, so
the backing pool hasn't deleted its objects because nothing's told it
to. You'll find that the cache pool has deleted the data for its
objects, but it's keeping around a small "whiteout" and the object
info metadata.
The "rados ls" you're using has never played nicely with cache tiering
and probably never will. :( Listings are expensive operations and
modifying them to do more than the simple info scan would be fairly
expensive in terms of computation and IO.

I think there are some caching commands you can send to flush updates
which would cause the objects to be entirely deleted, but I don't have
them off-hand. You can probably search the mailing list archives or
the docs for tiering commands. :)
-Greg

On Tue, Nov 3, 2015 at 12:40 AM, Дмитрий Глушенок <glush at jet.msk.su> wrote:
> Hi,
>
> While benchmarking tiered pool using rados bench it was noticed that objects are not being removed after test.
>
> Test was performed using "rados -p rbd bench 3600 write". The pool is not used by anything else.
>
> Just before end of test:
> POOLS:
>     NAME                      ID     USED       %USED     MAX AVAIL     OBJECTS
>     rbd-cache                 36     33110M      3.41          114G        8366
>     rbd                       37     43472M      4.47          237G       10858
>
> Some time later (few hundreds of writes are flushed, rados automatic cleanup finished):
> POOLS:
>     NAME                      ID     USED       %USED     MAX AVAIL     OBJECTS
>     rbd-cache                 36      22998         0          157G       16342
>     rbd                       37     46050M      4.74          234G       11503
>
> # rados -p rbd-cache ls | wc -l
> 16242
> # rados -p rbd ls | wc -l
> 11503
> #
>
> # rados -p rbd cleanup
> error during cleanup: -2
> error 2: (2) No such file or directory
> #
>
> # rados -p rbd cleanup --run-name "" --prefix prefix ""
>  Warning: using slow linear search
>  Removed 0 objects
> #
>
> # rados -p rbd ls | head -5
> benchmark_data_dropbox01.tzk_7641_object10901
> benchmark_data_dropbox01.tzk_7641_object9645
> benchmark_data_dropbox01.tzk_7641_object10389
> benchmark_data_dropbox01.tzk_7641_object10090
> benchmark_data_dropbox01.tzk_7641_object11204
> #
>
> #  rados -p rbd-cache ls | head -5
> benchmark_data_dropbox01.tzk_7641_object10901
> benchmark_data_dropbox01.tzk_7641_object9645
> benchmark_data_dropbox01.tzk_7641_object10389
> benchmark_data_dropbox01.tzk_7641_object5391
> benchmark_data_dropbox01.tzk_7641_object10090
> #
>
> So, it looks like the objects are still in place (in both pools?). But it is not possible to remove them:
>
> # rados -p rbd rm benchmark_data_dropbox01.tzk_7641_object10901
> error removing rbd>benchmark_data_dropbox01.tzk_7641_object10901: (2) No such file or directory
> #
>
> # ceph health
> HEALTH_OK
> #
>
>
> Can somebody explain the behavior? And is it possible to cleanup the benchmark data without recreating the pools?
>
>
> ceph version 0.94.5
>
> # ceph osd dump | grep rbd
> pool 36 'rbd-cache' replicated size 3 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 100 pgp_num 100 last_change 755 flags hashpspool,incomplete_clones tier_of 37 cache_mode writeback target_bytes 107374182400 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 3600s x1 stripe_width 0
> pool 37 'rbd' erasure size 5 min_size 3 crush_ruleset 2 object_hash rjenkins pg_num 100 pgp_num 100 last_change 745 lfor 745 flags hashpspool tiers 36 read_tier 36 write_tier 36 stripe_width 4128
> #
>
> # ceph osd pool get rbd-cache hit_set_type
> hit_set_type: bloom
> # ceph osd pool get rbd-cache hit_set_period
> hit_set_period: 3600
> # ceph osd pool get rbd-cache hit_set_count
> hit_set_count: 1
> # ceph osd pool get rbd-cache target_max_objects
> target_max_objects: 0
> # ceph osd pool get rbd-cache target_max_bytes
> target_max_bytes: 107374182400
> # ceph osd pool get rbd-cache cache_target_dirty_ratio
> cache_target_dirty_ratio: 0.1
> # ceph osd pool get rbd-cache cache_target_full_ratio
> cache_target_full_ratio: 0.2
> #
>
> Crush map:
> root cache_tier {
>         id -7           # do not change unnecessarily
>         # weight 0.450
>         alg straw
>         hash 0  # rjenkins1
>         item osd.0 weight 0.090
>         item osd.1 weight 0.090
>         item osd.2 weight 0.090
>         item osd.3 weight 0.090
>         item osd.4 weight 0.090
> }
> root store_tier {
>         id -8           # do not change unnecessarily
>         # weight 0.450
>         alg straw
>         hash 0  # rjenkins1
>         item osd.5 weight 0.090
>         item osd.6 weight 0.090
>         item osd.7 weight 0.090
>         item osd.8 weight 0.090
>         item osd.9 weight 0.090
> }
> rule cache {
>         ruleset 1
>         type replicated
>         min_size 0
>         max_size 5
>         step take cache_tier
>         step chooseleaf firstn 0 type osd
>         step emit
> }
> rule store {
>         ruleset 2
>         type erasure
>         min_size 0
>         max_size 5
>         step take store_tier
>         step chooseleaf firstn 0 type osd
>         step emit
> }
>
> Thanks
>
> --
> Dmitry Glushenok
> Jet Infosystems
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


More information about the ceph-users mailing list