[ceph-users] rados bench leaves objects in tiered pool

Дмитрий Глушенок glush at jet.msk.su
Tue Nov 3 00:40:44 PST 2015


Hi,

While benchmarking tiered pool using rados bench it was noticed that objects are not being removed after test.

Test was performed using "rados -p rbd bench 3600 write". The pool is not used by anything else.

Just before end of test:
POOLS:
    NAME                      ID     USED       %USED     MAX AVAIL     OBJECTS
    rbd-cache                 36     33110M      3.41          114G        8366
    rbd                       37     43472M      4.47          237G       10858

Some time later (few hundreds of writes are flushed, rados automatic cleanup finished):
POOLS:
    NAME                      ID     USED       %USED     MAX AVAIL     OBJECTS
    rbd-cache                 36      22998         0          157G       16342
    rbd                       37     46050M      4.74          234G       11503

# rados -p rbd-cache ls | wc -l
16242
# rados -p rbd ls | wc -l
11503
#

# rados -p rbd cleanup
error during cleanup: -2
error 2: (2) No such file or directory
#

# rados -p rbd cleanup --run-name "" --prefix prefix ""
 Warning: using slow linear search
 Removed 0 objects
#

# rados -p rbd ls | head -5
benchmark_data_dropbox01.tzk_7641_object10901
benchmark_data_dropbox01.tzk_7641_object9645
benchmark_data_dropbox01.tzk_7641_object10389
benchmark_data_dropbox01.tzk_7641_object10090
benchmark_data_dropbox01.tzk_7641_object11204
#

#  rados -p rbd-cache ls | head -5
benchmark_data_dropbox01.tzk_7641_object10901
benchmark_data_dropbox01.tzk_7641_object9645
benchmark_data_dropbox01.tzk_7641_object10389
benchmark_data_dropbox01.tzk_7641_object5391
benchmark_data_dropbox01.tzk_7641_object10090
#

So, it looks like the objects are still in place (in both pools?). But it is not possible to remove them:

# rados -p rbd rm benchmark_data_dropbox01.tzk_7641_object10901
error removing rbd>benchmark_data_dropbox01.tzk_7641_object10901: (2) No such file or directory
#

# ceph health
HEALTH_OK
#


Can somebody explain the behavior? And is it possible to cleanup the benchmark data without recreating the pools?


ceph version 0.94.5

# ceph osd dump | grep rbd
pool 36 'rbd-cache' replicated size 3 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 100 pgp_num 100 last_change 755 flags hashpspool,incomplete_clones tier_of 37 cache_mode writeback target_bytes 107374182400 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 3600s x1 stripe_width 0
pool 37 'rbd' erasure size 5 min_size 3 crush_ruleset 2 object_hash rjenkins pg_num 100 pgp_num 100 last_change 745 lfor 745 flags hashpspool tiers 36 read_tier 36 write_tier 36 stripe_width 4128
#

# ceph osd pool get rbd-cache hit_set_type
hit_set_type: bloom
# ceph osd pool get rbd-cache hit_set_period
hit_set_period: 3600
# ceph osd pool get rbd-cache hit_set_count
hit_set_count: 1
# ceph osd pool get rbd-cache target_max_objects
target_max_objects: 0
# ceph osd pool get rbd-cache target_max_bytes
target_max_bytes: 107374182400
# ceph osd pool get rbd-cache cache_target_dirty_ratio
cache_target_dirty_ratio: 0.1
# ceph osd pool get rbd-cache cache_target_full_ratio
cache_target_full_ratio: 0.2
#

Crush map:
root cache_tier {   
        id -7           # do not change unnecessarily
        # weight 0.450
        alg straw   
        hash 0  # rjenkins1
        item osd.0 weight 0.090
        item osd.1 weight 0.090
        item osd.2 weight 0.090
        item osd.3 weight 0.090
        item osd.4 weight 0.090
}
root store_tier {   
        id -8           # do not change unnecessarily
        # weight 0.450
        alg straw   
        hash 0  # rjenkins1
        item osd.5 weight 0.090
        item osd.6 weight 0.090
        item osd.7 weight 0.090
        item osd.8 weight 0.090
        item osd.9 weight 0.090
}
rule cache {
        ruleset 1
        type replicated
        min_size 0
        max_size 5
        step take cache_tier
        step chooseleaf firstn 0 type osd
        step emit
}
rule store {
        ruleset 2
        type erasure
        min_size 0
        max_size 5
        step take store_tier
        step chooseleaf firstn 0 type osd
        step emit
}

Thanks

--
Dmitry Glushenok
Jet Infosystems



More information about the ceph-users mailing list