[ceph-users] Ceph cache pool full

Christian Wuerdig christian.wuerdig at gmail.com
Thu Oct 5 22:58:04 PDT 2017


The default filesize limit for CephFS is 1TB, see also here:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-May/018208.html
(also includes a pointer on how to increase it)

On Fri, Oct 6, 2017 at 12:45 PM, Shawfeng Dong <shaw at ucsc.edu> wrote:
> Dear all,
>
> We just set up a Ceph cluster, running the latest stable release Ceph
> v12.2.0 (Luminous):
> # ceph --version
> ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
>
> The goal is to serve Ceph filesystem, for which we created 3 pools:
> # ceph osd lspools
> 1 cephfs_data,2 cephfs_metadata,3 cephfs_cache,
> where
> * cephfs_data is the data pool (36 OSDs on HDDs), which is erased-coded;
> * cephfs_metadata is the metadata pool
> * cephfs_cache is the cache tier (3 OSDs on NVMes) for cephfs_data. The
> cache-mode is writeback.
>
> Everything had worked fine, until today when we tried to copy a 1.3TB file
> to the CephFS.  We got the "No space left on device" error!
>
> 'ceph -s' says some OSDs are full:
> # ceph -s
>   cluster:
>     id:     e18516bf-39cb-4670-9f13-88ccb7d19769
>     health: HEALTH_ERR
>             full flag(s) set
>             1 full osd(s)
>             1 pools have many more objects per pg than average
>
>   services:
>     mon: 3 daemons, quorum pulpo-admin,pulpo-mon01,pulpo-mds01
>     mgr: pulpo-mds01(active), standbys: pulpo-admin, pulpo-mon01
>     mds: pulpos-1/1/1 up  {0=pulpo-mds01=up:active}
>     osd: 39 osds: 39 up, 39 in
>          flags full
>
>   data:
>     pools:   3 pools, 2176 pgs
>     objects: 347k objects, 1381 GB
>     usage:   2847 GB used, 262 TB / 265 TB avail
>     pgs:     2176 active+clean
>
>   io:
>     client:   19301 kB/s rd, 2935 op/s rd, 0 op/s wr
>
> And indeed the cache pool is full:
> # rados df
> POOL_NAME       USED  OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND
> DEGRADED RD_OPS   RD
>     WR_OPS  WR
> cephfs_cache    1381G  355385      0 710770                  0       0
> 0 10004954 15
> 22G 1398063  1611G
> cephfs_data         0       0      0      0                  0       0
> 0        0
>   0       0      0
> cephfs_metadata 8515k      24      0     72                  0       0
> 0        3  3
> 072    3953 10541k
>
> total_objects    355409
> total_used       2847G
> total_avail      262T
> total_space      265T
>
> However, the data pool is completely empty! So it seems that data has only
> been written to the cache pool, but not written back to the data pool.
>
> I am really at a loss whether this is due to a setup error on my part, or a
> Luminous bug. Could anyone shed some light on this? Please let me know if
> you need any further info.
>
> Best,
> Shaw
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


More information about the ceph-users mailing list