[ceph-users] OSD down ( rocksdb: submit_transaction error: Corruption: block checksum mismatch)

Karun Josy karunjosy1 at gmail.com
Thu Nov 23 10:27:49 PST 2017


Hi,

One OSD in the cluster is down. Tried to restart the service, but its still
failing.
I can see the below error in log file. Can this be a hardware issue ?

-------------------------------------

  -9> 2017-11-23 09:47:37.768969 7f368686a700  3 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/db_impl_compaction_flush.cc:1591]
Compaction error: Corruption: block checksum mismatch
    -8> 2017-11-23 09:47:37.768980 7f368686a700  4 rocksdb: (Original Log
Time 2017/11/23-09:47:37.768936)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/compaction_job.cc:621]
[default] compacted to: base level 1 max bytes base 268435456 files[11 1 0
0 0 0 0] max score 0.00, MB/sec: 2.3 rd, 2.0 wr, level 1, files in(11, 1)
out(1) MB in(0.1, 7.8) out(7.0), read-write-amplify(202.0)
write-amplify(94.6) Corruption: block checksum mismatch, records in: 42
    -7> 2017-11-23 09:47:37.768984 7f368686a700  4 rocksdb: (Original Log
Time 2017/11/23-09:47:37.768963) EVENT_LOG_v1 {"time_micros":
1511459257768950, "job": 3, "event": "compaction_finished",
"compaction_time_micros": 3667366, "output_level": 1, "num_output_files":
1, "total_output_size": 7317366, "num_input_records": 38738,
"num_output_records": 37539, "num_subcompactions": 1,
"num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0,
"lsm_state": [11, 1, 0, 0, 0, 0, 0]}
    -6> 2017-11-23 09:47:37.768988 7f368686a700  2 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/db_impl_compaction_flush.cc:1275]
Waiting after background compaction error: Corruption: block checksum
mismatch, Accumulated background error counts: 1
    -5> 2017-11-23 09:47:38.245022 7f369a708d00  5 osd.6 pg_epoch: 324
pg[3.98s5(unlocked)] enter Initial
    -4> 2017-11-23 09:47:38.245256 7f369a708d00  5 osd.6 pg_epoch: 324
pg[3.98s5( empty local-lis/les=323/324 n=0 ec=69/69 lis/c 323/323 les/c/f
324/324/0 323/323/69) [2,11,7,1,0,6,9,3] r=5 lpr=0 crt=0'0 unknown NOTIFY]
exit Initial 0.000235 0 0.000000
    -3> 2017-11-23 09:47:38.245275 7f369a708d00  5 osd.6 pg_epoch: 324
pg[3.98s5( empty local-lis/les=323/324 n=0 ec=69/69 lis/c 323/323 les/c/f
324/324/0 323/323/69) [2,11,7,1,0,6,9,3] r=5 lpr=0 crt=0'0 unknown NOTIFY]
enter Reset
    -2> 2017-11-23 09:47:38.245288 7f369a708d00  5 write_log_and_missing
with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: ,
clear_divergent_priors: 0
    -1> 2017-11-23 09:47:38.245355 7f368806d700 -1 rocksdb:
submit_transaction error: Corruption: block checksum mismatch code = 2
Rocksdb transaction:
Put( Prefix = M key = 0x000000000000052c'.can_rollback_to' Value size = 12)
Put( Prefix = M key = 0x000000000000052c'.rollback_info_trimmed_to' Value
size = 12)
Put( Prefix = O key =
0x8580000000000000031900000021213dfffffffffffffffeffffffffffffffff'o' Value
size = 29)
     0> 2017-11-23 09:47:38.247357 7f368806d700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/os/bluestore/BlueStore.cc:
In function 'void BlueStore::_kv_sync_thread()' thread 7f368806d700 time
2017-11-23 09:47:38.245386
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/os/bluestore/BlueStore.cc:
8453: FAILED assert(r == 0)

Karun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171123/2a7c026c/attachment.html>


More information about the ceph-users mailing list