[ceph-users] Getting errors on erasure pool writes k=2, m=1

Marc Roos M.Roos at f1-outsourcing.eu
Mon Nov 13 01:18:03 PST 2017


 

I have been asking myself (and here) the same question. I think it is 
because of having this in ceph.conf
enable experimental unrecoverable data corrupting features = bluestore
But I am not sure if I can remove this, or have to replace this with 
something else.

ceph-12.2.1-0.el7.x86_64
ceph-base-12.2.1-0.el7.x86_64
ceph-common-12.2.1-0.el7.x86_64
ceph-mds-12.2.1-0.el7.x86_64
ceph-mgr-12.2.1-0.el7.x86_64
ceph-mon-12.2.1-0.el7.x86_64
ceph-osd-12.2.1-0.el7.x86_64
ceph-selinux-12.2.1-0.el7.x86_64
collectd-ceph-5.7.1-2.el7.x86_64
libcephfs2-12.2.1-0.el7.x86_64
nfs-ganesha-ceph-2.5.2-.el7.x86_64
python-cephfs-12.2.1-0.el7.x86_64




-----Original Message-----
From: Caspar Smit [mailto:casparsmit at supernas.eu] 
Sent: maandag 13 november 2017 9:58
To: ceph-users
Subject: Re: [ceph-users] Getting errors on erasure pool writes k=2, m=1

Hi,

Why would Ceph 12.2.1 give you this message:

2017-11-10 20:39:31.296101 7f840ad45e40 -1 WARNING: the following 
dangerous and experimental features are enabled: bluestore



Or is that a leftover warning message from an old client?

Kind regards,
Caspar


2017-11-10 21:27 GMT+01:00 Marc Roos <M.Roos at f1-outsourcing.eu>:



	osd's are crashing when putting a (8GB) file in a erasure coded 
pool,
	just before finishing. The same osd's are used for replicated pools
	rbd/cephfs, and seem to do fine. Did I made some error is this a 
bug?
	Looks similar to
	https://www.spinics.net/lists/ceph-devel/msg38685.html 
<https://www.spinics.net/lists/ceph-devel/msg38685.html> 
	http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/
021045.html 
<http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/021045.html> 
	
	
	[@c01 ~]# date ; rados -p ec21 put  $(basename
	"/mnt/disk/blablablalbalblablalablalb.txt")
	blablablalbalblablalablalb.txt
	Fri Nov 10 20:27:26 CET 2017
	
	[Fri Nov 10 20:33:51 2017] libceph: osd9 down
	[Fri Nov 10 20:33:51 2017] libceph: osd9 down
	[Fri Nov 10 20:33:51 2017] libceph: osd0 192.168.10.111:6802 socket
	closed (con state OPEN)
	[Fri Nov 10 20:33:51 2017] libceph: osd0 192.168.10.111:6802 socket
	error on write
	[Fri Nov 10 20:33:52 2017] libceph: osd0 down
	[Fri Nov 10 20:33:52 2017] libceph: osd7 down
	[Fri Nov 10 20:33:55 2017] libceph: osd0 down
	[Fri Nov 10 20:33:55 2017] libceph: osd7 down
	[Fri Nov 10 20:34:41 2017] libceph: osd7 up
	[Fri Nov 10 20:34:41 2017] libceph: osd7 up
	[Fri Nov 10 20:35:03 2017] libceph: osd9 up
	[Fri Nov 10 20:35:03 2017] libceph: osd9 up
	[Fri Nov 10 20:35:47 2017] libceph: osd0 up
	[Fri Nov 10 20:35:47 2017] libceph: osd0 up
	
	[@c02 ~]# rados -p ec21 stat blablablalbalblablalablalb.txt
	2017-11-10 20:39:31.296101 7f840ad45e40 -1 WARNING: the following
	dangerous and experimental features are enabled: bluestore
	2017-11-10 20:39:31.296290 7f840ad45e40 -1 WARNING: the following
	dangerous and experimental features are enabled: bluestore
	2017-11-10 20:39:31.331588 7f840ad45e40 -1 WARNING: the following
	dangerous and experimental features are enabled: bluestore
	ec21/blablablalbalblablalablalb.txt mtime 2017-11-10 
20:32:52.000000,
	size 8585740288
	
	
	
	2017-11-10 20:32:52.287503 7f933028d700  4 rocksdb: EVENT_LOG_v1
	{"time_micros": 1510342372287484, "job": 32, "event": 
"flush_started",
	"num_memtables": 1, "num_entries": 728747, "num_deletes": 363960,
	"memory_usage": 263854696}
	2017-11-10 20:32:52.287509 7f933028d700  4 rocksdb:
	[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILAB
LE_AR
	CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rel
ease/
	12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/flush_job.cc:293]
	[default] [JOB 32] Level-0 flush table #25279: started
	2017-11-10 20:32:52.503311 7f933028d700  4 rocksdb: EVENT_LOG_v1
	{"time_micros": 1510342372503293, "cf_name": "default", "job": 32,
	"event": "table_file_creation", "file_number": 25279, "file_size":
	4811948, "table_properties": {"data_size": 4675796, "index_size":
	102865, "filter_size": 32302, "raw_key_size": 646440,
	"raw_average_key_size": 75, "raw_value_size": 4446103,
	"raw_average_value_size": 519, "num_data_blocks": 1180, 
"num_entries":
	8560, "filter_policy_name": "rocksdb.BuiltinBloomFilter",
	"kDeletedKeys": "0", "kMergeOperands": "330"}}
	2017-11-10 20:32:52.503327 7f933028d700  4 rocksdb:
	[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILAB
LE_AR
	CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rel
ease/
	12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/flush_job.cc:319]
	[default] [JOB 32] Level-0 flush table #25279: 4811948 bytes OK
	2017-11-10 20:32:52.572413 7f933028d700  4 rocksdb:
	[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILAB
LE_AR
	CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rel
ease/
	12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/db_impl_files.cc:24
2]
	adding log 25276 to recycle list
	
	2017-11-10 20:32:52.572422 7f933028d700  4 rocksdb: (Original Log 
Time
	2017/11/10-20:32:52.503339)
	[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILAB
LE_AR
	CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rel
ease/
	12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/memtable_list.cc:36
0]
	[default] Level-0 commit table #25279 started
	2017-11-10 20:32:52.572425 7f933028d700  4 rocksdb: (Original Log 
Time
	2017/11/10-20:32:52.572312)
	[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILAB
LE_AR
	CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rel
ease/
	12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/memtable_list.cc:38
3]
	[default] Level-0 commit table #25279: memtable #1 done
	2017-11-10 20:32:52.572428 7f933028d700  4 rocksdb: (Original Log 
Time
	2017/11/10-20:32:52.572328) EVENT_LOG_v1 {"time_micros":
	1510342372572321, "job": 32, "event": "flush_finished", 
"lsm_state": [4,
	4, 36, 140, 0, 0, 0], "immutable_memtables": 0}
	2017-11-10 20:32:52.572430 7f933028d700  4 rocksdb: (Original Log 
Time
	2017/11/10-20:32:52.572397)
	[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILAB
LE_AR
	CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rel
ease/
	12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/db_impl_compaction_
flush
	.cc:132] [default] Level summary: base level 1 max bytes base 
268435456
	files[4 4 36 140 0 0 0] max score 1.00
	
	2017-11-10 20:32:52.572491 7f933028d700  4 rocksdb:
	[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILAB
LE_AR
	CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rel
ease/
	12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/db_impl_files.cc:38
8]
	[JOB 32] Try to delete WAL files size 249697663, prev total WAL 
file
	size 260924266, number of live WAL files 2.
	
	2017-11-10 20:32:52.572548 7f933f2ab700  4 rocksdb:
	[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILAB
LE_AR
	CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rel
ease/
	12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/compaction_job.cc:1
403]
	[default] [JOB 33] Compacting 4 at 0 + 4 at 1 files to L1, score 1.00
	2017-11-10 20:32:52.572567 7f933f2ab700  4 rocksdb:
	[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILAB
LE_AR
	CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rel
ease/
	12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/compaction_job.cc:1
407]
	[default] Compaction start summary: Base version 32 Base level 0,
	inputs: [25279(4699KB) 25277(4627KB) 25275(4774KB) 25273(4407KB)],
	[25268(66MB) 25269(65MB) 25270(29MB) 25271(39MB)]
	
	2017-11-10 20:32:52.572597 7f933f2ab700  4 rocksdb: EVENT_LOG_v1
	{"time_micros": 1510342372572585, "job": 33, "event":
	"compaction_started", "files_L0": [25279, 25277, 25275, 25273],
	"files_L1": [25268, 25269, 25270, 25271], "score": 1, 
"input_data_size":
	230638516}
	2017-11-10 20:32:52.600626 7f9336299700 -1
	bluestore(/var/lib/ceph/osd/ceph-0) _txc_add_transaction error (7)
	Argument list too long not handled on operation 10 (op 2, counting 
from
	0)
	2017-11-10 20:32:52.600643 7f9336299700 -1
	bluestore(/var/lib/ceph/osd/ceph-0) unexpected error code
	2017-11-10 20:32:52.600644 7f9336299700  0
	bluestore(/var/lib/ceph/osd/ceph-0)  transaction dump:
	{
	    "ops": [
	        {
	            "op_num": 0,
	            "op_name": "setattrs",
	            "collection": "49.4s1_head",
	            "oid":
	"1#49:25f193eb:::blablablalbalblablalablalb.txt:head#",
	            "attr_lens": {
	                "_": 285,
	                "snapset": 35
	            }
	        },
	        {
	            "op_num": 1,
	            "op_name": "op_setallochint",
	            "collection": "49.4s1_head",
	            "oid":
	"1#49:25f193eb:::blablablalbalblablalablalb.txt:head#",
	            "expected_object_size": "0",
	            "expected_write_size": "0"
	        },
	        {
	            "op_num": 2,
	            "op_name": "write",
	            "collection": "49.4s1_head",
	            "oid":
	"1#49:25f193eb:::blablablalbalblablalablalb.txt:head#",
	            "length": 2097152,
	            "offset": 4292870144,
	            "bufferlist length": 2097152
	        },
	        {
	            "op_num": 3,
	            "op_name": "setattr",
	            "collection": "49.4s1_head",
	            "oid":
	"1#49:25f193eb:::blablablalbalblablalablalb.txt:head#",
	            "name": "hinfo_key",
	            "length": 30
	        }
	    ]
	}
	
	2017-11-10 20:32:52.605310 7f9336299700 -1
	/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL
E_ARC
	H/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele
ase/1
	2.2.1/rpm/el7/BUILD/ceph-12.2.1/src/os/bluestore/BlueStore.cc: In
	function 'void 
BlueStore::_txc_add_transaction(BlueStore::TransContext*,
	ObjectStore::Transaction*)' thread 7f9336299700 time 2017-11-10
	20:32:52.600707
	/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL
E_ARC
	H/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele
ase/1
	2.2.1/rpm/el7/BUILD/ceph-12.2.1/src/os/bluestore/BlueStore.cc: 
9299:
	FAILED assert(0 == "unexpected error")
	
	 ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) 
luminous
	(stable)
	 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
	const*)+0x110) [0x558b3ad7bf20]
	 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*,
	ObjectStore::Transaction*)+0x1487) [0x558b3ac51357]
	 3: (BlueStore::queue_transactions(ObjectStore::Sequencer*,
	std::vector<ObjectStore::Transaction,
	std::allocator<ObjectStore::Transaction> >&,
	boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3a0)
	[0x558b3ac523a0]
	 4:
	(PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transact
ion,
	std::allocator<ObjectStore::Transaction> >&,
	boost::intrusive_ptr<OpRequest>)+0x65) [0x558b3a9c2f35]
	 5: (ECBackend::handle_sub_write(pg_shard_t,
	boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace 
const&,
	Context*)+0x631) [0x558b3aae2cc1]
	 6: 
(ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x327)
	[0x558b3aaf3867]
	 7: 
(PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50)
	[0x558b3a9f83d0]
	 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&,
	ThreadPool::TPHandle&)+0x5ae) [0x558b3a963fbe]
	 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
	boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f9)
	[0x558b3a7f43a9]
	 10: 
(PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest>
	const&)+0x57) [0x558b3aa66797]
	 11: (OSD::ShardedOpWQ::_process(unsigned int,
	ceph::heartbeat_handle_d*)+0xfce) [0x558b3a81f9ee]
	 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned 
int)+0x839)
	[0x558b3ad81a39]
	 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10)
	[0x558b3ad839d0]
	 14: (()+0x7e25) [0x7f9350578e25]
	 15: (clone()+0x6d) [0x7f934f66c34d]
	 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
	needed to interpret this.
	
	--- begin dump of recent events ---
	-10000> 2017-11-10 20:31:17.056793 7f934da86700  1 --
	10.0.0.11:6803/29353 <== osd.7 10.0.0.11:6808/29354 3045 ====
	MOSDECSubOpWrite(49.4s1 18323/18322 ECSubWrite(tid=9201,
	reqid=client.2886221.0:1519, at_version=0'0, trim_to=18314'2200,
	roll_forward_to=18323'3739)) v2 ==== 838+0+0 (3687746787 0 0)
	0x558b48108800 con 0x558b86007800
	 -9999> 2017-11-10 20:31:17.065748 7f934da86700  5 --
	10.0.0.11:6803/29353 >> 10.0.0.11:6808/29354 conn(0x558b86007800 
:-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=3 l=0). rx 
osd.7
	seq 3046 0x558b48109000 MOSDECSubOpWrite(49.4s1 18323/18322
	ECSubWrite(tid=9202, reqid=client.2886221.0:1520, 
at_version=18323'3740,
	trim_to=18314'2200, roll_forward_to=18323'3739)) v2
	 -9998> 2017-11-10 20:31:17.065773 7f934da86700  1 --
	10.0.0.11:6803/29353 <== osd.7 10.0.0.11:6808/29354 3046 ====
	MOSDECSubOpWrite(49.4s1 18323/18322 ECSubWrite(tid=9202,
	reqid=client.2886221.0:1520, at_version=18323'3740, 
trim_to=18314'2200,
	roll_forward_to=18323'3739)) v2 ==== 2099384+0+0 (649622164 0 0)
	0x558b48109000 con 0x558b86007800
	 -9997> 2017-11-10 20:31:17.065893 7f9336299700  5 
write_log_and_missing
	with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
	writeout_from: 18323'3740, trimmed: , trimmed_dups: ,
	clear_divergent_priors: 0
	 -9996> 2017-11-10 20:31:17.071553 7f934d285700  5 --
	10.0.0.11:6805/29353 >> 10.0.0.12:0/3646 conn(0x558b85c3a800 :6805
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx 
osd.9
	seq 4839 0x558bd79e7c00 osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.071631) v4
	 -9995> 2017-11-10 20:31:17.071585 7f934d285700  1 --
	10.0.0.11:6805/29353 <== osd.9 10.0.0.12:0/3646 4839 ==== 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.071631) v4 ==== 2004+0+0 
(3550206936 0
	0) 0x558bd79e7c00 con 0x558b85c3a800
	 -9994> 2017-11-10 20:31:17.071574 7f934ca84700  5 --
	192.168.10.111:6805/29353 >> 192.168.10.112:0/3646 
conn(0x558b85c39000
	:6805 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 
l=1). rx
	osd.9 seq 4839 0x558b502e6600 osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.071631) v4
	 -9993> 2017-11-10 20:31:17.071598 7f934ca84700  1 --
	192.168.10.111:6805/29353 <== osd.9 192.168.10.112:0/3646 4839 ====
	osd_ping(ping e18323 stamp 2017-11-10 20:31:17.071631) v4 ==== 
2004+0+0
	(3550206936 0 0) 0x558b502e6600 con 0x558b85c39000
	 -9992> 2017-11-10 20:31:17.071615 7f934d285700  1 --
	10.0.0.11:6805/29353 --> 10.0.0.12:0/3646 -- osd_ping(ping_reply 
e18323
	stamp 2017-11-10 20:31:17.071631) v4 -- 0x558b4e52e200 con 0
	 -9991> 2017-11-10 20:31:17.071649 7f934ca84700  1 --
	192.168.10.111:6805/29353 --> 192.168.10.112:0/3646 --
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.071631) v4 --
	0x558b506ba200 con 0
	 -9990> 2017-11-10 20:31:17.081113 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9201,
	last_complete=18323'3739, committed=1, applied=0)) v2 -- 
0x558be49aa900
	con 0
	 -9989> 2017-11-10 20:31:17.081158 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9201,
	last_complete=0'0, committed=0, applied=1)) v2 -- 0x558bd26eb8c0 
con 0
	 -9988> 2017-11-10 20:31:17.094350 7f934d285700  5 --
	192.168.10.111:6805/29353 >> 192.168.10.113:0/2071 
conn(0x558b86009000
	:6805 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=166 cs=1 
l=1).
	rx osd.14 seq 4900 0x558bd0b81000 osd_ping(ping e18323 stamp 
2017-11-10
	20:31:17.094115) v4
	 -9987> 2017-11-10 20:31:17.094369 7f934d285700  1 --
	192.168.10.111:6805/29353 <== osd.14 192.168.10.113:0/2071 4900 
====
	osd_ping(ping e18323 stamp 2017-11-10 20:31:17.094115) v4 ==== 
2004+0+0
	(822418717 0 0) 0x558bd0b81000 con 0x558b86009000
	 -9986> 2017-11-10 20:31:17.094397 7f934d285700  1 --
	192.168.10.111:6805/29353 --> 192.168.10.113:0/2071 --
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.094115) v4 --
	0x558b4edf5c00 con 0
	 -9985> 2017-11-10 20:31:17.094405 7f934da86700  5 --
	10.0.0.11:6805/29353 >> 10.0.0.13:0/2071 conn(0x558b85f41000 :6805
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=165 cs=1 l=1). rx
	osd.14 seq 4900 0x558b50a95a00 osd_ping(ping e18323 stamp 
2017-11-10
	20:31:17.094115) v4
	 -9984> 2017-11-10 20:31:17.094426 7f934da86700  1 --
	10.0.0.11:6805/29353 <== osd.14 10.0.0.13:0/2071 4900 ==== 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.094115) v4 ==== 2004+0+0 
(822418717 0
	0) 0x558b50a95a00 con 0x558b85f41000
	 -9983> 2017-11-10 20:31:17.094436 7f934da86700  1 --
	10.0.0.11:6805/29353 --> 10.0.0.13:0/2071 -- osd_ping(ping_reply 
e18323
	stamp 2017-11-10 20:31:17.094115) v4 -- 0x558b51372000 con 0
	 -9982> 2017-11-10 20:31:17.196597 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9202,
	last_complete=18323'3740, committed=1, applied=0)) v2 -- 
0x558b4ea44ac0
	con 0
	 -9981> 2017-11-10 20:31:17.196647 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9202,
	last_complete=0'0, committed=0, applied=1)) v2 -- 0x558be2ed01c0 
con 0
	 -9980> 2017-11-10 20:31:17.197148 7f934da86700  5 --
	10.0.0.11:6803/29353 >> 10.0.0.11:6808/29354 conn(0x558b86007800 
:-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=3 l=0). rx 
osd.7
	seq 3047 0x558be4da1000 MOSDECSubOpWrite(49.4s1 18323/18322
	ECSubWrite(tid=9203, reqid=client.2886221.0:1520, at_version=0'0,
	trim_to=18314'2200, roll_forward_to=18323'3740)) v2
	 -9979> 2017-11-10 20:31:17.197176 7f934da86700  1 --
	10.0.0.11:6803/29353 <== osd.7 10.0.0.11:6808/29354 3047 ====
	MOSDECSubOpWrite(49.4s1 18323/18322 ECSubWrite(tid=9203,
	reqid=client.2886221.0:1520, at_version=0'0, trim_to=18314'2200,
	roll_forward_to=18323'3740)) v2 ==== 838+0+0 (3324294031 0 0)
	0x558be4da1000 con 0x558b86007800
	 -9978> 2017-11-10 20:31:17.199926 7f934d285700  5 --
	10.0.0.11:6805/29353 >> 10.0.0.13:0/2078 conn(0x558b85e13800 :6805
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=315 cs=1 l=1). rx
	osd.2 seq 4871 0x558bd79e7c00 osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.199759) v4
	 -9977> 2017-11-10 20:31:17.199923 7f934ca84700  5 --
	192.168.10.111:6805/29353 >> 192.168.10.113:0/2078 
conn(0x558b85e12000
	:6805 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=316 cs=1 
l=1).
	rx osd.2 seq 4871 0x558b4c9efc00 osd_ping(ping e18323 stamp 
2017-11-10
	20:31:17.199759) v4
	 -9976> 2017-11-10 20:31:17.199953 7f934d285700  1 --
	10.0.0.11:6805/29353 <== osd.2 10.0.0.13:0/2078 4871 ==== 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.199759) v4 ==== 2004+0+0 
(183496747 0
	0) 0x558bd79e7c00 con 0x558b85e13800
	 -9975> 2017-11-10 20:31:17.199957 7f934ca84700  1 --
	192.168.10.111:6805/29353 <== osd.2 192.168.10.113:0/2078 4871 ====
	osd_ping(ping e18323 stamp 2017-11-10 20:31:17.199759) v4 ==== 
2004+0+0
	(183496747 0 0) 0x558b4c9efc00 con 0x558b85e12000
	 -9974> 2017-11-10 20:31:17.199965 7f934d285700  1 --
	10.0.0.11:6805/29353 --> 10.0.0.13:0/2078 -- osd_ping(ping_reply 
e18323
	stamp 2017-11-10 20:31:17.199759) v4 -- 0x558b4edf5c00 con 0
	 -9973> 2017-11-10 20:31:17.200004 7f934ca84700  1 --
	192.168.10.111:6805/29353 --> 192.168.10.113:0/2078 --
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.199759) v4 --
	0x558b506ba200 con 0
	 -9972> 2017-11-10 20:31:17.206354 7f934da86700  5 --
	10.0.0.11:6803/29353 >> 10.0.0.11:6808/29354 conn(0x558b86007800 
:-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=3 l=0). rx 
osd.7
	seq 3048 0x558b5807b000 MOSDECSubOpWrite(49.4s1 18323/18322
	ECSubWrite(tid=9204, reqid=client.2886221.0:1521, 
at_version=18323'3741,
	trim_to=18314'2200, roll_forward_to=18323'3740)) v2
	 -9971> 2017-11-10 20:31:17.206391 7f934da86700  1 --
	10.0.0.11:6803/29353 <== osd.7 10.0.0.11:6808/29354 3048 ====
	MOSDECSubOpWrite(49.4s1 18323/18322 ECSubWrite(tid=9204,
	reqid=client.2886221.0:1521, at_version=18323'3741, 
trim_to=18314'2200,
	roll_forward_to=18323'3740)) v2 ==== 2099384+0+0 (3125295307 0 0)
	0x558b5807b000 con 0x558b86007800
	 -9970> 2017-11-10 20:31:17.206529 7f9336299700  5 
write_log_and_missing
	with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
	writeout_from: 18323'3741, trimmed: , trimmed_dups: ,
	clear_divergent_priors: 0
	 -9969> 2017-11-10 20:31:17.221442 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9203,
	last_complete=18323'3740, committed=1, applied=0)) v2 -- 
0x558bd2cf2b40
	con 0
	 -9968> 2017-11-10 20:31:17.221499 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9203,
	last_complete=0'0, committed=0, applied=1)) v2 -- 0x558be4b72000 
con 0
	 -9967> 2017-11-10 20:31:17.311622 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9204,
	last_complete=18323'3741, committed=1, applied=0)) v2 -- 
0x558bd38aa000
	con 0
	 -9966> 2017-11-10 20:31:17.311697 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9204,
	last_complete=0'0, committed=0, applied=1)) v2 -- 0x558bd361e400 
con 0
	 -9965> 2017-11-10 20:31:17.352921 7f934da86700  5 --
	10.0.0.11:6803/29353 >> 10.0.0.11:6808/29354 conn(0x558b86007800 
:-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=3 l=0). rx 
osd.7
	seq 3049 0x558b85dcd800 MOSDECSubOpWrite(49.4s1 18323/18322
	ECSubWrite(tid=9205, reqid=client.2886221.0:1521, at_version=0'0,
	trim_to=18314'2200, roll_forward_to=18323'3741)) v2
	 -9964> 2017-11-10 20:31:17.352954 7f934da86700  1 --
	10.0.0.11:6803/29353 <== osd.7 10.0.0.11:6808/29354 3049 ====
	MOSDECSubOpWrite(49.4s1 18323/18322 ECSubWrite(tid=9205,
	reqid=client.2886221.0:1521, at_version=0'0, trim_to=18314'2200,
	roll_forward_to=18323'3741)) v2 ==== 838+0+0 (3904252529 0 0)
	0x558b85dcd800 con 0x558b86007800
	 -9963> 2017-11-10 20:31:17.362086 7f934da86700  5 --
	10.0.0.11:6803/29353 >> 10.0.0.11:6808/29354 conn(0x558b86007800 
:-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=3 l=0). rx 
osd.7
	seq 3050 0x558b85b32800 MOSDECSubOpWrite(49.4s1 18323/18322
	ECSubWrite(tid=9206, reqid=client.2886221.0:1522, 
at_version=18323'3742,
	trim_to=18314'2200, roll_forward_to=18323'3741)) v2
	 -9962> 2017-11-10 20:31:17.362122 7f934da86700  1 --
	10.0.0.11:6803/29353 <== osd.7 10.0.0.11:6808/29354 3050 ====
	MOSDECSubOpWrite(49.4s1 18323/18322 ECSubWrite(tid=9206,
	reqid=client.2886221.0:1522, at_version=18323'3742, 
trim_to=18314'2200,
	roll_forward_to=18323'3741)) v2 ==== 2099384+0+0 (753006316 0 0)
	0x558b85b32800 con 0x558b86007800
	 -9961> 2017-11-10 20:31:17.362198 7f9336299700  5 
write_log_and_missing
	with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
	writeout_from: 18323'3742, trimmed: , trimmed_dups: ,
	clear_divergent_priors: 0
	 -9960> 2017-11-10 20:31:17.375825 7f934da86700  5 --
	192.168.10.111:6802/29353 >> 192.168.10.113:0/3046860687
	conn(0x558b862aa000 :6802 
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH
	pgs=40 cs=1 l=1). rx client.2885010 seq 3125 0x558b87b48100
	osd_op(client.2885010.1:70478 16.5972d171
	16:8e8b4e9a:::rbd_header.1bfcc12ae8944a:head [watch ping cookie 3 
gen
	83] snapc 0=[] ondisk+write+known_if_redirected e18323) v4
	 -9959> 2017-11-10 20:31:17.375888 7f934da86700  1 --
	192.168.10.111:6802/29353 <== client.2885010 
192.168.10.113:0/3046860687
	3125 ==== osd_op(client.2885010.1:70478 16.5972d171
	16:8e8b4e9a:::rbd_header.1bfcc12ae8944a:head [watch ping cookie 3 
gen
	83] snapc 0=[] ondisk+write+known_if_redirected e18323) v4 ==== 
164+0+0
	(4051385142 0 0) 0x558b87b48100 con 0x558b862aa000
	 -9958> 2017-11-10 20:31:17.377160 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9205,
	last_complete=18323'3741, committed=1, applied=0)) v2 -- 
0x558be4ace240
	con 0
	 -9957> 2017-11-10 20:31:17.377214 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9205,
	last_complete=0'0, committed=0, applied=1)) v2 -- 0x558be4b726c0 
con 0
	 -9956> 2017-11-10 20:31:17.377213 7f9336299700  1 --
	192.168.10.111:6802/29353 --> 192.168.10.113:0/3046860687 --
	osd_op_reply(70478 rbd_header.1bfcc12ae8944a [watch ping cookie 3 
gen
	83] v0'0 uv247 ondisk = 0) v8 -- 0x558b898ca300 con 0
	 -9955> 2017-11-10 20:31:17.404100 7f934da86700  5 --
	10.0.0.11:6805/29353 >> 10.0.0.13:0/4707 conn(0x558b85ce9000 :6805
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx
	osd.12 seq 4909 0x558b51372000 osd_ping(ping e18323 stamp 
2017-11-10
	20:31:17.403890) v4
	 -9954> 2017-11-10 20:31:17.404100 7f934ca84700  5 --
	192.168.10.111:6805/29353 >> 192.168.10.113:0/4707 
conn(0x558b85cea800
	:6805 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 
l=1). rx
	osd.12 seq 4909 0x558b502e6600 osd_ping(ping e18323 stamp 
2017-11-10
	20:31:17.403890) v4
	 -9953> 2017-11-10 20:31:17.404128 7f934da86700  1 --
	10.0.0.11:6805/29353 <== osd.12 10.0.0.13:0/4707 4909 ==== 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.403890) v4 ==== 2004+0+0 
(2724503982 0
	0) 0x558b51372000 con 0x558b85ce9000
	 -9952> 2017-11-10 20:31:17.404128 7f934ca84700  1 --
	192.168.10.111:6805/29353 <== osd.12 192.168.10.113:0/4707 4909 
====
	osd_ping(ping e18323 stamp 2017-11-10 20:31:17.403890) v4 ==== 
2004+0+0
	(2724503982 0 0) 0x558b502e6600 con 0x558b85cea800
	 -9951> 2017-11-10 20:31:17.404142 7f934da86700  1 --
	10.0.0.11:6805/29353 --> 10.0.0.13:0/4707 -- osd_ping(ping_reply 
e18323
	stamp 2017-11-10 20:31:17.403890) v4 -- 0x558bd043c600 con 0
	 -9950> 2017-11-10 20:31:17.404164 7f934ca84700  1 --
	192.168.10.111:6805/29353 --> 192.168.10.113:0/4707 --
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.403890) v4 --
	0x558b506ba200 con 0
	 -9949> 2017-11-10 20:31:17.437672 7f9334a96700  5 osd.0 18323
	heartbeat: osd_stat(3166 GB used, 4285 GB avail, 7451 GB total, 
peers
	[1,2,4,5,9,10,11,12,13,14] op hist [0,0,0,0,0,0,0,1])
	 -9948> 2017-11-10 20:31:17.437689 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.12:6808/2067 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558bd7e9ec00 con 0
	 -9947> 2017-11-10 20:31:17.437734 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.112:6808/2067 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558bd8893c00 con 0
	 -9946> 2017-11-10 20:31:17.437751 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.13:6803/2078 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558bc9e18400 con 0
	 -9945> 2017-11-10 20:31:17.437765 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.113:6803/2078 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558bce5db000 con 0
	 -9944> 2017-11-10 20:31:17.437790 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.12:6804/2069 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558bd6777200 con 0
	 -9943> 2017-11-10 20:31:17.437799 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.112:6804/2069 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558bd6777e00 con 0
	 -9942> 2017-11-10 20:31:17.437812 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.13:6807/2068 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558bcf9b5a00 con 0
	 -9941> 2017-11-10 20:31:17.437822 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.113:6808/2068 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558bce160c00 con 0
	 -9940> 2017-11-10 20:31:17.437833 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.12:6805/3646 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558bd9383200 con 0
	 -9939> 2017-11-10 20:31:17.437844 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.112:6805/3646 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558bca1f4c00 con 0
	 -9938> 2017-11-10 20:31:17.437857 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.12:6801/2063 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558b6eb93400 con 0
	 -9937> 2017-11-10 20:31:17.437868 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.112:6803/2063 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558bcd103800 con 0
	 -9936> 2017-11-10 20:31:17.437879 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.12:6809/18917 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558bb57fac00 con 0
	 -9935> 2017-11-10 20:31:17.437892 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.112:6809/18917 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558bca018200 con 0
	 -9934> 2017-11-10 20:31:17.437903 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.13:6801/4707 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558b51b44c00 con 0
	 -9933> 2017-11-10 20:31:17.437915 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.113:6802/4707 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558b8ab65800 con 0
	 -9932> 2017-11-10 20:31:17.437927 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.13:6805/26512 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558bd1410400 con 0
	 -9931> 2017-11-10 20:31:17.437939 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.113:6806/26512 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558bcc0a4c00 con 0
	 -9930> 2017-11-10 20:31:17.437951 7f9334a96700  1 -- 
10.0.0.11:0/29353
	--> 10.0.0.13:6809/2071 -- osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.437686) v4 -- 0x558bd556b600 con 0
	 -9929> 2017-11-10 20:31:17.437964 7f9334a96700  1 --
	192.168.10.111:0/29353 --> 192.168.10.113:6810/2071 -- 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.437686) v4 -- 0x558bcd4c3200 con 0
	 -9928> 2017-11-10 20:31:17.438075 7f934ca84700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.13:6803/2078 conn(0x558b86112000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=107 cs=1 l=1). rx
	osd.2 seq 4832 0x558b4c9efc00 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9927> 2017-11-10 20:31:17.438101 7f934da86700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.12:6808/2067 conn(0x558b8614d000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=172 cs=1 l=1). rx
	osd.1 seq 4832 0x558b51372600 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9926> 2017-11-10 20:31:17.438111 7f934ca84700  1 -- 
10.0.0.11:0/29353
	<== osd.2 10.0.0.13:6803/2078 4832 ==== osd_ping(ping_reply e18323 
stamp
	2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558b4c9efc00 con 0x558b86112000
	 -9925> 2017-11-10 20:31:17.438123 7f934da86700  1 -- 
10.0.0.11:0/29353
	<== osd.1 10.0.0.12:6808/2067 4832 ==== osd_ping(ping_reply e18323 
stamp
	2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558b51372600 con 0x558b8614d000
	 -9924> 2017-11-10 20:31:17.438125 7f934d285700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.12:6801/2063 conn(0x558b8614a000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95 cs=1 l=1). rx
	osd.10 seq 4832 0x558bd0b81000 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9923> 2017-11-10 20:31:17.438152 7f934d285700  1 -- 
10.0.0.11:0/29353
	<== osd.10 10.0.0.12:6801/2063 4832 ==== osd_ping(ping_reply e18323
	stamp 2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558bd0b81000 con 0x558b8614a000
	 -9922> 2017-11-10 20:31:17.438176 7f934ca84700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.12:6809/18917 conn(0x558b86181000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx
	osd.11 seq 4832 0x558b4c9efc00 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9921> 2017-11-10 20:31:17.438192 7f934ca84700  1 -- 
10.0.0.11:0/29353
	<== osd.11 10.0.0.12:6809/18917 4832 ==== osd_ping(ping_reply 
e18323
	stamp 2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558b4c9efc00 con 0x558b86181000
	 -9920> 2017-11-10 20:31:17.438195 7f934d285700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.13:6801/4707 conn(0x558b86115000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx
	osd.12 seq 4832 0x558bd0b81000 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9919> 2017-11-10 20:31:17.438198 7f934da86700  5 --
	192.168.10.111:0/29353 >> 192.168.10.112:6809/18917 
conn(0x558b86182800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). 
rx
	osd.11 seq 4832 0x558b51372600 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9918> 2017-11-10 20:31:17.438206 7f934d285700  1 -- 
10.0.0.11:0/29353
	<== osd.12 10.0.0.13:6801/4707 4832 ==== osd_ping(ping_reply e18323
	stamp 2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558bd0b81000 con 0x558b86115000
	 -9917> 2017-11-10 20:31:17.438213 7f934da86700  1 --
	192.168.10.111:0/29353 <== osd.11 192.168.10.112:6809/18917 4832 
====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558b51372600 con 0x558b86182800
	 -9916> 2017-11-10 20:31:17.438233 7f934ca84700  5 --
	192.168.10.111:0/29353 >> 192.168.10.113:6808/2068 
conn(0x558b8616e800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=105 cs=1 
l=1). rx
	osd.5 seq 4832 0x558b4c9efc00 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9915> 2017-11-10 20:31:17.438249 7f934da86700  5 --
	192.168.10.111:0/29353 >> 192.168.10.113:6803/2078 
conn(0x558b86113800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=107 cs=1 
l=1). rx
	osd.2 seq 4832 0x558b51372600 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9914> 2017-11-10 20:31:17.438250 7f934d285700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.13:6807/2068 conn(0x558b8616d000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=105 cs=1 l=1). rx
	osd.5 seq 4832 0x558bd0b81000 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9913> 2017-11-10 20:31:17.438251 7f934ca84700  1 --
	192.168.10.111:0/29353 <== osd.5 192.168.10.113:6808/2068 4832 ====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558b4c9efc00 con 0x558b8616e800
	 -9912> 2017-11-10 20:31:17.438258 7f934da86700  1 --
	192.168.10.111:0/29353 <== osd.2 192.168.10.113:6803/2078 4832 ====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558b51372600 con 0x558b86113800
	 -9911> 2017-11-10 20:31:17.438262 7f934d285700  1 -- 
10.0.0.11:0/29353
	<== osd.5 10.0.0.13:6807/2068 4832 ==== osd_ping(ping_reply e18323 
stamp
	2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558bd0b81000 con 0x558b8616d000
	 -9910> 2017-11-10 20:31:17.438286 7f934da86700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.12:6805/3646 conn(0x558b8617e000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx 
osd.9
	seq 4832 0x558b51372600 osd_ping(ping_reply e18323 stamp 2017-11-10
	20:31:17.437686) v4
	 -9909> 2017-11-10 20:31:17.438283 7f934ca84700  5 --
	192.168.10.111:0/29353 >> 192.168.10.113:6802/4707 
conn(0x558b86116800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). 
rx
	osd.12 seq 4832 0x558b4c9efc00 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9908> 2017-11-10 20:31:17.438296 7f934da86700  1 -- 
10.0.0.11:0/29353
	<== osd.9 10.0.0.12:6805/3646 4832 ==== osd_ping(ping_reply e18323 
stamp
	2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558b51372600 con 0x558b8617e000
	 -9907> 2017-11-10 20:31:17.438295 7f934d285700  5 --
	192.168.10.111:0/29353 >> 192.168.10.112:6808/2067 
conn(0x558b8614e800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=172 cs=1 
l=1). rx
	osd.1 seq 4832 0x558bd0b81000 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9906> 2017-11-10 20:31:17.438298 7f934ca84700  1 --
	192.168.10.111:0/29353 <== osd.12 192.168.10.113:6802/4707 4832 
====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558b4c9efc00 con 0x558b86116800
	 -9905> 2017-11-10 20:31:17.438313 7f934da86700  5 --
	192.168.10.111:0/29353 >> 192.168.10.113:6806/26512 
conn(0x558b8616b800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=147 cs=1 
l=1). rx
	osd.13 seq 4832 0x558b51372600 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9904> 2017-11-10 20:31:17.438310 7f934d285700  1 --
	192.168.10.111:0/29353 <== osd.1 192.168.10.112:6808/2067 4832 ====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558bd0b81000 con 0x558b8614e800
	 -9903> 2017-11-10 20:31:17.438324 7f934da86700  1 --
	192.168.10.111:0/29353 <== osd.13 192.168.10.113:6806/26512 4832 
====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558b51372600 con 0x558b8616b800
	 -9902> 2017-11-10 20:31:17.438328 7f934ca84700  5 --
	192.168.10.111:0/29353 >> 192.168.10.112:6803/2063 
conn(0x558b8614b800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=95 cs=1 l=1). 
rx
	osd.10 seq 4832 0x558b4c9efc00 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9901> 2017-11-10 20:31:17.438336 7f934d285700  5 --
	192.168.10.111:0/29353 >> 192.168.10.112:6805/3646 
conn(0x558b8617f800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). 
rx
	osd.9 seq 4832 0x558bd0b81000 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9900> 2017-11-10 20:31:17.438342 7f934da86700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.12:6804/2069 conn(0x558b86267000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=118 cs=1 l=1). rx
	osd.4 seq 4832 0x558b51372600 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9899> 2017-11-10 20:31:17.438345 7f934ca84700  1 --
	192.168.10.111:0/29353 <== osd.10 192.168.10.112:6803/2063 4832 
====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558b4c9efc00 con 0x558b8614b800
	 -9898> 2017-11-10 20:31:17.438347 7f934d285700  1 --
	192.168.10.111:0/29353 <== osd.9 192.168.10.112:6805/3646 4832 ====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558bd0b81000 con 0x558b8617f800
	 -9897> 2017-11-10 20:31:17.438350 7f934da86700  1 -- 
10.0.0.11:0/29353
	<== osd.4 10.0.0.12:6804/2069 4832 ==== osd_ping(ping_reply e18323 
stamp
	2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558b51372600 con 0x558b86267000
	 -9896> 2017-11-10 20:31:17.438374 7f934da86700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.13:6809/2071 conn(0x558b86120000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=268 cs=1 l=1). rx
	osd.14 seq 4832 0x558b51372600 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9895> 2017-11-10 20:31:17.438375 7f934d285700  5 --
	192.168.10.111:0/29353 >> 192.168.10.112:6804/2069 
conn(0x558b86268800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=118 cs=1 
l=1). rx
	osd.4 seq 4832 0x558bd0b81000 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9894> 2017-11-10 20:31:17.438389 7f934ca84700  5 -- 
10.0.0.11:0/29353
	>> 10.0.0.13:6805/26512 conn(0x558b8616a000 :-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=147 cs=1 l=1). rx
	osd.13 seq 4832 0x558b4c9efc00 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9893> 2017-11-10 20:31:17.438391 7f934da86700  1 -- 
10.0.0.11:0/29353
	<== osd.14 10.0.0.13:6809/2071 4832 ==== osd_ping(ping_reply e18323
	stamp 2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558b51372600 con 0x558b86120000
	 -9892> 2017-11-10 20:31:17.438403 7f934d285700  1 --
	192.168.10.111:0/29353 <== osd.4 192.168.10.112:6804/2069 4832 ====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558bd0b81000 con 0x558b86268800
	 -9891> 2017-11-10 20:31:17.438402 7f934ca84700  1 -- 
10.0.0.11:0/29353
	<== osd.13 10.0.0.13:6805/26512 4832 ==== osd_ping(ping_reply 
e18323
	stamp 2017-11-10 20:31:17.437686) v4 ==== 2004+0+0 (2089100414 0 0)
	0x558b4c9efc00 con 0x558b8616a000
	 -9890> 2017-11-10 20:31:17.438445 7f934d285700  5 --
	192.168.10.111:0/29353 >> 192.168.10.113:6810/2071 
conn(0x558b86121800
	:-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=268 cs=1 
l=1). rx
	osd.14 seq 4832 0x558bd0b81000 osd_ping(ping_reply e18323 stamp
	2017-11-10 20:31:17.437686) v4
	 -9889> 2017-11-10 20:31:17.438456 7f934d285700  1 --
	192.168.10.111:0/29353 <== osd.14 192.168.10.113:6810/2071 4832 
====
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.437686) v4 
====
	2004+0+0 (2089100414 0 0) 0x558bd0b81000 con 0x558b86121800
	 -9888> 2017-11-10 20:31:17.478607 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9206,
	last_complete=18323'3742, committed=1, applied=0)) v2 -- 
0x558b4f2d6fc0
	con 0
	 -9887> 2017-11-10 20:31:17.478656 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9206,
	last_complete=0'0, committed=0, applied=1)) v2 -- 0x558bc951eb40 
con 0
	 -9886> 2017-11-10 20:31:17.500945 7f934da86700  5 --
	10.0.0.11:6803/29353 >> 10.0.0.11:6808/29354 conn(0x558b86007800 
:-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=3 l=0). rx 
osd.7
	seq 3051 0x558be416c000 MOSDECSubOpWrite(49.4s1 18323/18322
	ECSubWrite(tid=9207, reqid=client.2886221.0:1522, at_version=0'0,
	trim_to=18314'2200, roll_forward_to=18323'3742)) v2
	 -9885> 2017-11-10 20:31:17.500974 7f934da86700  1 --
	10.0.0.11:6803/29353 <== osd.7 10.0.0.11:6808/29354 3051 ====
	MOSDECSubOpWrite(49.4s1 18323/18322 ECSubWrite(tid=9207,
	reqid=client.2886221.0:1522, at_version=0'0, trim_to=18314'2200,
	roll_forward_to=18323'3742)) v2 ==== 838+0+0 (810262189 0 0)
	0x558be416c000 con 0x558b86007800
	 -9884> 2017-11-10 20:31:17.510059 7f934da86700  5 --
	10.0.0.11:6803/29353 >> 10.0.0.11:6808/29354 conn(0x558b86007800 
:-1
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=3 l=0). rx 
osd.7
	seq 3052 0x558b5807c000 MOSDECSubOpWrite(49.4s1 18323/18322
	ECSubWrite(tid=9208, reqid=client.2886221.0:1523, 
at_version=18323'3743,
	trim_to=18314'2200, roll_forward_to=18323'3742)) v2
	 -9883> 2017-11-10 20:31:17.510097 7f934da86700  1 --
	10.0.0.11:6803/29353 <== osd.7 10.0.0.11:6808/29354 3052 ====
	MOSDECSubOpWrite(49.4s1 18323/18322 ECSubWrite(tid=9208,
	reqid=client.2886221.0:1523, at_version=18323'3743, 
trim_to=18314'2200,
	roll_forward_to=18323'3742)) v2 ==== 2099384+0+0 (4008278039 0 0)
	0x558b5807c000 con 0x558b86007800
	 -9882> 2017-11-10 20:31:17.510164 7f9336299700  5 
write_log_and_missing
	with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
	writeout_from: 18323'3743, trimmed: , trimmed_dups: ,
	clear_divergent_priors: 0
	 -9881> 2017-11-10 20:31:17.527328 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9207,
	last_complete=18323'3742, committed=1, applied=0)) v2 -- 
0x558be331bf80
	con 0
	 -9880> 2017-11-10 20:31:17.527451 7f93402ad700  1 --
	10.0.0.11:6803/29353 --> 10.0.0.11:6808/29354 --
	MOSDECSubOpWriteReply(49.4s0 18323/18322 ECSubWriteReply(tid=9207,
	last_complete=0'0, committed=0, applied=1)) v2 -- 0x558be2ffcac0 
con 0
	 -9879> 2017-11-10 20:31:17.572258 7f934d285700  5 --
	10.0.0.11:6805/29353 >> 10.0.0.12:0/3646 conn(0x558b85c3a800 :6805
	s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx 
osd.9
	seq 4840 0x558bd0b81000 osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.572252) v4
	 -9878> 2017-11-10 20:31:17.572258 7f934ca84700  5 --
	192.168.10.111:6805/29353 >> 192.168.10.112:0/3646 
conn(0x558b85c39000
	:6805 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 
l=1). rx
	osd.9 seq 4840 0x558b4c9efc00 osd_ping(ping e18323 stamp 2017-11-10
	20:31:17.572252) v4
	 -9877> 2017-11-10 20:31:17.572309 7f934d285700  1 --
	10.0.0.11:6805/29353 <== osd.9 10.0.0.12:0/3646 4840 ==== 
osd_ping(ping
	e18323 stamp 2017-11-10 20:31:17.572252) v4 ==== 2004+0+0 
(2754440270 0
	0) 0x558bd0b81000 con 0x558b85c3a800
	 -9876> 2017-11-10 20:31:17.572314 7f934ca84700  1 --
	192.168.10.111:6805/29353 <== osd.9 192.168.10.112:0/3646 4840 ====
	osd_ping(ping e18323 stamp 2017-11-10 20:31:17.572252) v4 ==== 
2004+0+0
	(2754440270 0 0) 0x558b4c9efc00 con 0x558b85c39000
	 -9875> 2017-11-10 20:31:17.572328 7f934d285700  1 --
	10.0.0.11:6805/29353 --> 10.0.0.12:0/3646 -- osd_ping(ping_reply 
e18323
	stamp 2017-11-10 20:31:17.572252) v4 -- 0x558bd8893c00 con 0
	 -9874> 2017-11-10 20:31:17.572357 7f934ca84700  1 --
	192.168.10.111:6805/29353 --> 192.168.10.112:0/3646 --
	osd_ping(ping_reply e18323 stamp 2017-11-10 20:31:17.572252) v4 --
	0x558b506ba200 con 0
	
	_______________________________________________
	ceph-users mailing list
	ceph-users at lists.ceph.com
	http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> 
	





More information about the ceph-users mailing list