[ceph-users] 答复: Question about BUG #11332
xuxuehan at 360.cn
Mon Dec 4 19:29:33 PST 2017
Thanks for your reply, greg:-)
Monitor processes its requests in the main dispatch loop, however, the "PAXOS COMMIT" transaction is executed by another thread MonitorDBStore::io_work, so I think it could be possible that they run concurrently. On the other hand, although the commit transaction is executed in another thread, the PaxosService's state, by which I mean "active" or not, is modified in the main dispatch loop. So we now think that it should work fine if we make "send_incremental" method check if the OSDMonitor is active and put a callback in the "wait_for_active" queue if it's not.
Is this right?
发件人: Gregory Farnum [mailto:gfarnum at redhat.com]
发送时间: 2017年12月5日 5:48
抄送: ceph-users at lists.ceph.com; 陈玉鹏
主题: Re: [ceph-users] Question about BUG #11332
On Thu, Nov 23, 2017 at 1:55 AM 许雪寒 <xuxuehan at 360.cn> wrote:
We also encountered this problem: http://tracker.ceph.com/issues/11332. And we found that this seems to be caused by the lack of mutual exclusion between applying "trim" and handling subscriptions. Since "build_incremental" operations doesn't go through the "PAXOS" procedure, and applying "trim" contains two phases, which are modifying "mondbstore" and updating "cached_first_committed", there could be a chance for "send_incremental" operations to happen between them. What's more, "build_incremental" operations also contain two phases, getting "cached_first_committed" and getting actual incrementals for MonDBStore. So, if "build_incremental" do happens concurrently with applying "trim", it could get an out-dated "cached_first_committed" and try to read a full map whose already trimmed.
Is this right?
I don't think this is right. Keep in mind that the monitors are basically a single-threaded event-driven machine. Both trimming and building incrementals happen in direct response to receiving messages, in the main dispatch loop, and while trimming is happening the PaxosService is not readable. So it won't be invoking build_incremental() and they won't run concurrently.
If it is, we think maybe all “READ” operations in monitor should be synchronized with paxos commit. Right? Should some kind of read-write locking mechanism be used here?
ceph-users mailing list
ceph-users at lists.ceph.com
More information about the ceph-users