Nov 25 10:40:10 storage4 ceph-osd[3915]: /build/ceph-12.2.8/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f714161c700 time 2018-11-25 10:40:10.954372
Nov 25 10:40:10 storage4 ceph-osd[3915]: /build/ceph-12.2.8/src/os/bluestore/BlueStore.cc: 9150: FAILED assert(0 == "unexpected error")
Nov 25 10:40:10 storage4 ceph-osd[3915]: 2018-11-25 10:40:10.954334 7f714161c700 -1 bluestore(/var/lib/ceph/osd/ceph-26) _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
Nov 25 10:40:10 storage4 ceph-osd[3915]:  ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
Nov 25 10:40:10 storage4 ceph-osd[3915]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5586123ad412]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x7ac) [0x558612261b8c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  3: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x55861226409e]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  4: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>)+0x259) [0x558612114cd9]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  5: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, ThreadPool::TPHandle&)+0x4a3) [0x558611d88b23]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  6: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0x11c) [0x558611df360c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  7: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb9) [0x5586123b4319]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  8: (ThreadPool::WorkThread::entry()+0x10) [0x5586123b5420]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  9: (()+0x76ba) [0x7f7164fb46ba]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  10: (clone()+0x6d) [0x7f716402b41d]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Nov 25 10:40:10 storage4 ceph-osd[3915]: 2018-11-25 10:40:10.956816 7f714161c700 -1 /build/ceph-12.2.8/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f714161c700 time 2018-11-25 10:40:10.954372
Nov 25 10:40:10 storage4 ceph-osd[3915]: /build/ceph-12.2.8/src/os/bluestore/BlueStore.cc: 9150: FAILED assert(0 == "unexpected error")
Nov 25 10:40:10 storage4 ceph-osd[3915]:  ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
Nov 25 10:40:10 storage4 ceph-osd[3915]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5586123ad412]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x7ac) [0x558612261b8c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  3: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x55861226409e]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  4: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>)+0x259) [0x558612114cd9]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  5: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, ThreadPool::TPHandle&)+0x4a3) [0x558611d88b23]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  6: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0x11c) [0x558611df360c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  7: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb9) [0x5586123b4319]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  8: (ThreadPool::WorkThread::entry()+0x10) [0x5586123b5420]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  9: (()+0x76ba) [0x7f7164fb46ba]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  10: (clone()+0x6d) [0x7f716402b41d]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Nov 25 10:40:10 storage4 ceph-osd[3915]:     -2> 2018-11-25 10:40:10.954334 7f714161c700 -1 bluestore(/var/lib/ceph/osd/ceph-26) _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)
Nov 25 10:40:10 storage4 ceph-osd[3915]:      0> 2018-11-25 10:40:10.956816 7f714161c700 -1 /build/ceph-12.2.8/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f714161c700 time 2018-11-25 10:40:10.954372
Nov 25 10:40:10 storage4 ceph-osd[3915]: /build/ceph-12.2.8/src/os/bluestore/BlueStore.cc: 9150: FAILED assert(0 == "unexpected error")
Nov 25 10:40:10 storage4 ceph-osd[3915]:  ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
Nov 25 10:40:10 storage4 ceph-osd[3915]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5586123ad412]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x7ac) [0x558612261b8c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  3: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x55861226409e]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  4: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>)+0x259) [0x558612114cd9]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  5: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, ThreadPool::TPHandle&)+0x4a3) [0x558611d88b23]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  6: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0x11c) [0x558611df360c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  7: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb9) [0x5586123b4319]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  8: (ThreadPool::WorkThread::entry()+0x10) [0x5586123b5420]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  9: (()+0x76ba) [0x7f7164fb46ba]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  10: (clone()+0x6d) [0x7f716402b41d]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Nov 25 10:40:10 storage4 ceph-osd[3915]: *** Caught signal (Aborted) **
Nov 25 10:40:10 storage4 ceph-osd[3915]:  in thread 7f714161c700 thread_name:tp_osd_disk
Nov 25 10:40:10 storage4 ceph-osd[3915]:  ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
Nov 25 10:40:10 storage4 ceph-osd[3915]:  1: (()+0xa87324) [0x55861236a324]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  2: (()+0x11390) [0x7f7164fbe390]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  3: (gsignal()+0x38) [0x7f7163f59428]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  4: (abort()+0x16a) [0x7f7163f5b02a]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5586123ad59e]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  6: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x7ac) [0x558612261b8c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  7: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x55861226409e]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  8: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>)+0x259) [0x558612114cd9]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  9: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, ThreadPool::TPHandle&)+0x4a3) [0x558611d88b23]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  10: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0x11c) [0x558611df360c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  11: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb9) [0x5586123b4319]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  12: (ThreadPool::WorkThread::entry()+0x10) [0x5586123b5420]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  13: (()+0x76ba) [0x7f7164fb46ba]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  14: (clone()+0x6d) [0x7f716402b41d]
Nov 25 10:40:10 storage4 ceph-osd[3915]: 2018-11-25 10:40:10.981675 7f714161c700 -1 *** Caught signal (Aborted) **
Nov 25 10:40:10 storage4 ceph-osd[3915]:  in thread 7f714161c700 thread_name:tp_osd_disk
Nov 25 10:40:10 storage4 ceph-osd[3915]:  ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
Nov 25 10:40:10 storage4 ceph-osd[3915]:  1: (()+0xa87324) [0x55861236a324]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  2: (()+0x11390) [0x7f7164fbe390]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  3: (gsignal()+0x38) [0x7f7163f59428]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  4: (abort()+0x16a) [0x7f7163f5b02a]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5586123ad59e]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  6: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x7ac) [0x558612261b8c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  7: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x55861226409e]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  8: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>)+0x259) [0x558612114cd9]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  9: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, ThreadPool::TPHandle&)+0x4a3) [0x558611d88b23]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  10: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0x11c) [0x558611df360c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  11: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb9) [0x5586123b4319]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  12: (ThreadPool::WorkThread::entry()+0x10) [0x5586123b5420]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  13: (()+0x76ba) [0x7f7164fb46ba]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  14: (clone()+0x6d) [0x7f716402b41d]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Nov 25 10:40:10 storage4 ceph-osd[3915]:      0> 2018-11-25 10:40:10.981675 7f714161c700 -1 *** Caught signal (Aborted) **
Nov 25 10:40:10 storage4 ceph-osd[3915]:  in thread 7f714161c700 thread_name:tp_osd_disk
Nov 25 10:40:10 storage4 ceph-osd[3915]:  ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
Nov 25 10:40:10 storage4 ceph-osd[3915]:  1: (()+0xa87324) [0x55861236a324]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  2: (()+0x11390) [0x7f7164fbe390]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  3: (gsignal()+0x38) [0x7f7163f59428]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  4: (abort()+0x16a) [0x7f7163f5b02a]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5586123ad59e]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  6: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x7ac) [0x558612261b8c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  7: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x55861226409e]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  8: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>)+0x259) [0x558612114cd9]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  9: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, ThreadPool::TPHandle&)+0x4a3) [0x558611d88b23]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  10: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0x11c) [0x558611df360c]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  11: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb9) [0x5586123b4319]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  12: (ThreadPool::WorkThread::entry()+0x10) [0x5586123b5420]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  13: (()+0x76ba) [0x7f7164fb46ba]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  14: (clone()+0x6d) [0x7f716402b41d]
Nov 25 10:40:10 storage4 ceph-osd[3915]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Nov 25 10:40:11 storage4 systemd[1]: ceph-osd@26.service: Main process exited, code=killed, status=6/ABRT
Nov 25 10:40:11 storage4 systemd[1]: ceph-osd@26.service: Unit entered failed state.
Nov 25 10:40:11 storage4 systemd[1]: ceph-osd@26.service: Failed with result 'signal'.
Nov 25 10:40:31 storage4 systemd[1]: ceph-osd@26.service: Service hold-off time over, scheduling restart.
Nov 25 10:40:31 storage4 systemd[1]: Stopped Ceph object storage daemon osd.26.
Nov 25 10:40:31 storage4 systemd[1]: Starting Ceph object storage daemon osd.26...
Nov 25 10:40:31 storage4 systemd[1]: Started Ceph object storage daemon osd.26.