[ceph-users] OSD fails to startup with bluestore "direct_read_unaligned (5) Input/output error"

Alexandre Gosset alexandre at k-net.pro
Mon Oct 8 15:50:54 PDT 2018


Hi,

We are experiencing a recurrent problem with some OSD that fails to
startup after a crash. Here it happened with an OSD during recovery and
this is a very annoying bug, because recovery takes time (small
increment of crush weight) and the way I found to fix this is to use
ceph-volume lvm zap /var/lib/ceph/osd/ceph-$i/block, remove the OSD, and
create it again. Thus needing to start the recovery all over again.

Is there another way to fix this ?

# /usr/bin/ceph-osd -d -i 40 --pid-file /var/run/ceph/osd.40.pid -c
/etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup ceph
2018-10-09 00:33:23.025913 7f91c7ae2e00  0 set uid:gid to 64045:64045
(ceph:ceph)
2018-10-09 00:33:23.025934 7f91c7ae2e00  0 ceph version 12.2.8
(ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable), process
ceph-osd, pid 322330
starting osd.40 at - osd_data /var/lib/ceph/osd/ceph-40
/var/lib/ceph/osd/ceph-40/journal
2018-10-09 00:33:23.203930 7f91c7ae2e00  0 load: jerasure load: lrc
load: isa
2018-10-09 00:33:23.204022 7f91c7ae2e00  1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
2018-10-09 00:33:23.204032 7f91c7ae2e00  1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
2018-10-09 00:33:23.204336 7f91c7ae2e00  1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
2018-10-09 00:33:23.204766 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes max 0.5 < ratio 0.99
2018-10-09 00:33:23.204790 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes cache_size
1073741824 meta 0.5 kv 0.5 data 0
2018-10-09 00:33:23.204796 7f91c7ae2e00  1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) close
2018-10-09 00:33:23.499876 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _mount path /var/lib/ceph/osd/ceph-40
2018-10-09 00:33:23.500355 7f91c7ae2e00  1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
2018-10-09 00:33:23.500368 7f91c7ae2e00  1 bdev(0x558d31b68d80
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
2018-10-09 00:33:23.500583 7f91c7ae2e00  1 bdev(0x558d31b68d80
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
2018-10-09 00:33:23.500884 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes max 0.5 < ratio 0.99
2018-10-09 00:33:23.500900 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes cache_size
1073741824 meta 0.5 kv 0.5 data 0
2018-10-09 00:33:23.501051 7f91c7ae2e00  1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
2018-10-09 00:33:23.501061 7f91c7ae2e00  1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
2018-10-09 00:33:23.501263 7f91c7ae2e00  1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
2018-10-09 00:33:23.501277 7f91c7ae2e00  1 bluefs add_block_device bdev
1 path /var/lib/ceph/osd/ceph-40/block size 1.81TiB
2018-10-09 00:33:23.501310 7f91c7ae2e00  1 bluefs mount
2018-10-09 00:33:23.589465 7f91c7ae2e00  0  set rocksdb option
compaction_readahead_size = 2097152
2018-10-09 00:33:23.589482 7f91c7ae2e00  0  set rocksdb option
compression = kNoCompression
2018-10-09 00:33:23.589487 7f91c7ae2e00  0  set rocksdb option
max_write_buffer_number = 4
2018-10-09 00:33:23.589490 7f91c7ae2e00  0  set rocksdb option
min_write_buffer_number_to_merge = 1
2018-10-09 00:33:23.589494 7f91c7ae2e00  0  set rocksdb option
recycle_log_file_num = 4
2018-10-09 00:33:23.589496 7f91c7ae2e00  0  set rocksdb option
writable_file_max_buffer_size = 0
2018-10-09 00:33:23.589500 7f91c7ae2e00  0  set rocksdb option
write_buffer_size = 268435456
2018-10-09 00:33:23.589517 7f91c7ae2e00  0  set rocksdb option
compaction_readahead_size = 2097152
2018-10-09 00:33:23.589520 7f91c7ae2e00  0  set rocksdb option
compression = kNoCompression
2018-10-09 00:33:23.589523 7f91c7ae2e00  0  set rocksdb option
max_write_buffer_number = 4
2018-10-09 00:33:23.589526 7f91c7ae2e00  0  set rocksdb option
min_write_buffer_number_to_merge = 1
2018-10-09 00:33:23.589528 7f91c7ae2e00  0  set rocksdb option
recycle_log_file_num = 4
2018-10-09 00:33:23.589531 7f91c7ae2e00  0  set rocksdb option
writable_file_max_buffer_size = 0
2018-10-09 00:33:23.589533 7f91c7ae2e00  0  set rocksdb option
write_buffer_size = 268435456
2018-10-09 00:33:23.589656 7f91c7ae2e00  4 rocksdb: RocksDB version: 5.4.0

2018-10-09 00:33:23.589667 7f91c7ae2e00  4 rocksdb: Git sha
rocksdb_build_git_sha:@0@
2018-10-09 00:33:23.589670 7f91c7ae2e00  4 rocksdb: Compile date Aug 30 2018
2018-10-09 00:33:23.589671 7f91c7ae2e00  4 rocksdb: DB SUMMARY

2018-10-09 00:33:23.589706 7f91c7ae2e00  4 rocksdb: CURRENT file:  CURRENT

2018-10-09 00:33:23.589707 7f91c7ae2e00  4 rocksdb: IDENTITY file:  IDENTITY

2018-10-09 00:33:23.589709 7f91c7ae2e00  4 rocksdb: MANIFEST file: 
MANIFEST-000011 size: 51827 Bytes

2018-10-09 00:33:23.589712 7f91c7ae2e00  4 rocksdb: SST files in db dir,
Total Num: 28, files: 000260.sst 000261.sst 000262.sst 000274.sst
000275.sst 000276.sst 000277.sst 000286.sst 000287.sst

2018-10-09 00:33:23.589714 7f91c7ae2e00  4 rocksdb: Write Ahead Log file
in db: 000310.log size: 258554009 ; 000312.log size: 256884429 ;

2018-10-09 00:33:23.589715 7f91c7ae2e00  4
rocksdb:                         Options.error_if_exists: 0
2018-10-09 00:33:23.589716 7f91c7ae2e00  4
rocksdb:                       Options.create_if_missing: 0
2018-10-09 00:33:23.589717 7f91c7ae2e00  4
rocksdb:                         Options.paranoid_checks: 1
2018-10-09 00:33:23.589718 7f91c7ae2e00  4
rocksdb:                                     Options.env: 0x558d31d42280
2018-10-09 00:33:23.589719 7f91c7ae2e00  4
rocksdb:                                Options.info_log: 0x558d31d42a00
2018-10-09 00:33:23.589720 7f91c7ae2e00  4
rocksdb:                          Options.max_open_files: -1
2018-10-09 00:33:23.589721 7f91c7ae2e00  4 rocksdb:               
Options.max_file_opening_threads: 16
2018-10-09 00:33:23.589722 7f91c7ae2e00  4
rocksdb:                               Options.use_fsync: 0
2018-10-09 00:33:23.589723 7f91c7ae2e00  4
rocksdb:                       Options.max_log_file_size: 0
2018-10-09 00:33:23.589724 7f91c7ae2e00  4 rocksdb:                 
Options.max_manifest_file_size: 18446744073709551615
2018-10-09 00:33:23.589725 7f91c7ae2e00  4 rocksdb:                  
Options.log_file_time_to_roll: 0
2018-10-09 00:33:23.589726 7f91c7ae2e00  4
rocksdb:                       Options.keep_log_file_num: 1000
2018-10-09 00:33:23.589726 7f91c7ae2e00  4 rocksdb:                   
Options.recycle_log_file_num: 4
2018-10-09 00:33:23.589727 7f91c7ae2e00  4
rocksdb:                         Options.allow_fallocate: 1
2018-10-09 00:33:23.589728 7f91c7ae2e00  4
rocksdb:                        Options.allow_mmap_reads: 0
2018-10-09 00:33:23.589728 7f91c7ae2e00  4
rocksdb:                       Options.allow_mmap_writes: 0
2018-10-09 00:33:23.589729 7f91c7ae2e00  4
rocksdb:                        Options.use_direct_reads: 0
2018-10-09 00:33:23.589730 7f91c7ae2e00  4
rocksdb:                       
Options.use_direct_io_for_flush_and_compaction: 0
2018-10-09 00:33:23.589731 7f91c7ae2e00  4 rocksdb:         
Options.create_missing_column_families: 0
2018-10-09 00:33:23.589731 7f91c7ae2e00  4
rocksdb:                              Options.db_log_dir:
2018-10-09 00:33:23.589732 7f91c7ae2e00  4
rocksdb:                                 Options.wal_dir: db
2018-10-09 00:33:23.589733 7f91c7ae2e00  4 rocksdb:               
Options.table_cache_numshardbits: 6
2018-10-09 00:33:23.589734 7f91c7ae2e00  4 rocksdb:                     
Options.max_subcompactions: 1
2018-10-09 00:33:23.589735 7f91c7ae2e00  4 rocksdb:                 
Options.max_background_flushes: 1
2018-10-09 00:33:23.589736 7f91c7ae2e00  4
rocksdb:                         Options.WAL_ttl_seconds: 0
2018-10-09 00:33:23.589737 7f91c7ae2e00  4
rocksdb:                       Options.WAL_size_limit_MB: 0
2018-10-09 00:33:23.589738 7f91c7ae2e00  4 rocksdb:            
Options.manifest_preallocation_size: 4194304
2018-10-09 00:33:23.589739 7f91c7ae2e00  4 rocksdb:                    
Options.is_fd_close_on_exec: 1
2018-10-09 00:33:23.589740 7f91c7ae2e00  4 rocksdb:                  
Options.advise_random_on_open: 1
2018-10-09 00:33:23.589741 7f91c7ae2e00  4 rocksdb:                   
Options.db_write_buffer_size: 0
2018-10-09 00:33:23.589741 7f91c7ae2e00  4 rocksdb:        
Options.access_hint_on_compaction_start: 1
2018-10-09 00:33:23.589742 7f91c7ae2e00  4 rocksdb: 
Options.new_table_reader_for_compaction_inputs: 1
2018-10-09 00:33:23.589743 7f91c7ae2e00  4 rocksdb:              
Options.compaction_readahead_size: 2097152
2018-10-09 00:33:23.589744 7f91c7ae2e00  4 rocksdb:          
Options.random_access_max_buffer_size: 1048576
2018-10-09 00:33:23.589745 7f91c7ae2e00  4 rocksdb:          
Options.writable_file_max_buffer_size: 0
2018-10-09 00:33:23.589746 7f91c7ae2e00  4 rocksdb:                     
Options.use_adaptive_mutex: 0
2018-10-09 00:33:23.589746 7f91c7ae2e00  4
rocksdb:                            Options.rate_limiter: (nil)
2018-10-09 00:33:23.589747 7f91c7ae2e00  4 rocksdb:    
Options.sst_file_manager.rate_bytes_per_sec: 0
2018-10-09 00:33:23.589748 7f91c7ae2e00  4
rocksdb:                          Options.bytes_per_sync: 0
2018-10-09 00:33:23.589749 7f91c7ae2e00  4 rocksdb:                     
Options.wal_bytes_per_sync: 0
2018-10-09 00:33:23.589749 7f91c7ae2e00  4
rocksdb:                       Options.wal_recovery_mode: 0
2018-10-09 00:33:23.589750 7f91c7ae2e00  4 rocksdb:                 
Options.enable_thread_tracking: 0
2018-10-09 00:33:23.589751 7f91c7ae2e00  4 rocksdb:        
Options.allow_concurrent_memtable_write: 1
2018-10-09 00:33:23.589762 7f91c7ae2e00  4 rocksdb:     
Options.enable_write_thread_adaptive_yield: 1
2018-10-09 00:33:23.589763 7f91c7ae2e00  4 rocksdb:            
Options.write_thread_max_yield_usec: 100
2018-10-09 00:33:23.589764 7f91c7ae2e00  4 rocksdb:           
Options.write_thread_slow_yield_usec: 3
2018-10-09 00:33:23.589765 7f91c7ae2e00  4
rocksdb:                               Options.row_cache: None
2018-10-09 00:33:23.589766 7f91c7ae2e00  4
rocksdb:                              Options.wal_filter: None
2018-10-09 00:33:23.589766 7f91c7ae2e00  4 rocksdb:            
Options.avoid_flush_during_recovery: 0
2018-10-09 00:33:23.589767 7f91c7ae2e00  4 rocksdb:            
Options.base_background_compactions: 1
2018-10-09 00:33:23.589768 7f91c7ae2e00  4 rocksdb:            
Options.max_background_compactions: 1
2018-10-09 00:33:23.589768 7f91c7ae2e00  4 rocksdb:            
Options.avoid_flush_during_shutdown: 0
2018-10-09 00:33:23.589769 7f91c7ae2e00  4 rocksdb:            
Options.delayed_write_rate : 16777216
2018-10-09 00:33:23.589770 7f91c7ae2e00  4 rocksdb:            
Options.max_total_wal_size: 0
2018-10-09 00:33:23.589770 7f91c7ae2e00  4 rocksdb:            
Options.delete_obsolete_files_period_micros: 21600000000
2018-10-09 00:33:23.589771 7f91c7ae2e00  4 rocksdb:                  
Options.stats_dump_period_sec: 600
2018-10-09 00:33:23.589772 7f91c7ae2e00  4 rocksdb: Compression
algorithms supported:
2018-10-09 00:33:23.589773 7f91c7ae2e00  4 rocksdb:     Snappy supported: 1
2018-10-09 00:33:23.589774 7f91c7ae2e00  4 rocksdb:     Zlib supported: 0
2018-10-09 00:33:23.589774 7f91c7ae2e00  4 rocksdb:     Bzip supported: 0
2018-10-09 00:33:23.589775 7f91c7ae2e00  4 rocksdb:     LZ4 supported: 0
2018-10-09 00:33:23.589776 7f91c7ae2e00  4 rocksdb:     ZSTD supported: 0
2018-10-09 00:33:23.589777 7f91c7ae2e00  4 rocksdb: Fast CRC32 supported: 1
2018-10-09 00:33:23.596185 7f91c7ae2e00  4 rocksdb:
[/build/ceph-12.2.8/src/rocksdb/db/version_set.cc:2609] Recovering from
manifest file: MANIFEST-000011

2018-10-09 00:33:23.596256 7f91c7ae2e00  4 rocksdb:
[/build/ceph-12.2.8/src/rocksdb/db/column_family.cc:407] ---------------
Options for column family [default]:

2018-10-09 00:33:23.596261 7f91c7ae2e00  4 rocksdb:              
Options.comparator: leveldb.BytewiseComparator
2018-10-09 00:33:23.596267 7f91c7ae2e00  4 rocksdb:          
Options.merge_operator: .T:int64_array.b:bitwise_xor
2018-10-09 00:33:23.596268 7f91c7ae2e00  4 rocksdb:       
Options.compaction_filter: None
2018-10-09 00:33:23.596272 7f91c7ae2e00  4 rocksdb:       
Options.compaction_filter_factory: None
2018-10-09 00:33:23.596274 7f91c7ae2e00  4 rocksdb:        
Options.memtable_factory: SkipListFactory
2018-10-09 00:33:23.596275 7f91c7ae2e00  4 rocksdb:           
Options.table_factory: BlockBasedTable
2018-10-09 00:33:23.596299 7f91c7ae2e00  4 rocksdb:           
table_factory options:   flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x558d31b641f0)
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 1
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x558d31e6f0b0
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 536870912
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  format_version: 2

2018-10-09 00:33:23.596304 7f91c7ae2e00  4 rocksdb:       
Options.write_buffer_size: 268435456
2018-10-09 00:33:23.596306 7f91c7ae2e00  4 rocksdb: 
Options.max_write_buffer_number: 4
2018-10-09 00:33:23.596307 7f91c7ae2e00  4 rocksdb:         
Options.compression: NoCompression
2018-10-09 00:33:23.596309 7f91c7ae2e00  4 rocksdb:                 
Options.bottommost_compression: Disabled
2018-10-09 00:33:23.596310 7f91c7ae2e00  4 rocksdb:      
Options.prefix_extractor: nullptr
2018-10-09 00:33:23.596312 7f91c7ae2e00  4 rocksdb:  
Options.memtable_insert_with_hint_prefix_extractor: nullptr
2018-10-09 00:33:23.596313 7f91c7ae2e00  4 rocksdb:            
Options.num_levels: 7
2018-10-09 00:33:23.596314 7f91c7ae2e00  4 rocksdb:       
Options.min_write_buffer_number_to_merge: 1
2018-10-09 00:33:23.596315 7f91c7ae2e00  4 rocksdb:    
Options.max_write_buffer_number_to_maintain: 0
2018-10-09 00:33:23.596317 7f91c7ae2e00  4 rocksdb:           
Options.compression_opts.window_bits: -14
2018-10-09 00:33:23.596318 7f91c7ae2e00  4 rocksdb:                 
Options.compression_opts.level: -1
2018-10-09 00:33:23.596319 7f91c7ae2e00  4 rocksdb:              
Options.compression_opts.strategy: 0
2018-10-09 00:33:23.596321 7f91c7ae2e00  4 rocksdb:        
Options.compression_opts.max_dict_bytes: 0
2018-10-09 00:33:23.596324 7f91c7ae2e00  4 rocksdb:     
Options.level0_file_num_compaction_trigger: 4
2018-10-09 00:33:23.596325 7f91c7ae2e00  4 rocksdb:         
Options.level0_slowdown_writes_trigger: 20
2018-10-09 00:33:23.596326 7f91c7ae2e00  4 rocksdb:             
Options.level0_stop_writes_trigger: 36
2018-10-09 00:33:23.596328 7f91c7ae2e00  4 rocksdb:                  
Options.target_file_size_base: 67108864
2018-10-09 00:33:23.596330 7f91c7ae2e00  4 rocksdb:            
Options.target_file_size_multiplier: 1
2018-10-09 00:33:23.596331 7f91c7ae2e00  4 rocksdb:               
Options.max_bytes_for_level_base: 268435456
2018-10-09 00:33:23.596332 7f91c7ae2e00  4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 0
2018-10-09 00:33:23.596334 7f91c7ae2e00  4 rocksdb:         
Options.max_bytes_for_level_multiplier: 10.000000
2018-10-09 00:33:23.596337 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
2018-10-09 00:33:23.596339 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
2018-10-09 00:33:23.596341 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
2018-10-09 00:33:23.596342 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
2018-10-09 00:33:23.596343 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
2018-10-09 00:33:23.596344 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
2018-10-09 00:33:23.596345 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
2018-10-09 00:33:23.596346 7f91c7ae2e00  4 rocksdb:      
Options.max_sequential_skip_in_iterations: 8
2018-10-09 00:33:23.596348 7f91c7ae2e00  4 rocksdb:                   
Options.max_compaction_bytes: 1677721600
2018-10-09 00:33:23.596349 7f91c7ae2e00  4
rocksdb:                        Options.arena_block_size: 33554432
2018-10-09 00:33:23.596351 7f91c7ae2e00  4 rocksdb:  
Options.soft_pending_compaction_bytes_limit: 68719476736
2018-10-09 00:33:23.596352 7f91c7ae2e00  4 rocksdb:  
Options.hard_pending_compaction_bytes_limit: 274877906944
2018-10-09 00:33:23.596353 7f91c7ae2e00  4 rocksdb:      
Options.rate_limit_delay_max_milliseconds: 100
2018-10-09 00:33:23.596354 7f91c7ae2e00  4 rocksdb:               
Options.disable_auto_compactions: 0
2018-10-09 00:33:23.596356 7f91c7ae2e00  4
rocksdb:                         Options.compaction_style:
kCompactionStyleLevel
2018-10-09 00:33:23.596357 7f91c7ae2e00  4
rocksdb:                           Options.compaction_pri:
kByCompensatedSize
2018-10-09 00:33:23.596359 7f91c7ae2e00  4 rocksdb: 
Options.compaction_options_universal.size_ratio: 1
2018-10-09 00:33:23.596360 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
2018-10-09 00:33:23.596361 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
2018-10-09 00:33:23.596362 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
2018-10-09 00:33:23.596363 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
2018-10-09 00:33:23.596364 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
2018-10-09 00:33:23.596366 7f91c7ae2e00  4 rocksdb:                  
Options.table_properties_collectors:
2018-10-09 00:33:23.596367 7f91c7ae2e00  4 rocksdb:                  
Options.inplace_update_support: 0
2018-10-09 00:33:23.596368 7f91c7ae2e00  4 rocksdb:                
Options.inplace_update_num_locks: 10000
2018-10-09 00:33:23.596369 7f91c7ae2e00  4 rocksdb:              
Options.memtable_prefix_bloom_size_ratio: 0.000000
2018-10-09 00:33:23.596371 7f91c7ae2e00  4 rocksdb:  
Options.memtable_huge_page_size: 0
2018-10-09 00:33:23.596372 7f91c7ae2e00  4
rocksdb:                           Options.bloom_locality: 0
2018-10-09 00:33:23.596374 7f91c7ae2e00  4 rocksdb:                   
Options.max_successive_merges: 0
2018-10-09 00:33:23.596374 7f91c7ae2e00  4 rocksdb:               
Options.optimize_filters_for_hits: 0
2018-10-09 00:33:23.596375 7f91c7ae2e00  4 rocksdb:               
Options.paranoid_file_checks: 0
2018-10-09 00:33:23.596380 7f91c7ae2e00  4 rocksdb:               
Options.force_consistency_checks: 0
2018-10-09 00:33:23.596381 7f91c7ae2e00  4 rocksdb:               
Options.report_bg_io_stats: 0
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: In function 'int
BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)'
thread 7f91ba517700 time 2018-10-09 00:33:26.394736
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: 916: FAILED assert(r == 0)
2018-10-09 00:33:26.394691 7f91ba517700 -1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) direct_read_unaligned
0xe0cd46efad~1c053 error: (5) Input/output error
 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x558d27c98a72]
 2: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
 3: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
 4: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
 5: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
 6: (()+0xe22646) [0x558d28037646]
 7:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
 8:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
 9: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
 10: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
 11:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
 12: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
 13: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
 14: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
 15: (()+0xb9e6f) [0x7f91c4bffe6f]
 16: (()+0x7494) [0x7f91c52ed494]
 17: (clone()+0x3f) [0x7f91c4374acf]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
2018-10-09 00:33:26.400883 7f91ba517700 -1
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: In function 'int
BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)'
thread 7f91ba517700 time 2018-10-09 00:33:26.394736
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: 916: FAILED assert(r == 0)

 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x558d27c98a72]
 2: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
 3: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
 4: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
 5: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
 6: (()+0xe22646) [0x558d28037646]
 7:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
 8:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
 9: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
 10: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
 11:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
 12: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
 13: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
 14: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
 15: (()+0xb9e6f) [0x7f91c4bffe6f]
 16: (()+0x7494) [0x7f91c52ed494]
 17: (clone()+0x3f) [0x7f91c4374acf]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

--- begin dump of recent events ---
  -222> 2018-10-09 00:33:23.018426 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command perfcounters_dump hook 0x558d31b661c0
  -221> 2018-10-09 00:33:23.018438 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command 1 hook 0x558d31b661c0
  -220> 2018-10-09 00:33:23.018441 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command perf dump hook 0x558d31b661c0
  -219> 2018-10-09 00:33:23.018443 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command perfcounters_schema hook 0x558d31b661c0
  -218> 2018-10-09 00:33:23.018445 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command perf histogram dump hook 0x558d31b661c0
  -217> 2018-10-09 00:33:23.018447 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command 2 hook 0x558d31b661c0
  -216> 2018-10-09 00:33:23.018449 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command perf schema hook 0x558d31b661c0
  -215> 2018-10-09 00:33:23.018451 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command perf histogram schema hook 0x558d31b661c0
  -214> 2018-10-09 00:33:23.018454 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command perf reset hook 0x558d31b661c0
  -213> 2018-10-09 00:33:23.018456 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command config show hook 0x558d31b661c0
  -212> 2018-10-09 00:33:23.018458 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command config help hook 0x558d31b661c0
  -211> 2018-10-09 00:33:23.018460 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command config set hook 0x558d31b661c0
  -210> 2018-10-09 00:33:23.018462 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command config get hook 0x558d31b661c0
  -209> 2018-10-09 00:33:23.018464 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command config diff hook 0x558d31b661c0
  -208> 2018-10-09 00:33:23.018466 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command config diff get hook 0x558d31b661c0
  -207> 2018-10-09 00:33:23.018469 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command log flush hook 0x558d31b661c0
  -206> 2018-10-09 00:33:23.018471 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command log dump hook 0x558d31b661c0
  -205> 2018-10-09 00:33:23.018473 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command log reopen hook 0x558d31b661c0
  -204> 2018-10-09 00:33:23.018486 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command dump_mempools hook 0x558d31e175c8
  -203> 2018-10-09 00:33:23.025913 7f91c7ae2e00  0 set uid:gid to
64045:64045 (ceph:ceph)
  -202> 2018-10-09 00:33:23.025934 7f91c7ae2e00  0 ceph version 12.2.8
(ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable), process
ceph-osd, pid 322330
  -201> 2018-10-09 00:33:23.025974 7f91c7ae2e00  5 object store type is
bluestore
  -200> 2018-10-09 00:33:23.039921 7f91c1f4d700  2 Event(0x558d31b68500
nevent=5000 time_id=1).set_owner idx=0 owner=140264001033984
  -199> 2018-10-09 00:33:23.039975 7f91c174c700  2 Event(0x558d31b68740
nevent=5000 time_id=1).set_owner idx=1 owner=140263992641280
  -198> 2018-10-09 00:33:23.039980 7f91c0f4b700  2 Event(0x558d31b68980
nevent=5000 time_id=1).set_owner idx=2 owner=140263984248576
  -197> 2018-10-09 00:33:23.040489 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:0/0 learned_addr learned my addr [2a03:4980:0:6::2e]:0/0
  -196> 2018-10-09 00:33:23.040510 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:6800/322330 _finish_bind bind my_inst.addr is
[2a03:4980:0:6::2e]:6800/322330
  -195> 2018-10-09 00:33:23.040805 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:0/0 learned_addr learned my addr [2a03:4980:0:6::2e]:0/0
  -194> 2018-10-09 00:33:23.040820 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:6801/322330 _finish_bind bind my_inst.addr is
[2a03:4980:0:6::2e]:6801/322330
  -193> 2018-10-09 00:33:23.041264 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:0/0 learned_addr learned my addr [2a03:4980:0:6::2e]:0/0
  -192> 2018-10-09 00:33:23.041279 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:6802/322330 _finish_bind bind my_inst.addr is
[2a03:4980:0:6::2e]:6802/322330
  -191> 2018-10-09 00:33:23.041565 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:0/0 learned_addr learned my addr [2a03:4980:0:6::2e]:0/0
  -190> 2018-10-09 00:33:23.041580 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:6803/322330 _finish_bind bind my_inst.addr is
[2a03:4980:0:6::2e]:6803/322330
  -189> 2018-10-09 00:33:23.044207 7f91c7ae2e00  5 asok(0x558d31bc74a0)
init /var/run/ceph/ceph-osd.40.asok
  -188> 2018-10-09 00:33:23.044224 7f91c7ae2e00  5 asok(0x558d31bc74a0)
bind_and_listen /var/run/ceph/ceph-osd.40.asok
  -187> 2018-10-09 00:33:23.044292 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command 0 hook 0x558d31b641a8
  -186> 2018-10-09 00:33:23.044299 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command version hook 0x558d31b641a8
  -185> 2018-10-09 00:33:23.044302 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command git_version hook 0x558d31b641a8
  -184> 2018-10-09 00:33:23.044306 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command help hook 0x558d31b66620
  -183> 2018-10-09 00:33:23.044310 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command get_command_descriptions hook 0x558d31b66630
  -182> 2018-10-09 00:33:23.044391 7f91c7ae2e00 10 monclient:
build_initial_monmap
  -181> 2018-10-09 00:33:23.044443 7f91bf77a700  5 asok(0x558d31bc74a0)
entry start
  -180> 2018-10-09 00:33:23.203930 7f91c7ae2e00  0 load: jerasure load:
lrc load: isa
  -179> 2018-10-09 00:33:23.203960 7f91c7ae2e00  5 adding auth protocol:
cephx
  -178> 2018-10-09 00:33:23.203962 7f91c7ae2e00  5 adding auth protocol:
cephx
  -177> 2018-10-09 00:33:23.204022 7f91c7ae2e00  1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
  -176> 2018-10-09 00:33:23.204032 7f91c7ae2e00  1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
  -175> 2018-10-09 00:33:23.204336 7f91c7ae2e00  1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
  -174> 2018-10-09 00:33:23.204766 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes max 0.5 < ratio 0.99
  -173> 2018-10-09 00:33:23.204790 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes cache_size
1073741824 meta 0.5 kv 0.5 data 0
  -172> 2018-10-09 00:33:23.204796 7f91c7ae2e00  1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) close
  -171> 2018-10-09 00:33:23.499587 7f91c7ae2e00  5 asok(0x558d31bc74a0)
register_command objecter_requests hook 0x558d31b666d0
  -170> 2018-10-09 00:33:23.499679 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:6800/322330 start start
  -169> 2018-10-09 00:33:23.499693 7f91c7ae2e00  1 -- - start start
  -168> 2018-10-09 00:33:23.499696 7f91c7ae2e00  1 -- - start start
  -167> 2018-10-09 00:33:23.499697 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:6803/322330 start start
  -166> 2018-10-09 00:33:23.499701 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:6802/322330 start start
  -165> 2018-10-09 00:33:23.499705 7f91c7ae2e00  1 --
[2a03:4980:0:6::2e]:6801/322330 start start
  -164> 2018-10-09 00:33:23.499708 7f91c7ae2e00  1 -- - start start
  -163> 2018-10-09 00:33:23.499855 7f91c7ae2e00  2 osd.40 0 init
/var/lib/ceph/osd/ceph-40 (looks like hdd)
  -162> 2018-10-09 00:33:23.499863 7f91c7ae2e00  2 osd.40 0 journal
/var/lib/ceph/osd/ceph-40/journal
  -161> 2018-10-09 00:33:23.499876 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _mount path /var/lib/ceph/osd/ceph-40
  -160> 2018-10-09 00:33:23.500355 7f91c7ae2e00  1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
  -159> 2018-10-09 00:33:23.500368 7f91c7ae2e00  1 bdev(0x558d31b68d80
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
  -158> 2018-10-09 00:33:23.500583 7f91c7ae2e00  1 bdev(0x558d31b68d80
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
  -157> 2018-10-09 00:33:23.500884 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes max 0.5 < ratio 0.99
  -156> 2018-10-09 00:33:23.500900 7f91c7ae2e00  1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes cache_size
1073741824 meta 0.5 kv 0.5 data 0
  -155> 2018-10-09 00:33:23.501051 7f91c7ae2e00  1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
  -154> 2018-10-09 00:33:23.501061 7f91c7ae2e00  1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
  -153> 2018-10-09 00:33:23.501263 7f91c7ae2e00  1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
  -152> 2018-10-09 00:33:23.501277 7f91c7ae2e00  1 bluefs
add_block_device bdev 1 path /var/lib/ceph/osd/ceph-40/block size 1.81TiB
  -151> 2018-10-09 00:33:23.501310 7f91c7ae2e00  1 bluefs mount
  -150> 2018-10-09 00:33:23.589465 7f91c7ae2e00  0  set rocksdb option
compaction_readahead_size = 2097152
  -149> 2018-10-09 00:33:23.589482 7f91c7ae2e00  0  set rocksdb option
compression = kNoCompression
  -148> 2018-10-09 00:33:23.589487 7f91c7ae2e00  0  set rocksdb option
max_write_buffer_number = 4
  -147> 2018-10-09 00:33:23.589490 7f91c7ae2e00  0  set rocksdb option
min_write_buffer_number_to_merge = 1
  -146> 2018-10-09 00:33:23.589494 7f91c7ae2e00  0  set rocksdb option
recycle_log_file_num = 4
  -145> 2018-10-09 00:33:23.589496 7f91c7ae2e00  0  set rocksdb option
writable_file_max_buffer_size = 0
  -144> 2018-10-09 00:33:23.589500 7f91c7ae2e00  0  set rocksdb option
write_buffer_size = 268435456
  -143> 2018-10-09 00:33:23.589517 7f91c7ae2e00  0  set rocksdb option
compaction_readahead_size = 2097152
  -142> 2018-10-09 00:33:23.589520 7f91c7ae2e00  0  set rocksdb option
compression = kNoCompression
  -141> 2018-10-09 00:33:23.589523 7f91c7ae2e00  0  set rocksdb option
max_write_buffer_number = 4
  -140> 2018-10-09 00:33:23.589526 7f91c7ae2e00  0  set rocksdb option
min_write_buffer_number_to_merge = 1
  -139> 2018-10-09 00:33:23.589528 7f91c7ae2e00  0  set rocksdb option
recycle_log_file_num = 4
  -138> 2018-10-09 00:33:23.589531 7f91c7ae2e00  0  set rocksdb option
writable_file_max_buffer_size = 0
  -137> 2018-10-09 00:33:23.589533 7f91c7ae2e00  0  set rocksdb option
write_buffer_size = 268435456
  -136> 2018-10-09 00:33:23.589656 7f91c7ae2e00  4 rocksdb: RocksDB
version: 5.4.0

  -135> 2018-10-09 00:33:23.589667 7f91c7ae2e00  4 rocksdb: Git sha
rocksdb_build_git_sha:@0@
  -134> 2018-10-09 00:33:23.589670 7f91c7ae2e00  4 rocksdb: Compile date
Aug 30 2018
  -133> 2018-10-09 00:33:23.589671 7f91c7ae2e00  4 rocksdb: DB SUMMARY

  -132> 2018-10-09 00:33:23.589706 7f91c7ae2e00  4 rocksdb: CURRENT
file:  CURRENT

  -131> 2018-10-09 00:33:23.589707 7f91c7ae2e00  4 rocksdb: IDENTITY
file:  IDENTITY

  -130> 2018-10-09 00:33:23.589709 7f91c7ae2e00  4 rocksdb: MANIFEST
file:  MANIFEST-000011 size: 51827 Bytes

  -129> 2018-10-09 00:33:23.589712 7f91c7ae2e00  4 rocksdb: SST files in
db dir, Total Num: 28, files: 000260.sst 000261.sst 000262.sst
000274.sst 000275.sst 000276.sst 000277.sst 000286.sst 000287.sst

  -128> 2018-10-09 00:33:23.589714 7f91c7ae2e00  4 rocksdb: Write Ahead
Log file in db: 000310.log size: 258554009 ; 000312.log size: 256884429 ;

  -127> 2018-10-09 00:33:23.589715 7f91c7ae2e00  4
rocksdb:                         Options.error_if_exists: 0
  -126> 2018-10-09 00:33:23.589716 7f91c7ae2e00  4
rocksdb:                       Options.create_if_missing: 0
  -125> 2018-10-09 00:33:23.589717 7f91c7ae2e00  4
rocksdb:                         Options.paranoid_checks: 1
  -124> 2018-10-09 00:33:23.589718 7f91c7ae2e00  4
rocksdb:                                     Options.env: 0x558d31d42280
  -123> 2018-10-09 00:33:23.589719 7f91c7ae2e00  4
rocksdb:                                Options.info_log: 0x558d31d42a00
  -122> 2018-10-09 00:33:23.589720 7f91c7ae2e00  4
rocksdb:                          Options.max_open_files: -1
  -121> 2018-10-09 00:33:23.589721 7f91c7ae2e00  4
rocksdb:                Options.max_file_opening_threads: 16
  -120> 2018-10-09 00:33:23.589722 7f91c7ae2e00  4
rocksdb:                               Options.use_fsync: 0
  -119> 2018-10-09 00:33:23.589723 7f91c7ae2e00  4
rocksdb:                       Options.max_log_file_size: 0
  -118> 2018-10-09 00:33:23.589724 7f91c7ae2e00  4
rocksdb:                  Options.max_manifest_file_size:
18446744073709551615
  -117> 2018-10-09 00:33:23.589725 7f91c7ae2e00  4
rocksdb:                   Options.log_file_time_to_roll: 0
  -116> 2018-10-09 00:33:23.589726 7f91c7ae2e00  4
rocksdb:                       Options.keep_log_file_num: 1000
  -115> 2018-10-09 00:33:23.589726 7f91c7ae2e00  4
rocksdb:                    Options.recycle_log_file_num: 4
  -114> 2018-10-09 00:33:23.589727 7f91c7ae2e00  4
rocksdb:                         Options.allow_fallocate: 1
  -113> 2018-10-09 00:33:23.589728 7f91c7ae2e00  4
rocksdb:                        Options.allow_mmap_reads: 0
  -112> 2018-10-09 00:33:23.589728 7f91c7ae2e00  4
rocksdb:                       Options.allow_mmap_writes: 0
  -111> 2018-10-09 00:33:23.589729 7f91c7ae2e00  4
rocksdb:                        Options.use_direct_reads: 0
  -110> 2018-10-09 00:33:23.589730 7f91c7ae2e00  4
rocksdb:                       
Options.use_direct_io_for_flush_and_compaction: 0
  -109> 2018-10-09 00:33:23.589731 7f91c7ae2e00  4 rocksdb:         
Options.create_missing_column_families: 0
  -108> 2018-10-09 00:33:23.589731 7f91c7ae2e00  4
rocksdb:                              Options.db_log_dir:
  -107> 2018-10-09 00:33:23.589732 7f91c7ae2e00  4
rocksdb:                                 Options.wal_dir: db
  -106> 2018-10-09 00:33:23.589733 7f91c7ae2e00  4
rocksdb:                Options.table_cache_numshardbits: 6
  -105> 2018-10-09 00:33:23.589734 7f91c7ae2e00  4
rocksdb:                      Options.max_subcompactions: 1
  -104> 2018-10-09 00:33:23.589735 7f91c7ae2e00  4
rocksdb:                  Options.max_background_flushes: 1
  -103> 2018-10-09 00:33:23.589736 7f91c7ae2e00  4
rocksdb:                         Options.WAL_ttl_seconds: 0
  -102> 2018-10-09 00:33:23.589737 7f91c7ae2e00  4
rocksdb:                       Options.WAL_size_limit_MB: 0
  -101> 2018-10-09 00:33:23.589738 7f91c7ae2e00  4 rocksdb:            
Options.manifest_preallocation_size: 4194304
  -100> 2018-10-09 00:33:23.589739 7f91c7ae2e00  4
rocksdb:                     Options.is_fd_close_on_exec: 1
   -99> 2018-10-09 00:33:23.589740 7f91c7ae2e00  4
rocksdb:                   Options.advise_random_on_open: 1
   -98> 2018-10-09 00:33:23.589741 7f91c7ae2e00  4
rocksdb:                    Options.db_write_buffer_size: 0
   -97> 2018-10-09 00:33:23.589741 7f91c7ae2e00  4 rocksdb:        
Options.access_hint_on_compaction_start: 1
   -96> 2018-10-09 00:33:23.589742 7f91c7ae2e00  4 rocksdb: 
Options.new_table_reader_for_compaction_inputs: 1
   -95> 2018-10-09 00:33:23.589743 7f91c7ae2e00  4
rocksdb:               Options.compaction_readahead_size: 2097152
   -94> 2018-10-09 00:33:23.589744 7f91c7ae2e00  4 rocksdb:          
Options.random_access_max_buffer_size: 1048576
   -93> 2018-10-09 00:33:23.589745 7f91c7ae2e00  4 rocksdb:          
Options.writable_file_max_buffer_size: 0
   -92> 2018-10-09 00:33:23.589746 7f91c7ae2e00  4
rocksdb:                      Options.use_adaptive_mutex: 0
   -91> 2018-10-09 00:33:23.589746 7f91c7ae2e00  4
rocksdb:                            Options.rate_limiter: (nil)
   -90> 2018-10-09 00:33:23.589747 7f91c7ae2e00  4 rocksdb:    
Options.sst_file_manager.rate_bytes_per_sec: 0
   -89> 2018-10-09 00:33:23.589748 7f91c7ae2e00  4
rocksdb:                          Options.bytes_per_sync: 0
   -88> 2018-10-09 00:33:23.589749 7f91c7ae2e00  4
rocksdb:                      Options.wal_bytes_per_sync: 0
   -87> 2018-10-09 00:33:23.589749 7f91c7ae2e00  4
rocksdb:                       Options.wal_recovery_mode: 0
   -86> 2018-10-09 00:33:23.589750 7f91c7ae2e00  4
rocksdb:                  Options.enable_thread_tracking: 0
   -85> 2018-10-09 00:33:23.589751 7f91c7ae2e00  4 rocksdb:        
Options.allow_concurrent_memtable_write: 1
   -84> 2018-10-09 00:33:23.589762 7f91c7ae2e00  4 rocksdb:     
Options.enable_write_thread_adaptive_yield: 1
   -83> 2018-10-09 00:33:23.589763 7f91c7ae2e00  4 rocksdb:            
Options.write_thread_max_yield_usec: 100
   -82> 2018-10-09 00:33:23.589764 7f91c7ae2e00  4 rocksdb:           
Options.write_thread_slow_yield_usec: 3
   -81> 2018-10-09 00:33:23.589765 7f91c7ae2e00  4
rocksdb:                               Options.row_cache: None
   -80> 2018-10-09 00:33:23.589766 7f91c7ae2e00  4
rocksdb:                              Options.wal_filter: None
   -79> 2018-10-09 00:33:23.589766 7f91c7ae2e00  4 rocksdb:            
Options.avoid_flush_during_recovery: 0
   -78> 2018-10-09 00:33:23.589767 7f91c7ae2e00  4 rocksdb:            
Options.base_background_compactions: 1
   -77> 2018-10-09 00:33:23.589768 7f91c7ae2e00  4 rocksdb:            
Options.max_background_compactions: 1
   -76> 2018-10-09 00:33:23.589768 7f91c7ae2e00  4 rocksdb:            
Options.avoid_flush_during_shutdown: 0
   -75> 2018-10-09 00:33:23.589769 7f91c7ae2e00  4 rocksdb:            
Options.delayed_write_rate : 16777216
   -74> 2018-10-09 00:33:23.589770 7f91c7ae2e00  4 rocksdb:            
Options.max_total_wal_size: 0
   -73> 2018-10-09 00:33:23.589770 7f91c7ae2e00  4 rocksdb:            
Options.delete_obsolete_files_period_micros: 21600000000
   -72> 2018-10-09 00:33:23.589771 7f91c7ae2e00  4
rocksdb:                   Options.stats_dump_period_sec: 600
   -71> 2018-10-09 00:33:23.589772 7f91c7ae2e00  4 rocksdb: Compression
algorithms supported:
   -70> 2018-10-09 00:33:23.589773 7f91c7ae2e00  4 rocksdb:     Snappy
supported: 1
   -69> 2018-10-09 00:33:23.589774 7f91c7ae2e00  4 rocksdb:     Zlib
supported: 0
   -68> 2018-10-09 00:33:23.589774 7f91c7ae2e00  4 rocksdb:     Bzip
supported: 0
   -67> 2018-10-09 00:33:23.589775 7f91c7ae2e00  4 rocksdb:     LZ4
supported: 0
   -66> 2018-10-09 00:33:23.589776 7f91c7ae2e00  4 rocksdb:     ZSTD
supported: 0
   -65> 2018-10-09 00:33:23.589777 7f91c7ae2e00  4 rocksdb: Fast CRC32
supported: 1
   -64> 2018-10-09 00:33:23.596185 7f91c7ae2e00  4 rocksdb:
[/build/ceph-12.2.8/src/rocksdb/db/version_set.cc:2609] Recovering from
manifest file: MANIFEST-000011

   -63> 2018-10-09 00:33:23.596256 7f91c7ae2e00  4 rocksdb:
[/build/ceph-12.2.8/src/rocksdb/db/column_family.cc:407] ---------------
Options for column family [default]:

   -62> 2018-10-09 00:33:23.596261 7f91c7ae2e00  4
rocksdb:               Options.comparator: leveldb.BytewiseComparator
   -61> 2018-10-09 00:33:23.596267 7f91c7ae2e00  4 rocksdb:          
Options.merge_operator: .T:int64_array.b:bitwise_xor
   -60> 2018-10-09 00:33:23.596268 7f91c7ae2e00  4 rocksdb:       
Options.compaction_filter: None
   -59> 2018-10-09 00:33:23.596272 7f91c7ae2e00  4 rocksdb:       
Options.compaction_filter_factory: None
   -58> 2018-10-09 00:33:23.596274 7f91c7ae2e00  4 rocksdb:        
Options.memtable_factory: SkipListFactory
   -57> 2018-10-09 00:33:23.596275 7f91c7ae2e00  4 rocksdb:           
Options.table_factory: BlockBasedTable
   -56> 2018-10-09 00:33:23.596299 7f91c7ae2e00  4 rocksdb:           
table_factory options:   flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x558d31b641f0)
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 1
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x558d31e6f0b0
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 536870912
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  format_version: 2

   -55> 2018-10-09 00:33:23.596304 7f91c7ae2e00  4 rocksdb:       
Options.write_buffer_size: 268435456
   -54> 2018-10-09 00:33:23.596306 7f91c7ae2e00  4 rocksdb: 
Options.max_write_buffer_number: 4
   -53> 2018-10-09 00:33:23.596307 7f91c7ae2e00  4 rocksdb:         
Options.compression: NoCompression
   -52> 2018-10-09 00:33:23.596309 7f91c7ae2e00  4
rocksdb:                  Options.bottommost_compression: Disabled
   -51> 2018-10-09 00:33:23.596310 7f91c7ae2e00  4 rocksdb:      
Options.prefix_extractor: nullptr
   -50> 2018-10-09 00:33:23.596312 7f91c7ae2e00  4 rocksdb:  
Options.memtable_insert_with_hint_prefix_extractor: nullptr
   -49> 2018-10-09 00:33:23.596313 7f91c7ae2e00  4 rocksdb:            
Options.num_levels: 7
   -48> 2018-10-09 00:33:23.596314 7f91c7ae2e00  4 rocksdb:       
Options.min_write_buffer_number_to_merge: 1
   -47> 2018-10-09 00:33:23.596315 7f91c7ae2e00  4 rocksdb:    
Options.max_write_buffer_number_to_maintain: 0
   -46> 2018-10-09 00:33:23.596317 7f91c7ae2e00  4 rocksdb:           
Options.compression_opts.window_bits: -14
   -45> 2018-10-09 00:33:23.596318 7f91c7ae2e00  4
rocksdb:                  Options.compression_opts.level: -1
   -44> 2018-10-09 00:33:23.596319 7f91c7ae2e00  4
rocksdb:               Options.compression_opts.strategy: 0
   -43> 2018-10-09 00:33:23.596321 7f91c7ae2e00  4 rocksdb:        
Options.compression_opts.max_dict_bytes: 0
   -42> 2018-10-09 00:33:23.596324 7f91c7ae2e00  4 rocksdb:     
Options.level0_file_num_compaction_trigger: 4
   -41> 2018-10-09 00:33:23.596325 7f91c7ae2e00  4 rocksdb:         
Options.level0_slowdown_writes_trigger: 20
   -40> 2018-10-09 00:33:23.596326 7f91c7ae2e00  4 rocksdb:             
Options.level0_stop_writes_trigger: 36
   -39> 2018-10-09 00:33:23.596328 7f91c7ae2e00  4
rocksdb:                   Options.target_file_size_base: 67108864
   -38> 2018-10-09 00:33:23.596330 7f91c7ae2e00  4 rocksdb:            
Options.target_file_size_multiplier: 1
   -37> 2018-10-09 00:33:23.596331 7f91c7ae2e00  4
rocksdb:                Options.max_bytes_for_level_base: 268435456
   -36> 2018-10-09 00:33:23.596332 7f91c7ae2e00  4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 0
   -35> 2018-10-09 00:33:23.596334 7f91c7ae2e00  4 rocksdb:         
Options.max_bytes_for_level_multiplier: 10.000000
   -34> 2018-10-09 00:33:23.596337 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
   -33> 2018-10-09 00:33:23.596339 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
   -32> 2018-10-09 00:33:23.596341 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
   -31> 2018-10-09 00:33:23.596342 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
   -30> 2018-10-09 00:33:23.596343 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
   -29> 2018-10-09 00:33:23.596344 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
   -28> 2018-10-09 00:33:23.596345 7f91c7ae2e00  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
   -27> 2018-10-09 00:33:23.596346 7f91c7ae2e00  4 rocksdb:      
Options.max_sequential_skip_in_iterations: 8
   -26> 2018-10-09 00:33:23.596348 7f91c7ae2e00  4
rocksdb:                    Options.max_compaction_bytes: 1677721600
   -25> 2018-10-09 00:33:23.596349 7f91c7ae2e00  4
rocksdb:                        Options.arena_block_size: 33554432
   -24> 2018-10-09 00:33:23.596351 7f91c7ae2e00  4 rocksdb:  
Options.soft_pending_compaction_bytes_limit: 68719476736
   -23> 2018-10-09 00:33:23.596352 7f91c7ae2e00  4 rocksdb:  
Options.hard_pending_compaction_bytes_limit: 274877906944
   -22> 2018-10-09 00:33:23.596353 7f91c7ae2e00  4 rocksdb:      
Options.rate_limit_delay_max_milliseconds: 100
   -21> 2018-10-09 00:33:23.596354 7f91c7ae2e00  4
rocksdb:                Options.disable_auto_compactions: 0
   -20> 2018-10-09 00:33:23.596356 7f91c7ae2e00  4
rocksdb:                         Options.compaction_style:
kCompactionStyleLevel
   -19> 2018-10-09 00:33:23.596357 7f91c7ae2e00  4
rocksdb:                           Options.compaction_pri:
kByCompensatedSize
   -18> 2018-10-09 00:33:23.596359 7f91c7ae2e00  4 rocksdb: 
Options.compaction_options_universal.size_ratio: 1
   -17> 2018-10-09 00:33:23.596360 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
   -16> 2018-10-09 00:33:23.596361 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
   -15> 2018-10-09 00:33:23.596362 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
   -14> 2018-10-09 00:33:23.596363 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
   -13> 2018-10-09 00:33:23.596364 7f91c7ae2e00  4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
   -12> 2018-10-09 00:33:23.596366 7f91c7ae2e00  4
rocksdb:                   Options.table_properties_collectors:
   -11> 2018-10-09 00:33:23.596367 7f91c7ae2e00  4
rocksdb:                   Options.inplace_update_support: 0
   -10> 2018-10-09 00:33:23.596368 7f91c7ae2e00  4
rocksdb:                 Options.inplace_update_num_locks: 10000
    -9> 2018-10-09 00:33:23.596369 7f91c7ae2e00  4
rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
    -8> 2018-10-09 00:33:23.596371 7f91c7ae2e00  4 rocksdb:  
Options.memtable_huge_page_size: 0
    -7> 2018-10-09 00:33:23.596372 7f91c7ae2e00  4
rocksdb:                           Options.bloom_locality: 0
    -6> 2018-10-09 00:33:23.596374 7f91c7ae2e00  4
rocksdb:                    Options.max_successive_merges: 0
    -5> 2018-10-09 00:33:23.596374 7f91c7ae2e00  4
rocksdb:                Options.optimize_filters_for_hits: 0
    -4> 2018-10-09 00:33:23.596375 7f91c7ae2e00  4
rocksdb:                Options.paranoid_file_checks: 0
    -3> 2018-10-09 00:33:23.596380 7f91c7ae2e00  4
rocksdb:                Options.force_consistency_checks: 0
    -2> 2018-10-09 00:33:23.596381 7f91c7ae2e00  4
rocksdb:                Options.report_bg_io_stats: 0
    -1> 2018-10-09 00:33:26.394691 7f91ba517700 -1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) direct_read_unaligned
0xe0cd46efad~1c053 error: (5) Input/output error
     0> 2018-10-09 00:33:26.400883 7f91ba517700 -1
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: In function 'int
BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)'
thread 7f91ba517700 time 2018-10-09 00:33:26.394736
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: 916: FAILED assert(r == 0)

 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x558d27c98a72]
 2: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
 3: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
 4: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
 5: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
 6: (()+0xe22646) [0x558d28037646]
 7:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
 8:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
 9: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
 10: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
 11:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
 12: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
 13: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
 14: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
 15: (()+0xb9e6f) [0x7f91c4bffe6f]
 16: (()+0x7494) [0x7f91c52ed494]
 17: (clone()+0x3f) [0x7f91c4374acf]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -1/-1 (syslog threshold)
  99/99 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file
--- end dump of recent events ---
*** Caught signal (Aborted) **
 in thread 7f91ba517700 thread_name:ceph-osd
 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
 1: (()+0xa3bba4) [0x558d27c50ba4]
 2: (()+0x110c0) [0x7f91c52f70c0]
 3: (gsignal()+0xcf) [0x7f91c42befff]
 4: (abort()+0x16a) [0x7f91c42c042a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x558d27c98bfe]
 6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
 7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
 8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
 9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
 10: (()+0xe22646) [0x558d28037646]
 11:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
 12:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
 13: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
 14: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
 15:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
 16: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
 17: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
 18: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
 19: (()+0xb9e6f) [0x7f91c4bffe6f]
 20: (()+0x7494) [0x7f91c52ed494]
 21: (clone()+0x3f) [0x7f91c4374acf]
2018-10-09 00:33:26.408732 7f91ba517700 -1 *** Caught signal (Aborted) **
 in thread 7f91ba517700 thread_name:ceph-osd

 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
 1: (()+0xa3bba4) [0x558d27c50ba4]
 2: (()+0x110c0) [0x7f91c52f70c0]
 3: (gsignal()+0xcf) [0x7f91c42befff]
 4: (abort()+0x16a) [0x7f91c42c042a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x558d27c98bfe]
 6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
 7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
 8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
 9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
 10: (()+0xe22646) [0x558d28037646]
 11:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
 12:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
 13: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
 14: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
 15:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
 16: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
 17: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
 18: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
 19: (()+0xb9e6f) [0x7f91c4bffe6f]
 20: (()+0x7494) [0x7f91c52ed494]
 21: (clone()+0x3f) [0x7f91c4374acf]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

--- begin dump of recent events ---
     0> 2018-10-09 00:33:26.408732 7f91ba517700 -1 *** Caught signal
(Aborted) **
 in thread 7f91ba517700 thread_name:ceph-osd

 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
 1: (()+0xa3bba4) [0x558d27c50ba4]
 2: (()+0x110c0) [0x7f91c52f70c0]
 3: (gsignal()+0xcf) [0x7f91c42befff]
 4: (abort()+0x16a) [0x7f91c42c042a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x558d27c98bfe]
 6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
 7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
 8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
 9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
 10: (()+0xe22646) [0x558d28037646]
 11:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
 12:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
 13: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
 14: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
 15:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
 16: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
 17: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
 18: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
 19: (()+0xb9e6f) [0x7f91c4bffe6f]
 20: (()+0x7494) [0x7f91c52ed494]
 21: (clone()+0x3f) [0x7f91c4374acf]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -1/-1 (syslog threshold)
  99/99 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file
--- end dump of recent events ---

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pEpkey.asc
Type: application/pgp-keys
Size: 2216 bytes
Desc: not available
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181009/ecc56b11/attachment.key>


More information about the ceph-users mailing list