Ceph luminous rbd map hangs forever

Question

Running a 1 node ceph cluster, and using the ceph-client from another node. Qemu is working fine with the RBD mounting. When I try to mount a RBD block device on the ceph-client I get an indefinite hang with no output. How to diagnose whats wrong?

System is ubuntu 16.04 server, and Ceph Luminous.

sudo ceph tell osd.* version
{
    "version": "ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)"
}


ceph -s
  cluster:
    id:     4bfcc109-e432-4ac0-ba9d-bf81243aea
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum gcmaster
    mgr: gcmaster(active)
    osd: 1 osds: 1 up, 1 in

  data:
    pools:   1 pools, 128 pgs
    objects: 1512 objects, 5879 MB
    usage:   7356 MB used, 216 GB / 223 GB avail
    pgs:     128 active+clean

rbd info gcbase
rbd image 'gcbase':
    size 512 MB in 128 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.376974b0dc51
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    flags: 
    create_timestamp: Fri Dec 29 17:58:02 2017

This hangs forever

rbd map gcbase --pool rbd

As does this

rbd map typo_gcbase --pool rbd

dmesg shows

Dec 29 13:27:32 cephclient1 kernel: [85798.195468] libceph: mon0 192.168.1.55:6789 feature set mismatch, my 106b84a842a42 < server's 40106b84a842a42, missing 400000000000000
Dec 29 13:27:32 cephclient1 kernel: [85798.222070] libceph: mon0 192.168.1.55:6789 missing required protocol features

mmlr mmlr · Accepted Answer · 2017-12-30T21:24:07

The dmesg output tells what's going on: The cluster requires a feature bit that is not supported by the libceph kernel module.

The feature bit in question is either CEPH_FEATURE_CRUSH_TUNABLES5, CEPH_FEATURE_NEW_OSDOPREPLY_ENCODING or CEPH_FEATURE_FS_FILE_LAYOUT_V2 (they are overlapping because they were introduced at the same time) which only became available on kernel 4.5, whereas Ubuntu 16.04 uses a 4.4 kernel.

A similar question (although related to CephFS) came up on the mailing list with a possible solution:

Yes, you should be able to set your CRUSH tunables profile to hammer with "ceph osd crush tunables hammer".

This will disable some features, but should make the older kernel compatible with the cluster.

Alternatively you could upgrade to a mainline kernel or to a newer OS release.

Ceph luminous rbd map hangs forever

1 Answers