Ceph镜像无法删除rbd: error: image still has watchers解决方案

前言

需要删除一些Ceph镜像，遇到标题的报错，这里写一下解决方案供后续查阅。其中pool是 Ceph-SSD-Rep3，磁盘镜像是vm-29101-disk-0
使用rbd rm Ceph-SSD-Rep3/vm-29101-disk-0 -f删除时报错：

2021-12-06T10:20:50.938+0100 7f20abfff700 -1 librbd::image::PreRemoveRequest: 0x55e44013b060 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

解决方案

字面意思就是有服务在占用这个镜像，所以只要找到后关闭就行，首先看磁盘信息：

root@CPU-FSN1-C115 / # rbd --pool Ceph-SSD-Rep3 info vm-29101-disk-0
rbd image 'vm-29101-disk-0':
        size 8 GiB in 2048 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 3abd77df337601
        block_name_prefix: rbd_data.3abd77df337601
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        op_features:
        flags:
        create_timestamp: Sat Nov 20 20:15:39 2021
        access_timestamp: Sat Nov 20 20:15:39 2021
        modify_timestamp: Sat Nov 20 20:15:39 2021

然后确定是哪个服务在占用这个磁盘：

root@CPU-FSN1-C115 / # rbd status Ceph-SSD-Rep3/vm-29101-disk-0
Watchers:
        watcher=10.2.0.112:0/1706250298 client.672648578 cookie=18446462598732841239

可以看到，这里10.2.0.112的服务器正在使用这个磁盘。

于是登录这个服务器，查看情况：

root@CPU-FSN1-C112 ~ # lsblk
NAME                                                                                                    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
rbd0                                                                                                    252:0    0     8G  0 disk 
rbd1                                                                                                    252:16   0    32G  0 disk 
└─rbd1p1                                                                                                252:17   0    32G  0 part /root/temp

卸载并取消映射：

root@CPU-FSN1-C112 ~ # umount /root/temp
root@CPU-FSN1-C112 ~ # rbd unmap /dev/rbd1
root@CPU-FSN1-C112 ~ # rbd unmap /dev/rbd0

重新删除镜像：

root@CPU-FSN1-C115 / # rbd --pool Ceph-SSD-Rep3 status vm-29101-disk-0
Watchers: none
root@CPU-FSN1-C115 / # rbd rm Ceph-SSD-Rep3/vm-29101-disk-0 -f
Removing image: 100% complete...done.
root@CPU-FSN1-C115 / # rbd rm Ceph-SSD-Rep3/vm-29101-disk-1 -f
Removing image: 100% complete...done.