Configure LVM resource on Redhat 7.4 cluster using pacemaker

Question

I am configuring the Red Hat cluster with pacemaker and I wanted to add a LVM resource. I have installed following packages,

OS: Red Hat 7.4

Packages Installed: lvm2-cluster, pacemaker, corosync, pcs, fence-agents-all

but my LVM resource have a failed state with following error:

[root@node1 ~]# pcs status
Cluster name: jcluster
WARNING: no stonith devices and stonith-enabled is not false
Stack: corosync
Current DC: node2 (version 1.1.16-12.el7_4.8-94ff4df) - partition with quorum
Last updated: Sat Mar 10 11:54:41 2018
Last change: Sat Mar 10 11:17:13 2018 by hacluster via cibadmin on node1

2 nodes configured
3 resources configured (2 DISABLED)

Online: [ node1 node2 ]

Full list of resources:

 Clone Set: juris-clvmd-clone [juris-clvmd]
     Stopped (disabled): [ node1 node2 ]
 juris-lvm      (ocf::heartbeat:LVM):   FAILED node1

Failed Actions:
* juris-lvm_monitor_0 on node1 'unknown error' (1): call=15, status=complete, exitreason='WARNING: jurisvg is active without the cluster tag, "pacemaker"',
last-rc-change='Fri Mar  9 20:38:50 2018', queued=0ms, exec=255ms
* juris-lvm_monitor_10000 on node1 'unknown error' (1): call=16, status=complete, exitreason='WARNING: jurisvg is active without the cluster tag, "pacemaker"',
last-rc-change='Sat Mar 10 10:24:55 2018', queued=0ms, exec=0ms


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

I'm using the iscsi to shared the disk for my both node. After i present the shared disk to the nodes, I have then create a pvcreate, vgcreate, lvcreate for the new presented disk.

After that I change the new vg and I created to the clustered attribute using following command.

[root@node1 ~]# vgchange -cy jurisvg
  /dev/jurisvg/ha_lv: read failed after 0 of 4096 at 0: Input/output error
  /dev/jurisvg/ha_lv: read failed after 0 of 4096 at 53687025664: Input/output error
  /dev/jurisvg/ha_lv: read failed after 0 of 4096 at 53687083008: Input/output error
  /dev/jurisvg/ha_lv: read failed after 0 of 4096 at 4096: Input/output error
LVM cluster daemon (clvmd) is not running. Make volume group "jurisvg" clustered anyway? [y/n]: y
  Volume group "jurisvg" successfully changed

For configuring the LVM resource, do we need a clvmd service running? then for the pacemaker I can find the /usr/sbin/clvmd service but couldnt start it.

[root@node1 ~]# /usr/sbin/clvmd
clvmd could not connect to cluster manager
Consult syslog for more information

Is there anyone know why my LVM resource have that such an error and failed?

Answer 1

I have Fix my issue with the following step to create a LVM resource. sdb is my shared disk I represent from the iscsi hosts.

[root@rhel-1 ~]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda             8:0    0   20G  0 disk
├─sda1          8:1    0    1G  0 part /boot
└─sda2          8:2    0   19G  0 part
  ├─rhel-root 253:0    0   17G  0 lvm  /
  └─rhel-swap 253:1    0    2G  0 lvm  [SWAP]
sdb             8:16   0   50G  0 disk
sr0            11:0    1  3.8G  0 rom  /mnt

Then I created a new partition for sdb.

[root@rhel-1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xf8a80986.

Command (m for help): p

Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33550336 bytes
Disk label type: dos
Disk identifier: 0xf8a80986

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
First sector (65528-104857599, default 65528):
Using default value 65528
Last sector, +sectors or +size{K,M,G} (65528-104857599, default 104857599):
Using default value 104857599
Partition 1 of type Linux and of size 50 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

[root@rhel-1 ~]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda             8:0    0   20G  0 disk
├─sda1          8:1    0    1G  0 part /boot
└─sda2          8:2    0   19G  0 part
  ├─rhel-root 253:0    0   17G  0 lvm  /
  └─rhel-swap 253:1    0    2G  0 lvm  [SWAP]
sdb             8:16   0   50G  0 disk
└─sdb1          8:17   0   50G  0 part
sr0            11:0    1  3.8G  0 rom  /mnt

Then I created physical volume, volume group and logical volume.

[root@rhel-1 ~]# pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created.
[root@rhel-1 ~]# vgcreate cluster_vg /dev/sdb1
  Volume group "cluster_vg" successfully created
[root@rhel-1 ~]# lvcreate -L 40G -n cluster_lv cluster_vg
  Logical volume "cluster_lv" created.

Create an ext4 file system on the logical volume cluster_lv.

[root@rhel-1 ~]# mkfs.ext4 /dev/mapper/cluster_vg-cluster_lv
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=8191 blocks
2621440 inodes, 10485760 blocks
524288 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2157969408
320 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,   2654208, 4096000, 7962624

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

After that I need to make an Exclusive activation for Volume group in a cluster, but before that I need to ensure that locking_type is set to 1 and use_lvmetad is set to 0 in the /etc/lvm/lvm.conf file. I use the following command to make the changes on lvm.conf file to apply on both nodes.

[root@rhel-1 ~]# lvmconf --enable-halvm --services --startstopservices
Warning: Stopping lvm2-lvmetad.service, but it can still be activated by lvm2-lvmetad.socket
Removed symlink /etc/systemd/system/sysinit.target.wants/lvm2-lvmetad.socket.

After that I need to make sure to add the volume groups other than my cluster vg as entries to volume_list in the /etc/lvm/lvm.conf. I have made this changes on my both nodes.

[root@rhel-1 ~]# grep "volume_list = " /etc/lvm/lvm.conf
        # volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
        volume_list = [ "rhel" ]
        # auto_activation_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
        # read_only_volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]

Rebuild the initramfs boot image to guarantee that the boot image will not try to activate a volume group controlled by the cluster. Also reboot is required after rebuild the initramfs.

[root@rhel-1 ~]# dracut -f -v
[root@rhel-1 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
[root@rhel-1 ~]# init 6

Create the LVM resource

[root@rhel-1 ~]# pcs resource create db2inst1_lvm LVM volgrpname=cluster_vg exclusive=true
Assumed agent name 'ocf:heartbeat:LVM' (deduced from 'LVM')

pcs status

[root@rhel-1 ~]# pcs status
Cluster name: juriscluster
Stack: corosync
Current DC: rhel-1 (version 1.1.16-12.el7_4.8-94ff4df) - partition with quorum
Last updated: Thu Mar 15 14:27:16 2018
Last change: Thu Mar 15 14:14:33 2018 by root via cibadmin on rhel-1

2 nodes configured
2 resources configured

Online: [ rhel-1 rhel-2 ]

Full list of resources:

 db2inst1_scsi  (stonith:fence_scsi):   Started rhel-1
 db2inst1_lvm   (ocf::heartbeat:LVM):   Started rhel-2

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled