Adding a block device to DRBD with Corosync and Pacemaker for use with Highly Available NFS

Over the past few weeks I have been working with DRBD, Corosync and Pacemaker and adding addition block devices to DRBD to create new volumes and noticed that there was no solid methodology or steps to complete this task but after some trial and error I came up with steps to avoid taking an outage and adding disk and getting initial replication to occur from the primary node to the secondary node.

Prepare underlying block device

• If you’re partitioning the drive create a new partition on the underlying block device specifically if you do not have plans to use the entire device for DRBD.
• parted -s /dev/sdc — mklabel msdos \ mkpart primary xfs 1049kB 50GB \ set 2
• Zero out the block device to remove any known data on the drive
• dd if=/dev/zero of=/dev/sdc bs=512 count=4096K

Add configuration for a new volume to /etc/drbd.d/nfs.res

Note: This needs to be added to each DRBD node.

volume 1 {         
    device    /dev/drbd1;
    disk      /dev/sdc1; 
    meta-disk internal;  
} 

Note: If adding the entire block device, you can replace /dev/sdc1 with /dev/sdc. Note

Create DRBD Metadata and Initialize new volume

Note: create-md and adjust utilize the volume number for the minor information. These commands are run on both DRBD Nodes.

drbdadm create-md nfs/1 
drbdadm adjust nfs/1 

Note: If asked to overwrite metadata on new block device answer is “yes”

Locate Secondary DRBD Node using drbdadm status

drbdadm status

Disconnect Secondary Node from DRBD

drbdadm disconnect nfs 

Clear the bitmap UUID on the Primary Node which forces sync

Note: This command also takes minor device number which is the volume

drbdadm -v --clear-bitmap new-current-uuid nfs/1 

On the Secondary Node initiate a connection to create initial sync of the new block device

drbdadm secondary nfs 
drbdadm connect nfs 

On the Primary Node Disconnect and Promote Node to Primary and Reconnect

drbdadm disconnect nfs 
drbdadm primary nfs 
drbdadm connect nfs 

Verify DRBD Status that the nodes are synchronizing

drbdadm status
while true; do cat /sys/kernel/debug/drbd/resources/nfs/connections/<hostname>/1/proc_drbd; sleep 1; done 

Check and Verify that the new block device is in read/write mode

Note: If device is in read only mode use blockdev –setrw to put device into read/write mode.

blockdev --report /dev/drbd1 
blockdev --setrw /dev/drbd1 
cat /sys/block/drbd1/ro