AWS NVME mounted to different directory after reboot

Question

I'm trying to mount a few volumes to my instance and they are all NVME. I read that NVME volumes don't keeping their mapping the same, each time they are given serialized numbers randomly. The point is I need to keep the mapping consistent, it's for a db and 1 of the volumes suppose to keep the data. Now if I reboot the instance the volumes get mixed up, so it's possible that the volume that has the data will be mounted to a different directory and hence the db service starts and doesn't find any data.

Of course it happens also after creating an image, so I can't configure 1 instance and the spin up more using an image.

How can I force the mapping to be consistent? or stop using NVME? (I read this random serialization happens only with NVME)

Azize Azize · Accepted Answer · 2021-05-09T00:57:46

You need to use device UUID. See my example below.

I have 3 disks, 8 GB, 10 GB and 12 GB.
They show as device nvme0n1 (8 GB), nvme1n1 (10 GB) and nvme2n1 (12 GB).

$ lsblk
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1       259:0    0  10G  0 disk
└─nvme1n1p1   259:6    0  10G  0 part /mnt/disk10
nvme2n1       259:1    0  12G  0 disk
└─nvme2n1p1   259:7    0  12G  0 part /mnt/disk12
nvme0n1       259:2    0   8G  0 disk
├─nvme0n1p1   259:3    0   8G  0 part /
└─nvme0n1p128 259:4    0   1M  0 part

Look that I have a file on disk of size 10 GB called /mnt/disk10/file10.txt.
Also a file on disk of size 12 GB called /mnt/disk12/file12.txt.

$ ls -l /mnt/*
/mnt/disk10:
total 0
-rw-r--r-- 1 root root 0 May  9 00:37 file10.txt

/mnt/disk12:
total 0
-rw-r--r-- 1 root root 0 May  9 00:38 file12.txt

My fstab file use UUID to refer those disks, as you can see below.

$ cat /etc/fstab
# Disk 8 GB
UUID=7b355c6b-f82b-4810-94b9-4f3af651f629     /           xfs    defaults,noatime  1   1
# Disk 10 GB
UUID=2b19004b-795f-4da3-b220-d531c7cde1dc     /mnt/disk10           xfs    defaults,noatime  0   0
# Disk 12 GB
UUID=1b18a2f2-f48f-4977-adf8-aa483e1fa91f     /mnt/disk12           xfs    defaults,noatime  0   0

If you want to know what is the UUID for each device, use blkid, as you can see below.

$ blkid
/dev/nvme1n1: PTUUID="2e6aaa33" PTTYPE="dos"
/dev/nvme1n1p1: UUID="2b19004b-795f-4da3-b220-d531c7cde1dc" TYPE="xfs" PARTUUID="2e6aaa33-01"
/dev/nvme2n1: PTUUID="10565c83" PTTYPE="dos"
/dev/nvme2n1p1: UUID="1b18a2f2-f48f-4977-adf8-aa483e1fa91f" TYPE="xfs" PARTUUID="10565c83-01"
/dev/nvme0n1: PTUUID="1760802e-28df-44e2-b0e0-d1964f72a39e" PTTYPE="gpt"
/dev/nvme0n1p1: LABEL="/" UUID="7b355c6b-f82b-4810-94b9-4f3af651f629" TYPE="xfs" PARTLABEL="Linux" PARTUUID="a5dcc974-1013-4ea3-9942-1ac147266613"
/dev/nvme0n1p128: PARTLABEL="BIOS Boot Partition" PARTUUID="dc255fff-03c6-40e6-a8dc-054ec864a155"

Now I will stop my machine, force change device order and start it again.
Look how the disk changes the device name, but they are still being mounted on the same path, with the same files on it.

Before: nvme0n1 (8 GB), nvme1n1 (10 GB) and nvme2n1 (12 GB).
Now: nvme0n1 (8 GB), nvme1n1 (12 GB) and nvme2n1 (10 GB).

$ lsblk
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1       259:0    0  12G  0 disk
└─nvme1n1p1   259:1    0  12G  0 part /mnt/disk12
nvme2n1       259:2    0  10G  0 disk
└─nvme2n1p1   259:3    0  10G  0 part /mnt/disk10
nvme0n1       259:4    0   8G  0 disk
├─nvme0n1p1   259:5    0   8G  0 part /
└─nvme0n1p128 259:6    0   1M  0 part

$ ls -l /mnt/*
/mnt/disk10:
total 0
-rw-r--r-- 1 root root 0 May  9 00:37 file10.txt

/mnt/disk12:
total 0
-rw-r--r-- 1 root root 0 May  9 00:38 file12.txt

UUID is an attribute from filesystem, so any time you create an filesystem it will generate an UUID. Also any time you generate an AMI or snapshot, the UUID is the same, as it belongs to filesystem, not to EBS volume.

AWS NVME mounted to different directory after reboot

1 Answers