How To Mount NVMe SSDs on EC2 instances

Situations exist when disk storage is not enough on instances. To expand storage, you have to either upgrade EC2 instance type or add SSD storages. Today, I will introduce how you can mount SSDs on EC2 instance and obtain a larger disk storage.

To access additional storage, you can check descriptions of each instance type that can be found in AWS documentation, and determine if SSD is available on the instance and how many SSDs the instance has. You can decide which instance type to use based on how much storage you need in your use case. Then, you need to mount SSDs on the instance so that they can show up in your file system and becomes accessible in code.

Mount one SSD on the instance

Mounting only one SSD is simpler. Take r5d.2xlarge instance for example: It has 1 x 300GB NVMe SSD. After you launch the instance, you need to run additional commands as below on the instance to make the SSD accessible:

sudo mkfs.ext4 /dev/nvme1n1
sudo mkdir -p /local
sudo mount /dev/nvme1n1 /local

You can find where partition SSD is located using fdisk l. The partition that has roughly 300GB is the one. In our example, SSD is on /dev/nvme1n1 partition. After figuring the partition, we create the file system on that partition using sudo mkfs.ext4 /dev/nvme1n1. Then, we mount it to a folder called /local so that everything saved in /local folder goes to the SSD. sudo mkdir -p /local creates /local folder and sudo mount /dev/nvme1n1 /local mounts SSD partition on /local.

Mount two SSDs or more on the instance

If you need more storage, you have to pick a more powerful instance. Lets say, you need more than 300GB storage, then r5d.2xlarge does not suffice your need. r5d.4xlarge is a better choice, which has 2 x 300GB NVMe SSDs.

The commands above used to mount one SSD do not apply when you mount 2 or more SSDs on the same folder /local. To mount both SSDs on /local and let it have an addition of 2 storage(300 + 300 = 600GB), there are multiple methods to achieve it. Following commands are one of them using RAID:

sudo yum install mdadm -y
sudo mdadm --create /dev/md0 --level=0 --name=MY_RAID --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1
sudo mkfs.ext4 -L MY_RAID /dev/md0
sudo mkdir -p /local
sudo mount LABEL=MY_RAID /local

RAID(Redundant Array of Independent Disks) is a way of logically putting multiple disks together into a single array. There are various types of RAID: RAID 0, RAID 1, RAID 5/6 and RAID 10. In our use case, we use RAID 0, which takes any number of disks and merges them together into one large volume. mdadm utility can be used to create and manage storage arrays using Linux’s software RAID capabilities. We install mdadm by running sudo yum install mdadm -y and create the RAID 0 array by running sudo mdadm --create /dev/md0 --level=0 --name=MY_RAID --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1. You can ensure RAID array is created successfully by checking /proc/mdstat file:

cat /proc/mdstat

Then, /dev/md0 is the RAID 0 array. sudo mkfs.ext4 -L MY_RAID /dev/md0 creates the file system and assign a label, which is used in mount later. sudo mkdir -p /local creates the /local folder and sudo mount LABEL=MY_RAID /local mounts RAID 0 array on /local folder. Now, we have both SSDs mounted on single point.

You can easily modify the commands if you mount more than 2 SSDs: update --raid-devices parameters to match the number of SSDs in mdadm create command and specify partitions of each SSD.

Mount SSD automatically after instance launch

Manually running commands to mount SSDs on instances is not scalable. You have the option to pass user data to the instance when you launch an instance in Amazon EC2. These user data is either shell scripts or cloud-init directives, which can be used to perform common automated configuration tasks and even run scripts after the instance starts. This means that you can put the commands above to mount SSDs in the user data and pass it to the instance, so that it is executed right after instance is launched automatically.

Reference

1. R5 instance description: https://aws.amazon.com/ec2/instance-types/r5/
2. Amazon EC2 user data: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html