Migrating from a small hard drive to a bigger hard drive usually means copying the raw data of the drive using dd, increasing the partition size using cfdisk, and then finally resizing the file system to fit the whole partition with something like resize2fs. This process is usually done while booted from another drive or live USB, but it is possible to modify partitions on mounted drives in relatively modern systems.

This process is always scary and time consuming, especially when booting from another drive. Any small mistake can brick your drive and cause data loss. And whether you use a nice GUI utility like gparted or not, there are many steps that can go very wrong if you’re not paying attention. The recommended backup step makes this process even longer.

All the complexity and potential for data loss made me appreciate EBS even more. It was a pleasent surprise when my file system was automatically the right size after a few button clicks in EBS. I didn’t even SSH into the machine and it was done. Just modify the EBS volume while the machine is running and reboot when it’s done (it is possible to skip the reboot but you would have to extend the partition manually).

So how does this work? Does EBS automatically modify the partition and resize the file system? Is the volume attached to a hidden EC2 instance that handles it for you? Is it something else?

It is your EC2 instance itself that extends the partition and resizes the file system. This is done automatically by cloud-init which is a program that comes preloaded on most AMIs. This program is in charge of initializing cloud instances and works on AWS, GCP, Azure, and others. It can take care of common tasks like retrieving instance metadata, setting up SSH keys, and it even executes UserData on AWS.

If you check out the log file at /var/log/cloud-init.log after increasing a volume size and rebooting, you will find something like the following.

Sep 03 23:58:49 cloud-init[2276]: cc_growpart.py[DEBUG]: No 'growpart' entry in cfg.  Using default: {'ignore_growroot_disabled': False, 'mode': 'auto', 'devices': ['/']}
Sep 03 23:58:49 cloud-init[2276]: util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/nvme0n1', '1'] with allowed return codes [0] (shell=False, capture=True)
Sep 03 23:58:49 cloud-init[2276]: util.py[DEBUG]: Running command ['growpart', '/dev/nvme0n1', '1'] with allowed return codes [0] (shell=False, capture=True)
Sep 03 23:58:50 cloud-init[2276]: util.py[DEBUG]: resize_devices took 0.116 seconds
Sep 03 23:58:50 cloud-init[2276]: cc_growpart.py[INFO]: '/' resized: changed (/dev/nvme0n1, 1) from 8587820544 to 10735304192
Sep 03 23:58:50 cloud-init[2276]: stages.py[DEBUG]: Running module resizefs (<module 'cloudinit.config.cc_resizefs' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_resizefs.pyc'>) with frequency always
Sep 03 23:58:50 cloud-init[2276]: handlers.py[DEBUG]: start: init-network/config-resizefs: running config-resizefs with frequency always
Sep 03 23:58:50 cloud-init[2276]: helpers.py[DEBUG]: Running config-resizefs using lock (<cloudinit.helpers.DummyLock object at 0x7fd3a1a50290>)
Sep 03 23:58:50 cloud-init[2276]: cc_resizefs.py[DEBUG]: resize_info: dev=/dev/nvme0n1p1 mnt_point=/ path=/
Sep 03 23:58:50 cloud-init[2276]: cc_resizefs.py[DEBUG]: Resizing / (xfs) using xfs_growfs /
Sep 03 23:58:50 cloud-init[2276]: cc_resizefs.py[DEBUG]: Resizing (via forking) root filesystem (type=xfs, val=noblock)
Sep 03 23:58:50 cloud-init[2452]: util.py[DEBUG]: Running command ('xfs_growfs', '/') with allowed return codes [0] (shell=False, capture=True)

Notice cloud-init detected the volume size changed, automatically called growpart with the right parameters to increase the partition size to fill the volume, detected the file system type, and called xfs_growfs to grow the file system.

cloud-init is configured with /etc/cloud/cloud.cfg which contains various configuration and a list of all the modules that should be executed. On most AMIs this includes the two modules we saw in the log: growpart and resizefs which are loaded from cc_growpart.py and cc_resizefs.py. In the source code you can see all the magic of detecting the size, file system, and choosing the right tools for the job.

This solution allows EBS to remain simple and file-system agnostic, while providing good yet configurable user experience. I was pretty impressed when I realized how it works.