Cloud images for 28 systems: 8 traps we fixed

Our catalog now has 28 operating systems. Each one starts from the cloud image its distribution publishes, and cloud-init does the first-boot work: the account, your SSH key, the network, the hostname. Every image speaks cloud-init, yet no two behave quite the same way.

Here are the eight differences that cost us the most time while adding openSUSE, Oracle Linux and the others: what you see when one of them bites, and the fix our images now carry. If you build your own images, this is the list we wish we had started with.

1. The default user changes with the image

cloud-init creates one account at first boot, the default_user declared in the image's own configuration. It is never root, and the name depends on who built the image:

openSUSE caught us out: its cloud image creates sles, the name of SUSE's enterprise distribution, not opensuse. Alpine has no sudo at all; the account gets doas, which you use the same way (doas apk add htop). To know which account an image will create, read its cloud-init configuration. Here is the one from openSUSE Leap 16.0:

$ grep -A3 'default_user:' /etc/cloud/cloud.cfg
  default_user:
    name: sles
    lock_passwd: True
    gecos: sles Cloud User

Oracle Linux keeps it in /etc/cloud/cloud.cfg.d/90_ol.cfg instead. The console shows the right account next to each machine, and the images page lists it for every system.

2. A root on LVM that does not grow

Oracle's official KVM images come with a 37 GB disk: a 32 GB XFS root and 4 GB of swap, both on LVM. cloud-init grows a plain partition and its filesystem, but on this image it left the whole LVM chain alone. On a 40 GB disk we got a 31 GB root and a volume group with nothing free:

$ sudo vgs -o vg_name,vg_size,vg_free
  VG      VSize  VFree
  vg_main 35.80g     0

XFS cannot shrink, so the image will never fit a disk under 37 GB: Oracle Linux starts at our Starter plan. On bigger disks, the extra space was simply out of reach. We added a small service to the image that runs three commands at every boot (the real script finds the disk and the partition by itself):

growpart /dev/vda 4
pvresize /dev/vda4
lvextend --resizefs -l +100%FREE /dev/vg_main/lv_root

Every boot, not just the first: a disk enlarged by a plan change is picked up at the next restart. The first boot on a 40 GB disk looks like this in the journal:

ffxf-grow-root[936]: data blocks changed from 8336384 to 9122816
ffxf-grow-root[919]:   Extended file system xfs on vg_main/lv_root.
ffxf-grow-root[919]:   Logical volume vg_main/lv_root successfully resized.

The root went to 34.8 GB, then to 74.8 GB after we moved the disk to 80 GB and rebooted. When there is nothing left to take, growpart answers NOCHANGE and the service ends without touching anything. With SELinux enforcing, we set the SELinux labels on the files we add before the first boot, then checked the audit log: no denials.

3. sshd installed, but switched off

Arch's cloud image ships OpenSSH installed and the sshd unit disabled. cloud-init sets the password and drops your key in place, and nothing listens on port 22. The machine is up and unreachable.

Our first-boot configuration used to run systemctl try-restart after changing the SSH settings. That command leaves a stopped unit alone, which was fine on every other image and fatal on Arch. It now enables the service first:

runcmd:
  - systemctl enable --now sshd

4. No DNS, no packages

With a static network configuration and no nameserver in the cloud-init data, cloud-init writes no DNS settings at all. Debian and Ubuntu get by thanks to systemd-resolved. Alpine started with no /etc/resolv.conf: apk update exited with code 4, and every package planned for first boot failed, including the agent that reports disk and memory usage to the console.

We now send nameservers with the network configuration, and keep a safety net in bootcmd, which runs before any package is installed:

bootcmd:
  - |
    if [ ! -L /etc/resolv.conf ] && [ ! -s /etc/resolv.conf ]; then
      printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\n' > /etc/resolv.conf
    fi

Both tests matter. On Debian and Ubuntu, /etc/resolv.conf is a symlink managed by systemd-resolved, and writing over it would break name resolution there. One more detail: cloud-init's packages step gives up at the first failure and never tries again, so we retry the agent later in runcmd, once the network is up.

5. A config drive the kernel cannot see

Debian's genericcloud images are trimmed for virtual hardware and leave out drivers a cloud never needs, the old IDE controllers among them. Attach the cloud-init data as a CD-ROM on an IDE bus and the kernel never sees it. ds-identify finds no data source and disables cloud-init without a word on the console. The machine boots as localhost, with no IP address, and ssh.service fails because no host keys were generated.

The data has to sit on a bus the image's kernel knows, virtio or SCSI. On a machine that came up like this, these two tell you why:

cat /run/cloud-init/ds-identify.log
cloud-init status --long

6. Host keys that change after a reboot

cloud-init regenerates the SSH host keys every time it believes it runs on a new instance. On our platform, changing the hostname or the password, or adding a network card, gives the next boot a new instance ID. A customer then got WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED from a machine nobody had touched.

The fix is one line, written at the end of the first boot only, so that each machine still gets its own keys when it is created:

# /etc/cloud/cloud.cfg.d/99-ffxf-keep-hostkeys.cfg
ssh_deletekeys: false

7. A full upgrade during the first boot

cloud-init can upgrade every package on first boot (package_upgrade: true), and some platforms turn it on by default. The upgrade holds the apt or dnf lock for several minutes, and your first apt update hangs, or fails with Could not get lock /var/lib/dpkg/lock-frontend.

We leave it off. The images are up to date when we build them, and when to upgrade is your decision. If you script your first login, wait for cloud-init before touching the package manager:

cloud-init status --wait
sudo apt update && sudo apt full-upgrade -y

The rest of that first session is in our guide to securing an Ubuntu VPS.

8. Version 10 needs a recent processor

Rocky Linux 10, AlmaLinux 10, Oracle Linux 10 and CentOS Stream 10 are compiled for x86-64-v3, which includes AVX2. On a processor without it, nothing runs. We met it on a build machine while preparing the Oracle Linux 10 image, as soon as we tried to run a shell inside it:

Fatal glibc error: CPU does not support x86-64-v3

Our VMs see the host processor, which supports v3, so these systems run normally. If you run them on your own hypervisor with a generic virtual CPU model, check first. On that build machine, v3 is not marked as supported:

$ /lib64/ld-linux-x86-64.so.2 --help | grep x86-64-v
  x86-64-v4
  x86-64-v3
  x86-64-v2 (supported, searched)

Version 9 of the same distributions only asks for x86-64-v2 and still starts on that machine.