Cloud-Init: Fast, Consistent, and Automated VM Provisioning

Introduction #

Cloud‑init is the industry‑standard tool for automating the initial configuration of cloud virtual machines (VMs). It enables rapid, repeatable and consistent provisioning across diverse cloud platforms, ensuring that each VM conforms to your desired baseline from the moment it boots.

Purpose and Benefits #

Cloud‑init automates essential VM configuration tasks, eliminating manual setup and reducing human error. The main benefits include:

• Automated initial configuration
  Automatically performs tasks like creating users, setting SSH keys, updating network settings, installing packages and running scripts.
• Consistency across environments
  Ensures that every VM, across different regions or cloud providers, receives the same setup.
• Flexibility via user data
  Accepts custom instructions—shell commands, cloud‑config files, or scripts—which are executed at first boot or on every boot, depending on configuration.
• Broad compatibility
  Works with most Linux distributions (Ubuntu, CentOS, Debian, RHEL, etc.) and integrates seamlessly with major public clouds (AWS, Azure, GCP, Linode).

Core Concepts #

User Data

User data is configuration passed to an instance at launch. Cloud-init reads this data during the first boot to define how the VM should be provisioned.

Supported formats include:

• cloud-config YAML: Declarative, human-readable configuration marked with #cloud-config.
• Shell scripts: Raw command-line scripts, prefixed with #!, executed directly.
• MIME multi-part archives: Bundles multiple content types in a single payload (e.g. shell script + cloud-config).

Metadata Service

On boot, cloud-init contacts the cloud provider’s metadata service—a local HTTP endpoint (e.g. http://169.254.169.254)—to retrieve dynamic instance-specific data:

• Hostname
• Instance ID
• Region or availability zone
• Network configuration
• SSH keys

These values help tailor the configuration to each instance context.

Modules and Stages

Cloud-init uses a modular architecture with defined stages, each responsible for part of the provisioning process. Modules are grouped and executed in these stages.

Key stages:

• Init: Retrieves metadata and user data, prepares the environment.
• Config: Executes configuration modules (e.g. user creation, package installs).
• Final: Executes any remaining scripts or post-configuration tasks.

This structure ensures reliable and repeatable provisioning.

How It Works #

Cloud-init is the standard tool for automating the initial setup of cloud instances. It runs automatically when an instance boots for the first time, using configuration provided as user data.

1. Add User Data

During instance creation (e.g. on Linode), you paste a cloud-config.yaml file into the User Data field. This file defines everything the system should do after booting, such as creating users, installing packages, writing configuration files, setting the timezone, or configuring the firewall.

2. Runs on First Boot

When the instance starts for the first time, cloud-init processes the user data in several internal stages:

• Initial setup (local boot): Prepares the system environment, such as mounting disks and setting up networking.
• User data fetch: Loads the configuration from the metadata service or from the provided user data.
• Apply configuration: Executes the specified tasks like installing packages, configuring SSH, and creating users.
• Finalise: Completes provisioning, restarts services if needed, and reboots the system if configured to do so.

Cloud-init only runs on the first boot. This ensures a clean, automated setup and prevents it from reapplying configuration during future reboots unless explicitly triggered. It is ideal for provisioning consistent and secure systems in cloud environments.

Terminology #

Installation #

sudo apt install cloud-init -y

Verify installation:

cloud-init --version

Cloud-config File #

A cloud-config file is a YAML file that tells cloud-init what to do when a cloud instance starts. It automates setup tasks like creating users, installing packages, and configuring services.

Key Features #

• Human-readable and easy to write
• Supports a wide range of configuration options
• Allows execution of scripts and commands
• Enables setting up users, packages, networking, and more

File Structure #

The file follows YAML syntax and has two main parts:

1. Header (Required)

The very first line must be:

#cloud-config

This tells cloud-init to treat the file as cloud-config. Without this line, cloud-init will ignore the file.

2. Modules (Configuration)

After the header, you define your system settings using modules, which are written as YAML key-value pairs.

Format:

#cloud-config
top-level-module:
  config-option-1: config-value-1
  config-option-2: config-value-2
  list-option:
    - list-value-1
    - list-value-2

Example:

#cloud-config
users:
  - name: <user>
    groups: sudo
    shell: /bin/bash

packages:
  - git
  - nginx

runcmd:
  - systemctl enable nginx
  - systemctl start nginx

Common Modules

Frequently used cloud-init modules:

For the full list of available modules and their configuration options, refer to the official documentation: Cloud-init Modules Reference

Commands #

Syntax #

Cloud-init commands follow the pattern:

cloud-init <action> [OPTIONS]

General #

Show installed cloud-init version:

cloud-init --version

Report cloud-init status or wait on completion:

cloud-init status

Possible states: running, done, disabled, or error.

Options:

• --long Shows detailed status including errors and timestamps
• --wait Blocks until cloud-init completes before returning
• --format=json Outputs a structured JSON including detailed timing, errors, and module status

Execution #

Re-run cloud-init:

sudo cloud-init clean [OPTIONS]

Options:

• --logs
  • Deletes all cloud-init log files in /var/log/
  • Useful for clearing logs before capturing or debugging an image
• --seed
  • Deletes the seed directory, typically /var/lib/cloud/seed/
  • This is where cloud-init stores static metadata used to initialise a datasource
  • Useful if you plan to update or regenerate metadata from a new seed source
• --reboot
  • Reboots the system after cleaning
  • Ensures cloud-init runs fresh on the next boot

Debugging #

View cloud-init log:

sudo tail -f /var/log/cloud-init.log | ccze

View cloud-init output log:

sudo tail -f /var/log/cloud-init-output.log | ccze

Collect and tar all cloud-init debug info:

sudo cloud-init collect-logs

This produces an archive file (e.g. cloud-init.tar.gz) in the current directory or a specified location.

What it collects:

• Cloud-init logs:

  • /var/log/cloud-init.log

  • /var/log/cloud-init-output.log

• Runtime metadata:

  • /run/cloud-init

• User data:

  • /var/lib/cloud/instance/user-data.txt

• System information:

  • Installed cloud-init version

  • dmesg output (kernel messages)

  • journalctl logs (systemd journal)

Validate cloud-config file:

sudo cloud-init schema [OPTIONS]

Options:

Workflow #

Provision Ubuntu Server #

Overview

Build and deploy a secure, minimal Ubuntu server ready for web hosting or further configuration. The system will be provisioned automatically with:

• A non-root user (john) with sudo privileges and SSH key authentication
• Hardened SSH access (password login disabled, access limited to specific user)
• Hostname set to linode-test and timezone configured to Europe/Brussels
• Full system package update and upgrade on first boot
• Essential packages pre-installed (bat, ccze, git, nginx, ufw, unattended-upgrades)
• Strict inbound and outbound firewall rules configured with ufw
• Automatic security updates via unattended-upgrades
• Initial services configured and system rebooted post-setup

Step 1: Create Hashed Password

Generate a secure SHA-512 hashed password:

mkpasswd --method=SHA-512 '<password>'

Insert the hashed password into cloud-config.yaml:

users:
  - name: john
    passwd: '****'

Step 2: Paste Cloud Config

Paste the following YAML into the user data field when creating the instance:

#cloud-config

# Basic system setup
hostname: linode-test
timezone: 'Europe/Brussels'

# User setup configuration
users:
  - name: john
    sudo: ALL=(ALL) ALL
    groups: sudo
    shell: /bin/bash
    lock_passwd: false
    passwd: '****'
    ssh_authorized_keys:
      - ssh-ed25519 **** [email protected]

# Package management
package_reboot_if_required: true
package_update: true
package_upgrade: true
packages:
  - bat
  - ccze
  - git
  - nginx
  - ufw
  - unattended-upgrades

# Write files to the instance
write_files:
  # Configure unattended upgrades
  - path: /etc/apt/apt.conf.d/99-custom-unattended-upgrades
    content: |
      Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";
      };
      Unattended-Upgrade::Remove-Unused-Dependencies "true";
      Unattended-Upgrade::Automatic-Reboot "true";
      Unattended-Upgrade::Automatic-Reboot-Time "02:00";
    owner: root:root
    permissions: '0644'

  # Harden SSH configuration
  - path: /etc/ssh/sshd_config.d/99-hardening.conf
    content: |
      LoginGraceTime 30s
      PermitRootLogin no
      AllowUsers john
      PubkeyAuthentication yes
      PasswordAuthentication no
    owner: root:root
    permissions: '0644'

# Commands to run at the end of the cloud-init process
runcmd:
  # Record the instance creation time
  - date > /etc/birth_certificate

  # Enable and start NGINX
  - systemctl enable nginx
  - systemctl start nginx

  # Configure firewall rules
  - ufw default deny incoming
  - ufw default deny outgoing
  - ufw limit 22/tcp
  - ufw allow 80/tcp
  - ufw allow 443/tcp
  - ufw allow out 22/tcp
  - ufw allow out 53/tcp
  - ufw allow out 53/udp
  - ufw allow out 80/tcp
  - ufw allow out 443/tcp

  # Enable firewall without prompt
  - ufw --force enable

  # Apply SSH configuration changes
  - systemctl restart ssh

# Reboot the instance after configuration
power_state:
  mode: reboot
  message: Rebooting after initial setup
  timeout: 30
  condition: True

Step 3: Verify Execution

Ensure your cloud-init configuration was applied successfully by checking the following:

Cloud-init status:

cloud-init status

Expected output:

status: done

Hostname:

hostnamectl

System time:

timedatectl

or:

date

Group membership:

groups <user>

SSH public key:

cat /home/<user>/.ssh/authorized_keys

Packages:

for cmd in batcat ccze git nginx ufw unattended-upgrades; do
  if command -v "$cmd" >/dev/null; then
    echo "$cmd: Installed"
  else
    echo "$cmd: Not found"
  fi
done

UFW firewall:

sudo ufw status verbose

Unattended-upgrades config:

cat /etc/apt/apt.conf.d/99-custom-unattended-upgrades

Unattended-upgrades service status:

systemctl status unattended-upgrades.service

SSH disabled password login:

ssh -i ~/.ssh/linode-test root@<pub_ipv4>

Expected output:

root@<pub_ipv4>: Permission denied (publickey).

NGINX status:

systemctl status nginx

View instance creation timestamp:

vim /etc/birth_certificate

Best Practices #

Recommended Approaches #

• Always start with #cloud-config
  This tells cloud-init your file is a configuration script. Without it, your instructions may not run properly.
• Write repeatable scripts
  Make sure running your commands multiple times won’t cause errors or unexpected changes. This keeps your setup stable even if cloud-init runs again.
• Use built-in cloud-init features instead of complex scripts
  Cloud-init has modules like users, packages, and write_files designed to do common tasks cleanly. Use these instead of long shell scripts whenever possible.
• Put complicated commands into separate scripts
  If you need complex setup steps, put them in standalone scripts and call them from cloud-init. This makes your config easier to read and maintain.
• Test your config before deploying
  Use tools like cloud-init devel and test in a virtual machine or sandbox. This prevents surprises when launching real servers.
• Backup config files before changing them
  Always save a copy before modifying important files like SSH settings. This helps you recover if something goes wrong.
• Write config files safely
  Use temporary files and then move them into place to avoid corrupt or partial files if something interrupts the process.
• Set correct permissions on files
  SSH keys and other sensitive files must be owned by the right user and have tight permissions (like 600 or 644). Otherwise, they won’t work or could be a security risk.
• Disable root login and password authentication for SSH
  Use SSH keys and allow only non-root users to log in. This greatly improves security.
• Pin package versions if needed
  If your application depends on specific versions, specify those to avoid unexpected updates breaking your system.
• Check logs to troubleshoot
  If something doesn’t work, look at /var/log/cloud-init.log and /var/log/cloud-init-output.log for clues.

Common Mistakes to Avoid #

• YAML syntax errors
  YAML is sensitive to spaces and indentation. Even a small mistake can cause your config to fail silently.
• Running network-dependent commands too early
  Some commands require the network but might run before it’s ready. Use runcmd instead of bootcmd for these.
• Writing long shell scripts inline
  It’s hard to read and debug. Use external scripts whenever possible.
• Not testing before production
  Always test your config in a safe environment before using it on real servers.
• Package manager conflicts
  If another process is using apt or yum during cloud-init, package installation may fail. Consider waiting or retrying.
• Overwriting user data unintentionally
  Some cloud providers append user data instead of replacing it. Be explicit about file contents.
• Ignoring version differences
  Cloud-init behaves slightly differently depending on OS and version. Always check your environment.
• Forgetting about reboot requirements
  Some updates or changes need a reboot to apply. Use package_reboot_if_required or handle reboots explicitly.
• Leaving secrets in plain text
  Don’t put passwords or keys directly in cloud-init. Use secure vaults or encryption.
• Weak SSH security
  Leaving password login or root SSH access enabled makes your server vulnerable.

Performance Optimisation Tips #

• Limit Package Installations
  Install only essential packages during initial provisioning to minimise download and install time:
  • Avoid large packages or development tools unless strictly required
  • Defer non-critical packages to later stages or use configuration management tools
• Use Pre-Baked Images
  Whenever possible, build custom images with commonly used software already installed. This avoids repeated installation at boot time:
  • Use tools like Packer or cloud provider image builders
  • Include security updates and dependencies during image build
• Minimise runcmd and bootcmd Tasks
  Keep these sections lightweight:
  • Avoid long-running scripts
  • Use backgrounding or defer tasks to final-message scripts or external automation
• Reduce Metadata and Network Wait Times
  Configure cloud-init to reduce timeouts and retries if your metadata service is fast and reliable:
  • Set shorter timeouts in /etc/cloud/cloud.cfg.d/ for network or datasource timeouts
• Optimise YAML Structure
  Clean, minimal configuration reduces processing time:
  • Avoid unnecessary keys
  • Validate YAML with tools like yamllint before deployment

Resources #

Official documentation:
https://cloudinit.readthedocs.io/en/latest/