IncusOS Amazon EC2 AMI Support - Implementation Plan

2024-12-17-ec2-ami-design.md

IncusOS Amazon EC2 AMI Support

Date: 2024-12-17 Status: Prototype Target Region: us-east-1

Overview

This plan describes adding Amazon EC2 AMI support to IncusOS, enabling deployment on AWS Nitro instances with full security features intact.

Goals

• Create an AMI that runs on EC2 Nitro instances
• Maintain security features: NitroTPM, UEFI Secure Boot, dm-verity
• Support configuration via EC2 user-data (IMDS)
• Provide a simple shell script for AMI creation (prototype)

Constraints

• Nitro instances only - Required for NitroTPM support
• us-east-1 - Single region for prototype
• Manual build - No CI automation initially

Architecture

New Files

incus-os/
├── mkosi.conf.d/
│   └── ec2.conf                    # EC2-specific mkosi overrides
├── mkosi.extra/ec2/
│   └── usr/lib/systemd/system/     # EC2-specific systemd units
├── incus-osd/api/seed/
│   └── provider_ec2.go             # IMDS seed provider
├── scripts/
│   └── create-ec2-ami.sh           # AMI creation script
└── doc/
    └── ec2-ami.md                  # Usage documentation

Component Responsibilities

Component	Purpose
ec2.conf	Add ENA drivers, NVMe support, serial console output
provider_ec2.go	Fetch seed YAML from IMDS user-data endpoint
create-ec2-ami.sh	Upload image to S3, import as snapshot, register AMI

Boot Flow on EC2

1. EC2 instance launches with user-data containing seed YAML
2. Kernel boots with UEFI, TPM measurements recorded to NitroTPM
3. incus-osd starts, detects EC2 environment
4. Fetches seed from http://169.254.169.254/latest/user-data
5. Applies configuration
6. Network auto-configured via VPC DHCP

Implementation Tasks

Task 1: EC2 mkosi Configuration

File: mkosi.conf.d/ec2.conf

Add EC2-specific packages and configuration:

[Distribution]
# Inherit from base

[Content]
Packages=
    # ENA network driver (usually built into Debian kernel)
    # NVMe tools for EBS
    nvme-cli
    # Serial console support

[Output]
# Same as base - raw disk image

File: mkosi.extra/ec2/usr/lib/systemd/system/serial-getty@ttyS0.service.d/override.conf

[Service]
ExecStart=
ExecStart=-/sbin/agetty -o '-p -- \\u' --keep-baud 115200,38400,9600 %I $TERM

Task 2: IMDS Seed Provider

File: incus-osd/api/seed/provider_ec2.go

package seed

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strings"
    "time"
)

const (
    imdsBase     = "http://169.254.169.254"
    imdsUserData = imdsBase + "/latest/user-data"
    imdsToken    = imdsBase + "/latest/api/token"
)

// EC2Provider fetches seed configuration from EC2 Instance Metadata Service
type EC2Provider struct{}

// Name returns the provider identifier
func (p *EC2Provider) Name() string {
    return "ec2"
}

// Detect checks if running on EC2 Nitro
func (p *EC2Provider) Detect() bool {
    // Check DMI for Nitro
    data, err := os.ReadFile("/sys/class/dmi/id/board_name")
    if err != nil {
        return false
    }
    return strings.Contains(string(data), "Nitro")
}

// FetchSeed retrieves seed YAML from IMDS user-data
func (p *EC2Provider) FetchSeed() ([]byte, error) {
    // IMDSv2: Get token first
    client := &http.Client{Timeout: 5 * time.Second}

    tokenReq, _ := http.NewRequest("PUT", imdsToken, nil)
    tokenReq.Header.Set("X-aws-ec2-metadata-token-ttl-seconds", "300")

    tokenResp, err := client.Do(tokenReq)
    if err != nil {
        return nil, fmt.Errorf("failed to get IMDS token: %w", err)
    }
    defer tokenResp.Body.Close()

    token, _ := io.ReadAll(tokenResp.Body)

    // Fetch user-data with token
    req, _ := http.NewRequest("GET", imdsUserData, nil)
    req.Header.Set("X-aws-ec2-metadata-token", string(token))

    resp, err := client.Do(req)
    if err != nil {
        return nil, fmt.Errorf("failed to fetch user-data: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode == 404 {
        // No user-data, return empty seed
        return nil, nil
    }

    return io.ReadAll(resp.Body)
}

Integration: Register provider in incus-osd/api/seed/seed.go:

func init() {
    RegisterProvider(&EC2Provider{})
}

Task 3: AMI Creation Script

File: scripts/create-ec2-ami.sh

#!/bin/bash
set -euo pipefail

# IncusOS EC2 AMI Creation Script
# Creates an AMI from a raw IncusOS disk image

REGION="us-east-1"
ARCHITECTURE="x86_64"

usage() {
    cat <<EOF
Usage: $0 --image <path> --bucket <s3-bucket> --name <ami-name>

Options:
    --image       Path to IncusOS raw disk image (.img)
    --bucket      S3 bucket for temporary image upload
    --name        Name for the resulting AMI
    --arch        Architecture (x86_64 or arm64, default: x86_64)
    --help        Show this help message

Required IAM permissions:
    - s3:PutObject, s3:GetObject, s3:DeleteObject
    - ec2:ImportSnapshot
    - ec2:DescribeImportSnapshotTasks
    - ec2:RegisterImage
    - ec2:DescribeSnapshots

Example:
    $0 --image mkosi.output/IncusOS_1.0.img --bucket my-ami-bucket --name "IncusOS-v1.0"
EOF
    exit 1
}

# Parse arguments
IMAGE=""
BUCKET=""
AMI_NAME=""

while [[ $# -gt 0 ]]; do
    case $1 in
        --image) IMAGE="$2"; shift 2 ;;
        --bucket) BUCKET="$2"; shift 2 ;;
        --name) AMI_NAME="$2"; shift 2 ;;
        --arch) ARCHITECTURE="$2"; shift 2 ;;
        --help) usage ;;
        *) echo "Unknown option: $1"; usage ;;
    esac
done

# Validate arguments
[[ -z "$IMAGE" ]] && { echo "Error: --image required"; usage; }
[[ -z "$BUCKET" ]] && { echo "Error: --bucket required"; usage; }
[[ -z "$AMI_NAME" ]] && { echo "Error: --name required"; usage; }
[[ ! -f "$IMAGE" ]] && { echo "Error: Image file not found: $IMAGE"; exit 1; }

# Check AWS CLI
command -v aws >/dev/null 2>&1 || { echo "Error: AWS CLI not installed"; exit 1; }

S3_KEY="incus-os/$(basename "$IMAGE")"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

echo "=== IncusOS AMI Creation ==="
echo "Image: $IMAGE"
echo "Bucket: $BUCKET"
echo "AMI Name: $AMI_NAME"
echo "Region: $REGION"
echo "Architecture: $ARCHITECTURE"
echo ""

# Step 1: Upload to S3
echo "[1/4] Uploading image to S3..."
aws s3 cp "$IMAGE" "s3://${BUCKET}/${S3_KEY}" --region "$REGION"
echo "Uploaded to s3://${BUCKET}/${S3_KEY}"

# Step 2: Import as snapshot
echo "[2/4] Importing as EBS snapshot..."
IMPORT_TASK=$(aws ec2 import-snapshot \
    --region "$REGION" \
    --description "IncusOS import ${TIMESTAMP}" \
    --disk-container "Format=RAW,UserBucket={S3Bucket=${BUCKET},S3Key=${S3_KEY}}" \
    --query 'ImportTaskId' \
    --output text)

echo "Import task: $IMPORT_TASK"
echo "Waiting for import to complete (this may take 5-15 minutes)..."

# Wait for snapshot import
while true; do
    STATUS=$(aws ec2 describe-import-snapshot-tasks \
        --region "$REGION" \
        --import-task-ids "$IMPORT_TASK" \
        --query 'ImportSnapshotTasks[0].SnapshotTaskDetail.Status' \
        --output text)

    PROGRESS=$(aws ec2 describe-import-snapshot-tasks \
        --region "$REGION" \
        --import-task-ids "$IMPORT_TASK" \
        --query 'ImportSnapshotTasks[0].SnapshotTaskDetail.Progress' \
        --output text 2>/dev/null || echo "0")

    echo "  Status: $STATUS, Progress: ${PROGRESS}%"

    if [[ "$STATUS" == "completed" ]]; then
        break
    elif [[ "$STATUS" == "error" ]]; then
        MSG=$(aws ec2 describe-import-snapshot-tasks \
            --region "$REGION" \
            --import-task-ids "$IMPORT_TASK" \
            --query 'ImportSnapshotTasks[0].SnapshotTaskDetail.StatusMessage' \
            --output text)
        echo "Error: Import failed - $MSG"
        exit 1
    fi

    sleep 30
done

SNAPSHOT_ID=$(aws ec2 describe-import-snapshot-tasks \
    --region "$REGION" \
    --import-task-ids "$IMPORT_TASK" \
    --query 'ImportSnapshotTasks[0].SnapshotTaskDetail.SnapshotId' \
    --output text)

echo "Snapshot created: $SNAPSHOT_ID"

# Step 3: Register AMI
echo "[3/4] Registering AMI..."

# Determine root device size from snapshot
SNAPSHOT_SIZE=$(aws ec2 describe-snapshots \
    --region "$REGION" \
    --snapshot-ids "$SNAPSHOT_ID" \
    --query 'Snapshots[0].VolumeSize' \
    --output text)

AMI_ID=$(aws ec2 register-image \
    --region "$REGION" \
    --name "$AMI_NAME" \
    --description "IncusOS - Immutable OS for Incus" \
    --architecture "$ARCHITECTURE" \
    --root-device-name /dev/xvda \
    --block-device-mappings "[{
        \"DeviceName\": \"/dev/xvda\",
        \"Ebs\": {
            \"SnapshotId\": \"${SNAPSHOT_ID}\",
            \"VolumeSize\": ${SNAPSHOT_SIZE},
            \"VolumeType\": \"gp3\",
            \"DeleteOnTermination\": true
        }
    }]" \
    --boot-mode uefi \
    --tpm-support v2.0 \
    --ena-support \
    --query 'ImageId' \
    --output text)

echo "AMI registered: $AMI_ID"

# Step 4: Cleanup S3
echo "[4/4] Cleaning up S3..."
aws s3 rm "s3://${BUCKET}/${S3_KEY}" --region "$REGION"

echo ""
echo "=== Complete ==="
echo "AMI ID: $AMI_ID"
echo "Region: $REGION"
echo ""
echo "Launch with:"
echo "  aws ec2 run-instances \\"
echo "    --image-id $AMI_ID \\"
echo "    --instance-type m5.large \\"
echo "    --user-data file://seed.yaml \\"
echo "    --region $REGION"

Task 4: Documentation

File: doc/ec2-ami.md

# Running IncusOS on Amazon EC2

IncusOS can run on Amazon EC2 using Nitro instances with full security features.

## Requirements

- **Nitro instance type** (m5, c5, r5, t3, etc.) - required for NitroTPM
- **UEFI boot support** - all Nitro instances support this
- **us-east-1 region** (prototype limitation)

## Creating an AMI

### Prerequisites

1. AWS CLI configured with appropriate permissions
2. An S3 bucket for temporary image storage
3. Built IncusOS image (`make build`)

### Build and Create AMI

\`\`\`bash
# Build IncusOS
make build

# Create AMI
./scripts/create-ec2-ami.sh \
    --image mkosi.output/IncusOS_*.img \
    --bucket your-s3-bucket \
    --name "IncusOS-$(date +%Y%m%d)"
\`\`\`

## Launching an Instance

### Prepare Configuration (seed.yaml)

\`\`\`yaml
provider: ec2

incus:
  enabled: true
  config: |
    config: {}
    networks: []
    storage_pools:
      - name: default
        driver: dir
    profiles:
      - name: default
        devices:
          root:
            path: /
            pool: default
            type: disk

applications:
  - incus
\`\`\`

### Launch Instance

\`\`\`bash
aws ec2 run-instances \
    --image-id ami-XXXXXXXXX \
    --instance-type m5.large \
    --user-data file://seed.yaml \
    --key-name your-keypair \
    --security-group-ids sg-XXXXXXXX \
    --subnet-id subnet-XXXXXXXX
\`\`\`

## Instance Types

Recommended instance types for IncusOS:

| Use Case | Instance Type | vCPUs | Memory |
|----------|---------------|-------|--------|
| Testing | t3.medium | 2 | 4 GB |
| Small workloads | m5.large | 2 | 8 GB |
| Medium workloads | m5.xlarge | 4 | 16 GB |
| Large workloads | m5.2xlarge | 8 | 32 GB |

## Security Considerations

- **NitroTPM**: Disk encryption keys are bound to the TPM
- **UEFI Secure Boot**: Enabled by default
- **EBS Encryption**: Recommended for additional protection
- **Security Groups**: Restrict access to Incus API (port 8443)

## Troubleshooting

### Instance won't boot

1. Verify instance type is Nitro-based
2. Check system logs: `aws ec2 get-console-output --instance-id i-XXX`

### No user-data applied

1. Verify user-data is valid YAML
2. Check incus-osd logs via serial console

### Network issues

1. Verify security group allows required ports
2. Check VPC/subnet configuration
\`\`\`

Implementation Order

1. Task 2: IMDS Provider - Core functionality needed first
2. Task 1: mkosi Config - Build configuration
3. Task 3: AMI Script - Creation tooling
4. Task 4: Documentation - Usage guide

Testing Plan

1. Build image with EC2 configuration
2. Run AMI creation script
3. Launch Nitro instance (m5.large recommended)
4. Verify boot via serial console
5. Verify seed applied from user-data
6. Test Incus functionality

Future Enhancements

• Multi-region support
• GitHub Actions integration (Packer)
• ARM64 (Graviton) support
• Marketplace listing