Timothy J. Aveni
121 lines
3.9 KiB
Markdown
121 lines
3.9 KiB
Markdown
# NixOS ZFS EC2 AMI Builder
|
|
|
|
This project publishes a matched EC2 AMI artifact:
|
|
|
|
- root volume: bootable NixOS EC2 image
|
|
- secondary volume: preseeded ZFS pool mounted at `/nix`
|
|
|
|
The root snapshot and `/nix` snapshot are registered together in one AMI block
|
|
device mapping. Runtime deployments should consume the AMI ID directly and
|
|
should not independently choose a `/nix` snapshot.
|
|
|
|
The image is intentionally project-agnostic. It provides NixOS, ZFS, SSM,
|
|
flakes, and systemd ordering so the Nix daemon waits for `/nix`. The final AMI
|
|
also disables the default shell-oriented `amazon-init` service and enables a
|
|
small generic service that treats EC2 user-data as `/etc/nixos/flake.nix`, then
|
|
runs `nixos-rebuild switch --flake /etc/nixos`.
|
|
|
|
## First-Boot Build Cache Hints
|
|
|
|
The first user-data switch is built by the AMI's existing Nix configuration,
|
|
before the target NixOS system has activated its own `nix.settings`. A caller
|
|
flake can optionally expose `quixosEarlyBuildNixConfig` to make that first
|
|
build use known binary caches without baking any project-specific cache into
|
|
the AMI:
|
|
|
|
```nix
|
|
{
|
|
quixosEarlyBuildNixConfig = {
|
|
hosts = {
|
|
"2600:..." = [ "nix-cache.internal.example.org" ];
|
|
};
|
|
substituters = [ "http://nix-cache.internal.example.org/cache" ];
|
|
trustedPublicKeys = [ "nix-cache.internal.example.org-1:..." ];
|
|
};
|
|
}
|
|
```
|
|
|
|
The AMI runner evaluates this output after writing `/etc/nixos/flake.nix`,
|
|
appends the host hints to `/etc/hosts`, and passes the substituters and trusted
|
|
public keys through `NIX_CONFIG` only for the first `nixos-rebuild switch`.
|
|
The target NixOS configuration should still declare the same cache in
|
|
`nix.settings` for all later rebuilds.
|
|
|
|
## Publish
|
|
|
|
The normal flow builds on a temporary EC2 instance, so your local machine does
|
|
not need to be the same architecture as the target AMI:
|
|
|
|
```bash
|
|
nix run ./tofu/ami-builder#publish-ami
|
|
```
|
|
|
|
`publish-ami.sh` applies this OpenTofu project to create a temporary builder
|
|
instance and `/nix` volume. The builder user data installs a ZFS-capable NixOS
|
|
boot generation, reboots once, seeds the ZFS `/nix` volume, installs the final
|
|
boot generation that expects `/nix`, and powers off.
|
|
|
|
After the instance is stopped, `publish-ami.sh` creates the final AMI from the
|
|
stopped instance, tags the AMI and snapshots, writes `artifact.json`, and then
|
|
destroys the temporary builder resources without a confirmation prompt. Use
|
|
`--keep-builder` when debugging.
|
|
|
|
The OpenTofu state in this directory owns only temporary builder resources.
|
|
The published AMI and its snapshots are created by the script and are not in
|
|
OpenTofu state, so `tofu destroy` here can clean up the builder without
|
|
deregistering the AMI.
|
|
|
|
Pass extra OpenTofu variables after `--`:
|
|
|
|
```bash
|
|
nix run ./tofu/ami-builder#publish-ami -- -- \
|
|
-var base_nixos_ami_id=ami-...
|
|
```
|
|
|
|
## Artifact Metadata
|
|
|
|
The final AMI is tagged with:
|
|
|
|
- `Architecture`
|
|
- `InputHash`
|
|
- `BuilderGitRev`
|
|
- `FlakeLockRev`
|
|
- `RootSnapshotId`
|
|
- `NixSnapshotId`
|
|
- `NixosSystemClosure`
|
|
|
|
The publisher also writes SSM parameters that point to the AMI ID:
|
|
|
|
- `/quixos/amis/<name>/<architecture>/recommended`
|
|
- `/quixos/amis/<name>/<architecture>/by-builder-rev/<rev>`
|
|
- `/quixos/amis/<name>/<architecture>/by-input-hash/<hash>`
|
|
|
|
Central services read the `recommended` parameter by default.
|
|
|
|
Do not deregister the AMI or delete either snapshot while runtime deployments
|
|
use it.
|
|
|
|
## Artifact Cleanup
|
|
|
|
List published AMIs and cross-reference qx-deploy runtime registry records:
|
|
|
|
```bash
|
|
nix run ./tofu/ami-builder#ami-artifacts -- list
|
|
```
|
|
|
|
Publish SSM parameters for an existing AMI, useful for AMIs created before SSM
|
|
publication was added:
|
|
|
|
```bash
|
|
nix run ./tofu/ami-builder#ami-artifacts -- publish-ssm --ami-id ami-...
|
|
```
|
|
|
|
Delete an unused AMI and its root and `/nix` snapshots:
|
|
|
|
```bash
|
|
nix run ./tofu/ami-builder#ami-artifacts -- delete --ami-id ami-...
|
|
```
|
|
|
|
The delete command refuses to delete AMIs referenced by runtime deployment
|
|
registry records unless you pass `--force`.
|