bootstrap-state/: one-time project that creates the S3 bucket used for central OpenTofu state.
ami-builder/: submodule that publishes a project-agnostic NixOS EC2 AMI with a matched root snapshot and preseeded ZFS /nix snapshot.
central/: singleton central infrastructure used by runtime deployments. Today this owns the S3 bucket for runtime deployment state/registry, the long-lived encrypted EBS volume for central /persist state such as Caddy ACME material and SSH host keys, the platform VPC/subnets/security groups, and the optional central app-server singleton. This is also the intended home for future central services such as auth, public site, VPCs, or other shared platform infrastructure.
nix-cache/: singleton Attic binary cache service with its own lifecycle. It runs on a small NixOS EC2 instance, stores Attic blobs in a dedicated S3 bucket, keeps service state under its own /persist volume, and only accepts HTTP from the platform VPC.
runtime/: Quixos deployment stack. It consumes a published AMI for the system image and creates app/runtime resources separately. The deploy wrapper can resolve that AMI from the AMI-builder submodule revision and target architecture.

Destroying the runtime stack destroys the EC2/app/runtime resources only. It must not deregister the AMI or delete the AMI root or /nix snapshots. Those belong to the image-builder state. It also must not delete central infrastructure such as the runtime state bucket or central persist volume; those belong to central.

Central Infrastructure

Bootstrap the S3 bucket for central state once:

cd tofu/bootstrap-state
cp terraform.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars
tofu init
tofu apply

bucket_name should be a stable, globally unique operator/team value. Commit the example, not your local terraform.tfvars.

Then initialize central against that bucket. OpenTofu backend config cannot read normal Terraform variables or other state directly, so use the wrapper to render backend.hcl from the bootstrap outputs and run tofu init:

cd ../central
./init-backend.sh

backend.hcl.example shows the generated shape if you need to do it manually.

Create central infrastructure before runtime deploys:

tofu apply

The deploy wrapper reads tofu/central outputs unless you pass explicit overrides:

runtime_state_bucket_name: S3 bucket for runtime deployment state and registry.
persist_volume_id: EBS volume mounted by the central app server at /persist.
platform_vpc_id and platform_runtime_subnet_id: the managed VPC/subnet intended for runtime deployments.
central_proxy_security_group_id: when the central app server is enabled, runtime deployments can use this to accept HTTP/HTTPS only from the proxy while still keeping public IPv4 egress.

Nix Cache

Initialize the cache stack against its own state key:

cd tofu/nix-cache
./init-backend.sh

Then plan or apply it independently:

tofu plan
tofu apply

By default the stack:

reads the platform VPC and central subnet from tofu/central remote state;
reads the recommended AMI from /quixos/amis/nixos-zfs-ec2/arm64/recommended;
creates a t4g.nano cache host;
creates a dedicated Attic S3 bucket;
creates a dedicated /persist EBS volume for Attic SQLite metadata, Caddy state, and the Attic signing secret;
serves http://nix-cache.internal.quixos.org/quixos only to the platform VPC.

The cache bootstraps itself on first boot. It creates/configures the public Attic cache named quixos, then publishes:

/quixos/nix-cache/public-key: Nix substituter public key.
/quixos/nix-cache/central-push-token: token intended for central-only uploads.

Use the hosts_entry output to add an internal /etc/hosts mapping on clients instead of creating private Route53 records.

Central treats the nix-cache as required by default. If the cache host has not published /quixos/nix-cache/public-key yet, tofu/central fails instead of silently applying without cache wiring. For the first bootstrap apply only, set nix_cache_bootstrap_optional = true; remove it once the cache has published its key.

The central app server uses the recommended AMI published by the AMI builder:

route53_zone_id = "Z..."

By default, central reads the AMI ID from SSM parameter /quixos/amis/nixos-zfs-ec2/arm64/recommended, which is written by tofu/ami-builder#publish-ami. Set ami_id only to pin or override that recommended artifact. app_server_ami_id remains available only when the app-server needs to differ from other central EC2 services.

Runtime deployments share this bucket but not a state object. Each deployment uses its own key:

runtime/deployments/<deployment-id>/terraform.tfstate

The wrapper initializes tofu/runtime with OpenTofu's S3 backend and use_lockfile=true, so concurrent operations against the same deployment key are locked. The registry is stored as per-deployment JSON objects under:

registry/deployments/<deployment-id>.json

Use the same aws_region, vpc_id, and subnet_id values as the runtime stack if you override them. The central persist volume must be in the same availability zone as the runtime EC2 instance.

Runtime Deploy

Create tofu/runtime/terraform.tfvars:

aws_region      = "us-west-2"
route53_zone_id = "Z..."

# Optional, if the package tree is private:
package_tree_git_username = "quixos"
package_tree_git_password = "..."

Keep deployment-specific values out of terraform.tfvars. Local tfvars are operator config: AWS region, Route53 zone, VPC/subnet overrides, package git credentials, instance sizing, and similar static environment inputs.

qx-deploy owns deployment choices: deployment id, runtime state key, package-tree ref, Quixos ref, AMI, generated domain, dev mode, and SSH settings. For each run it resolves those values, rejects local tfvars that try to set deployment-owned keys such as subdomain or dev_mode, writes a temporary zz-qx-deploy.auto.tfvars.json, plans to a saved plan, and applies that saved plan. The generated tfvars file is removed after the run.

Then:

cd tofu/runtime
nix run ../..#quixos-deploy -- deploy --packages-ref master

nix run ../..#quixos-deploy runs the Nix-provided deploy command, which supplies aws, git, node, and tofu. The command itself is TypeScript bundled by the root flake. It prompts when an interactive choice is missing and prints the equivalent noninteractive command before planning.

By default, qx-deploy assigns one DNS name per deployment:

prod: <deployment-id>.<base-domain>
dev: <deployment-id>.dev.<base-domain>

Pass --base-domain <name> to override the base domain from the deploy command, or --domain <name> to override the full generated name for one deployment.

Useful registry commands:

nix run ../..#quixos-deploy -- list
nix run ../..#quixos-deploy -- list --include-destroyed
nix run ../..#quixos-deploy -- ssh
nix run ../..#quixos-deploy -- ssh --deployment-id test-deployment
nix run ../..#quixos-deploy -- doctor
nix run ../..#quixos-deploy -- doctor --deployment-id test-deployment
nix run ../..#quixos-deploy -- destroy --deployment-id test-deployment
nix run ../..#quixos-deploy -- destroy --deployment-id test-deployment --detach

list prints deployment id, status, mode (dev, prod, or unknown), URL, and state key. The ssh command defaults to quixos-dev for dev deployments and quixos-admin for prod deployments; override with --ssh-user.

The deploy wrapper resolves AMIs by listing owned EC2 images tagged with the selected project and target architecture. In interactive mode it always asks which AMI to use, ranking images built from the current AMI-builder submodule HEAD first. Noninteractive deploys must pass --ami-id ami-..., or --auto-select-ami to select the newest image that exactly matches the current AMI-builder rev. The selected value is written to resolved-ami.auto.tfvars.json, which is intentionally local state and ignored by Git. The ami_id Terraform variable has a null default so tofu destroy does not prompt for an AMI just to destroy resources already recorded in state.

The deploy wrapper resolves the package-tree ref. If .quixos-package-tree.json contains quixos.rev, that rev is deployed. Use --quixos-ref master to override it.

If the package tree or its submodules are private, provide HTTPS credentials through the wrapper. The token is written to /persist/secrets/quixos-git.env and consumed by the package-tree systemd unit through GIT_ASKPASS.

QUIXOS_PACKAGE_TREE_GIT_USERNAME=oauth2 \
QUIXOS_PACKAGE_TREE_GIT_TOKEN=... \
nix run ../..#quixos-deploy -- deploy --packages-ref master

Runtime Shape

The runtime AMI is expected to provide /nix from the AMI-attached nixos-zfs-nix ZFS volume from first boot. Runtime user data creates/imports no filesystems; it is a NixOS flake consumed by the AMI's user-data-flake service.

The runtime /persist volume contains deployment-local Quixos workspace, secret, package state, service state, and dev home data. It uses a ZFS pool named quixos-persist with a quixos-persist/persist dataset. Runtime NixOS initializes that pool on the attached runtime persist EBS volume if the volume is blank, imports it if it already exists, and fails on unknown existing filesystem signatures.

Runtime /persist survives runtime reboots, but it is scoped to one runtime deployment and is destroyed with that deployment. It is never attached to central or to another runtime deployment. Central has its own /persist volume with central lifecycle.

The visualizer static derivation uses project-visualizer/yarn-project.nix, generated by yarn-plugin-nixify.