aac-trust-anchor-publisher

aac-trust-anchor-publisher

The trust-anchor publisher is a small daemon that runs inside your organization and keeps the AAC control plane's copy of your public trust material current. It reads public keys and CA certificates from directories you control, packages them, signs the upload with your tenant-admin key, and pushes the result to the control plane at startup and whenever a file changes. Counterparties never contact the publisher: their sidecars fetch what it published from the control plane's public endpoints.

It plays up to two roles, and you configure only the roles your tenant needs:

• root-keys, for a tenant whose workloads mint authority chains. The publisher packages the root signing public keys as a JWK Set and publishes them so verifiers can check your chain roots.
• spiffe-bundle, for a tenant whose workloads present certificates. The publisher packages the CA certificates for one SPIFFE trust domain into a trust bundle and publishes it so receivers can verify your workload identities.

Run one publisher per tenant for root keys, and one per trust domain for bundles (see "One tenant, two trust domains" below). Each publisher is the single writer for what it publishes.

What the publisher holds, and what it never sees

• It holds the tenant-admin private key, the credential that authenticates uploads to the control plane. This is the only private key the daemon touches. Its public half is registered for your tenant when the tenant is registered (aac tenant register --tenant-admin-pubkey-file names it); until that registration exists every upload is rejected with ERR_TENANT_ADMIN_KEY_UNKNOWN.
• It reads public material only: root signing public keys and CA certificates. Root signing private keys stay with the workloads that mint chains and never pass through the publisher.
• The control plane never receives any private key of yours.

Keep the private halves apart: mount the tenant-admin private key and the public directories read-only into the publisher, and mount root signing private keys only into the sidecars that mint. When you retire an environment, revoke the public root key, stop the publisher, and remove its tenant-admin key under your retention procedure. The control plane keeps tenant-admin key history; there is no separate revocation call for it.

Install

Pick the channel that matches your host. Both run the same daemon.

Container (Kubernetes, Docker, or any orchestrator):

docker pull ghcr.io/cascadeauth/aac-trust-anchor-publisher:0.2.3

The image is multi-architecture (linux/amd64 and linux/arm64). Pull by version, not latest, so a redeploy runs what you tested.

pip (a Linux host you manage, where systemd runs the daemon):

sudo python3 -m venv /opt/aac/venv
sudo /opt/aac/venv/bin/pip install aac-trust-anchor-publisher
/opt/aac/venv/bin/aac-trust-anchor-publisher

Use a dedicated virtual environment: distribution Pythons refuse system-wide pip installs. The wheel installs the aac-trust-anchor-publisher command. Run with no configuration, as in the last line, it exits with code 2 and names AAC_TAP_TENANT_ID as missing. That is the expected result and proves the command is installed.

Configuration

The daemon is configured entirely through environment variables. It reads them once at startup and refuses to start on a mistake: a missing required variable, a half-configured role, zero roles, or an unreadable admin key prints the reason and exits with code 2. Restarting does not fix that class of error; fix the configuration. An unreachable or refusing control plane is not fatal: the daemon keeps polling and retries.

Always required:

Root-keys role. Set both or neither; setting one is a startup error:

SPIFFE-bundle role. Set the first three together; the fourth is optional:

Optional, with defaults:

Your onboarding material names the ingest URLs and, when it differs, the public trust host. Both are used over HTTPS with certificate verification; plain HTTP works only for local test deployments.

After a failed upload the daemon waits longer before the next attempt: the wait grows exponentially with random jitter, never past AAC_TAP_BACKOFF_CAP_SECONDS, and resets after the next accepted upload or unchanged poll.

Publishing root keys

Root-key directory

AAC_TAP_ROOT_KEYS_DIR holds one <key_id>.pub.pem file per root signing public key. The whole filename stem is the key id (dots are allowed): the JWK your counterparties see carries it as kid, and every chain your workloads mint names it, so choose ids you can rotate by, such as root-2026-q3. Keys are Ed25519 or EC P-256. Files that do not end in .pub.pem are ignored, and a file that fails to parse is skipped with a warning and re-checked at every poll.

Never place a private key in this directory. The daemon would not publish it, but it does not belong on the publisher host.

An empty directory is not published: the daemon logs no publishable keys and waits for files to appear.

Rotating root signing keys

Rotation is a transition set: add the new <key_id>.pub.pem beside the old one, wait one poll interval and confirm an ingest accepted line, roll your minting workloads to the new private key, then delete the old file. The control plane serves a removed key as DEPRECATED for 24 hours so chains signed just before the change still verify. Keep at least one key from the current set while you transition: a set that drops every known key at once is refused.

Publishing a SPIFFE bundle

CA directory

AAC_TAP_SPIFFE_BUNDLE_DIR holds one <anchor_id>.ca.pem file per CA certificate; the stem is the anchor id. Each certificate must parse, carry BasicConstraints CA=TRUE, and be unexpired; a file that fails is skipped with a warning. The bundle covers every workload identity under the configured trust domain, spiffe://<trust_domain>/*.

CA rotation is whole-document: add the new CA beside the old one (one bundle carries both), wait for the next upload, then delete the old file once every workload certificate has rolled. The bundle replaces atomically; the overlap lives in the document, not in a server-side grace period.

At every poll that finds CA files, the daemon reads the binding state for the trust domain from the public read URL (or the URL it derives from the ingest URL) and uploads only while the binding is ACTIVE, at the binding's current version. A 503 from the read URL while the binding is ACTIVE is normal before the first bundle exists. A revoked binding, a trust domain with no binding, or a read response that names a different trust domain stops publication with a diagnostic naming the cause; the daemon never falls back to another host.

Separate ingest and trust hosts

When the control plane serves public reads from a different host than signed uploads, set all three:

AAC_TAP_SPIFFE_TRUST_DOMAIN=tenant.example
AAC_TAP_SPIFFE_BUNDLE_INGEST_URL=https://api.example/v1/spiffe-bundle/ingest
AAC_TAP_SPIFFE_BUNDLE_READ_URL=https://trust.example/.well-known/spiffe-bundle/tenant.example

The read URL must be the public bundle URL for exactly the configured trust domain. The daemon sends plain GET requests there and signed POST requests only to the ingest URL. When the read URL is unset, the daemon derives it from the ingest URL: it removes /v1/spiffe-bundle/ingest (or, for any other ingest path, keeps only the scheme and host) and appends /.well-known/spiffe-bundle/<trust_domain>. Root-key-only publishers need no read URL.

One tenant, two trust domains

A tenant that runs workloads under its assigned hosted domain and under a custom domain runs two publisher processes, one per trust domain, with separate directories. Only one of them publishes root keys.

Adding a custom domain does not retire the hosted one. Workloads move to the custom domain by receiving new identities, not by renaming existing ones.

Rotating the tenant-admin key

The tenant-admin key signs every upload. The daemon reads it once at startup, and the control plane activates a replacement key the instant it is registered, with no grace period: the old key stops verifying at once. Rotation is therefore a short, deliberate sequence, and it differs between the two deployment paths.

Generate the new pair anywhere safe; only the public half leaves the host you will run on:

openssl genpkey -algorithm ed25519 -out tenant-admin.new.pem
openssl pkey -in tenant-admin.new.pem -pubout -out tenant-admin.new.public.pem

pip and systemd: stage the new private key, stop the service, register the public half with the AAC CLI, point the environment file at the new key, and start the service:

sudo install -m 640 -o root -g aac tenant-admin.new.pem /etc/aac/keys/tenant-admin.new.pem
sudo systemctl stop aac-trust-anchor-publisher
aac tenant rotate-admin-key --tenant-id tnt-550e8400-e29b-41d4-9716-446655440000 \
  --tenant-admin-pubkey-file tenant-admin.new.public.pem
sudo sed -i 's|^AAC_TAP_ADMIN_KEY_FILE=.*|AAC_TAP_ADMIN_KEY_FILE=/etc/aac/keys/tenant-admin.new.pem|' \
  /etc/aac/trust-anchor-publisher.env
sudo systemctl start aac-trust-anchor-publisher

Container: a container's environment and mounts are fixed when it is created, so overwrite the key at the existing bind-mount source and recreate the container. Do not repoint AAC_TAP_ADMIN_KEY_FILE, and do not rely on docker start, which can keep the old file:

docker stop aac-trust-anchor-publisher
aac tenant rotate-admin-key --tenant-id tnt-550e8400-e29b-41d4-9716-446655440000 \
  --tenant-admin-pubkey-file tenant-admin.new.public.pem
sudo install -m 640 -o root -g aac tenant-admin.new.pem /etc/aac/keys/tenant-admin.pem
docker rm -f aac-trust-anchor-publisher

Then repeat the docker run command from "Running in a container" unchanged; the paths it names now hold the new key.

Either way, the first upload after the restart proves the rotation: watch for ingest accepted. A rejection with ERR_TENANT_ADMIN_KEY_UNKNOWN or ERR_TENANT_ADMIN_SIGNATURE_INVALID means the daemon came back on the old key.

If the key is compromised, rotate first to kill the exposed key, then replace the file and restart as fast as practical. Only new uploads pause while the publisher is down; anchors already published keep serving, and agent traffic does not depend on this key.

Running under systemd

The daemon is a plain foreground process (no forking, no pid file, logs to stdout and stderr) and lets systemd own its lifecycle. Create /etc/aac/keys and /etc/aac/root-keys, copy the tenant-admin private key to /etc/aac/keys/tenant-admin.pem and your <key_id>.pub.pem files into /etc/aac/root-keys/ first; the complete unit file ships inside the wheel, so the rest is a short sequence:

sudo useradd --system --home /var/lib/aac aac
sudo tee /etc/aac/trust-anchor-publisher.env > /dev/null << 'EOF'
AAC_TAP_TENANT_ID=tnt-550e8400-e29b-41d4-9716-446655440000
AAC_TAP_ADMIN_KEY_FILE=/etc/aac/keys/tenant-admin.pem
AAC_TAP_ROOT_KEYS_DIR=/etc/aac/root-keys
AAC_TAP_ROOT_KEYS_INGEST_URL=https://api.example/v1/root-keys/ingest
EOF
sudo chmod 600 /etc/aac/trust-anchor-publisher.env
sudo chown -R root:aac /etc/aac/keys /etc/aac/root-keys
sudo chmod 750 /etc/aac/keys /etc/aac/root-keys
sudo chmod 640 /etc/aac/keys/tenant-admin.pem
sudo find /etc/aac/root-keys -name '*.pub.pem' -exec chmod 640 {} +
/opt/aac/venv/bin/aac-trust-anchor-publisher --print-systemd-unit | sudo tee /etc/systemd/system/aac-trust-anchor-publisher.service > /dev/null
sudo systemctl daemon-reload
sudo systemctl enable --now aac-trust-anchor-publisher
journalctl -u aac-trust-anchor-publisher -f

systemd reads the environment file as root, so it stays root:root with mode 600. The daemon itself, running as the aac user, opens the admin key and every public key file, so those are root:aac with mode 640 in directories the group can traverse (the find form enumerates the public keys under sudo, because your own shell cannot list a directory it can no longer read). A root-only admin key exits 2 at startup. A root-only public key is skipped with a warning while the remaining readable keys are still uploaded; nothing is published only when no readable key is left. Add the SPIFFE-bundle variables to the same file to enable that role.

The unit sets Restart=on-failure with RestartPreventExitStatus=2, so a configuration error stays down and visible instead of restart-looping. It runs as the dedicated aac user and sandboxes the process read-only (ProtectSystem=strict); the daemon never writes to the filesystem. If your virtual environment is not at /opt/aac/venv, edit ExecStart. Logs go to the journal only: journalctl -u aac-trust-anchor-publisher is the complete history.

Running in a container

Place the tenant-admin private key and the <key_id>.pub.pem files on the host first (/etc/aac/keys/tenant-admin.pem and /etc/aac/root-keys/ below), then bind-mount them read-only and pass the configuration as environment variables. The daemon needs outbound HTTPS to the control plane and no inbound ports. The image runs as a non-root aac user whose ids do not match your host's, so join the container to the host group that owns the key material instead of loosening file modes:

docker run -d --name aac-trust-anchor-publisher \
  --restart unless-stopped \
  --group-add "$(getent group aac | cut -d: -f3)" \
  -e AAC_TAP_TENANT_ID=tnt-550e8400-e29b-41d4-9716-446655440000 \
  -e AAC_TAP_ADMIN_KEY_FILE=/keys/tenant-admin.pem \
  -e AAC_TAP_ROOT_KEYS_DIR=/root-keys \
  -e AAC_TAP_ROOT_KEYS_INGEST_URL=https://api.example/v1/root-keys/ingest \
  -v /etc/aac/keys/tenant-admin.pem:/keys/tenant-admin.pem:ro \
  -v /etc/aac/root-keys:/root-keys:ro \
  ghcr.io/cascadeauth/aac-trust-anchor-publisher:0.2.3
docker logs -f aac-trust-anchor-publisher

On the host, the key material is root:aac with mode 640 and the directories mode 750 (groupadd --system aac first if the group does not exist). Every host path in a -v option must exist before the first run: Docker creates a missing path as a directory, and the daemon then fails with cannot read admin key file ... Is a directory. Prefer --restart unless-stopped over --rm: a container that exits 2 at startup keeps its logs. A container stuck in Restarting is the container face of exit 2: read docker logs, fix the configuration, and recreate.

What the log tells you

One line per upload attempt, per role:

Every upload succeeds or fails as a whole, and the daemon retries at the next poll, so an unreachable or refusing control plane needs no restart. Changes on the publisher host are different: the admin key and the environment are read once at startup, so a replaced key file or an edited variable takes effect only after you restart the service or recreate the container, as in the rotation section above. The proof of a successful publish is the control plane serving your material from the same public endpoints your counterparties read:

curl -fsS https://trust.example/.well-known/aac-root-keys/tnt-550e8400-e29b-41d4-9716-446655440000
curl -fsS https://trust.example/.well-known/spiffe-bundle/tenant.example

Root keys appear as {"keys": [...]}: one entry with aac_key_status ACTIVE per file in your directory, plus any key you removed within the last 24 hours, served as DEPRECATED until its grace period ends. Seeing the old key as DEPRECATED after a rotation is the expected result, not a stale publish.