7. Cluster Authentication with the Bundled Auth Stack

  • Status: Accepted

  • Date: 2026-08-29

  • Related: ADR 0009 (authentication service), docs/source/Security.md

7.1. Problem

This section describes the state before this design was implemented.

ADR 0009 defines one logical issuer per cluster. It defines the node roles standalone, owner, and follower. The runtime already contains the required pieces: role resolution, the inert identity App on followers, and bearer validation at the gateway and at the target. Three gaps remain. Each gap blocks a working multi-node deployment of the bundled stack.

  1. script/docker/docker-compose.yaml starts a master and three slaves. Every node keeps the default role standalone. Each node then runs its own Dex process with its own signing keys. The issuer string is identical on all nodes, but the keys are different. A token from one node fails signature validation on another node. Cross-node forwarding cannot authenticate.

  2. Joining a node to the owner requires manual edits of auth-stack.yaml and of environment variables. There is no standard setup action. setup.sh has no role option.

  3. Authorization data (authorization.yaml) is node-local. No sync mechanism exists. The documentation requires “shared authorization data”. It does not say how an operator keeps the nodes consistent.

7.2. Goals

  • Provide one standard install-time action. The action configures a node as the authentication owner or as a follower of an owner.

  • Followers must not run Dex. Followers must not create built-in credentials. Followers must verify tokens from the owner’s issuer.

  • Show the owner/follower model and authenticated forwarding in the packaged docker-compose cluster.

  • Document the consistency procedure for authorization data.

7.3. Non-goals

  • Automatic authorization-data replication between nodes. Every mutation endpoint writes node-local state today. Replication needs a separate design. This document defines the manual procedure only.

  • Active-active authentication. One active writer is allowed. See ADR 0009 and Security.md.

  • Windows. The bundled stack ships on Linux and macOS. Windows installations stay external-issuer consumers. setup.ps1 stays unchanged.

7.4. Topology

                 ┌─ owner node ────────────────────────────────┐
 clients ──────► │ Agent :6060 (public TLS)                    │
                 │   └─ /auth/* ──► Dex 127.0.0.1:6062 (loopback) │
                 │ Engine :6058 HTTPS + WSS / :6059 TCP         │
                 └──────────────┬───────────────────────────────┘
                                │ forwarded requests (TLS TCP :6059,
                                │ msgpack, bearer unchanged)
                 ┌─ follower nodes ────────────────────────────┐
                 │ Engine only. identity App holds inert.       │
                 │ Discovery/JWKS fetched from the owner.       │
                 └──────────────────────────────────────────────┘

  • Exactly one node is standalone or owner (is_auth_owner). Only that node runs Dex. Only that node bootstraps credentials (src/auth/appmesh-auth.sh).

  • Every node configures the same issuer string. Tokens carry iss equal to that string. Every Engine validates tokens locally against it.

  • access_url is per-node routing. It can differ from the issuer: the owner can use its loopback Dex, and followers use a route to the owner.

Two routes give followers access to the owner:

Route issuer / access_url When to use
A. Agent proxy (default) https://owner.example.com:6060/auth TLS front door. This route matches the rule "the issuer listener stays on loopback, Agent is the only public proxy".
B. Protected network http://<owner-cluster-address>:6062/auth with APPMESH_AUTH_LISTEN bound to the cluster interface Container or compose networks and private networks only. Do not expose the plaintext listener publicly.

The docker-compose cluster uses route B. The compose network is a protected cluster network.

7.5. Install and configuration

7.5.1. Owner

sudo /opt/appmesh/script/setup.sh --auth-mode builtin --auth-role owner
# Optional public issuer (default: http://127.0.0.1:6062/auth):
sudo /opt/appmesh/script/setup.sh --auth-mode builtin --auth-role owner \
     --oidc-issuer https://owner.example.com:6060/auth

  • --auth-role owner persists APPMESH_AUTH_ROLE=owner in the daemon environment file. The resolution order stays: environment first, then auth-stack.yaml (AuthorizationStore::resolveAuthRole, appmesh-auth.sh).

  • The issuer is one setting that controls three things: the issuer that Dex advertises in discovery, the iss value that the Engine verifies, and the path that the Agent reverse proxy mounts.

7.5.2. Follower (the join action)

sudo /opt/appmesh/script/setup.sh --auth-mode builtin --auth-role follower \
     --oidc-issuer   https://owner.example.com:6060/auth \
     --oidc-access-url https://owner.example.com:6060/auth \
     --oidc-browser-entry https://owner.example.com

Effects:

  • The identity App stays enabled and inert (hold_system_app). It stays healthy, starts no local Dex, writes no authentication state, and does not trigger the restart policy.

  • The Engine fetches discovery and JWKS through access_url. It rejects every token whose iss differs from the configured issuer.

  • Startup tolerates an unreachable owner for 120 seconds (AppMeshDaemon::prewarmAuthentication). Startup fails after that period. Start followers after the owner, or rely on the restart policy.

If you omit --auth-role, the setup keeps the current role selection. The packaged auth-stack.yaml default (standalone) then applies.

7.5.3. Configuration matrix

Setting Owner Follower
AuthStack.role / APPMESH_AUTH_ROLE owner follower
OIDC.issuer / APPMESH_AUTH_ISSUER canonical issuer identical string
OIDC.access_url / APPMESH_AUTH_ACCESS_URL loopback http://127.0.0.1:6062/auth is valid a route that reaches the owner's Dex
OIDC.tls_verify, OIDC.ca_path as the route requires as the route requires
Dex process runs as identity System App never runs
first-admin enrollment window open until claimed, loopback only none (AuthorizationStore opens it only for standalone/owner)

7.6. Forward authentication

The sequence below describes appm -H <gateway> -F <target> ....

  1. The CLI sends GET /appmesh/auth/config with X-Target-Host: <target>. Two GET requests can forward without a bearer: GET /appmesh/auth/config and GET /.well-known/oauth-protected-resource. The gateway rejects every other anonymous forward request (Worker::isPublicForwardRequest).

  2. The CLI gets tokens directly from the shared authentication service. Issuer-side discovery is a direct client-to-issuer fetch. It never forwards. Session reuse requires a match of issuer, client ID, and audience with the stored session.

  3. The CLI sends the request with Authorization: Bearer and X-Target-Host: <target>.

  4. The gateway validates the bearer before it opens an outbound connection. HTTP and TCP requests are verified at the gateway (Worker::process). A WSS session is authenticated at the upgrade. A forwarded WSS frame repeats the same bearer. The bearer must resolve to the principal that is pinned to the connection.

  5. The gateway erases X-Target-Host and injects X-AppMesh-Forwarded: 1 plus a route UUID for event correlation. The target never sees a forwarding selector. The target cannot re-forward. A forwarding loop cannot occur. No TTL and no visited list is necessary.

  6. The gateway forwards the request over its TLS TCP (msgpack) channel to the target’s TCP port. The Authorization header is forwarded without a change.

  7. The target validates the same bearer through the normal route path. It checks iss for an exact match, checks aud: appmesh-api, and fetches JWKS from its own access_url, which points to the owner. It resolves the principal from its local authorization data and applies permissions. X-AppMesh-Forwarded marks the request as forwarded. Direct-only operations reject it. Examples: first-admin enrollment and workflow capability.

Gateway validation is admission control. It prevents anonymous use of the outbound connection. Target validation is the authorization decision. The gateway and the target share no secret. They only agree on the issuer.

7.7. Authorization data consistency

authorization.yaml is node-local. The principal and role APIs write the node that serves the request. A forwarded POST /appmesh/principal/<id> changes only the target node. Apply this rule:

  • Mutate authorization data on one node. The owner is the natural administrative entry point.

  • Distribute the change in one of two ways: apply the same API calls on every node, or copy the owner’s runtime work/config/authorization.yaml to the followers and restart their daemons. Keep the file mode at 0600. Keep the runtime user as owner. A follower bootstrap creates no work/config directory; create it first (mkdir -p <home>/work/config) before the first copy.

  • A follower without a principal entry authenticates a user. It grants nothing beyond the configured provisioning policy (explicit-or-minimal by default).

Automated replication is out of scope here. It needs its own design, for example watch-and-push or mutation forwarding to a configured writer node.

7.8. docker-compose cluster

script/docker/docker-compose.yaml changes:

  • appmesh_master: APPMESH_AUTH_ROLE=owner, APPMESH_AUTH_ISSUER=http://appmesh_master:6062/auth, and APPMESH_AUTH_LISTEN=0.0.0.0:6062. The compose network only reaches this listener. Do not publish the port publicly. A container healthcheck calls appmesh-auth.sh service-health.

  • appmesh_slave*: depends_on: appmesh_master: service_healthy, APPMESH_AUTH_ROLE=follower, the identical issuer, and an explicit APPMESH_AUTH_ACCESS_URL=http://appmesh_master:6062/auth. The packaged oidc.yaml default points at loopback. That default is wrong on a follower.

  • Clients reach the issuer in-network at http://appmesh_master:6062/auth. Host-side CLI usage publishes the master REST port. For password and device flows only, bind the Dex port to host loopback. The CLI then passes --auth-access-url. This argument is routing-only. The published endpoints still must sit on the issuer origin.

7.8.1. First sign-in in the compose cluster

The compose network runs the issuer over plain HTTP. The enrollment accepts a loopback client only. Do the first sign-in inside the master container. Use the loopback WSS listener and the packaged administrator password:

docker exec appmesh_master sh -c 'printf "%s\n" "$(grep "^password=" /opt/appmesh/work/auth/secrets/initial-admin-credentials | cut -d= -f2-)" | /opt/appmesh/bin/appm -H wss://127.0.0.1:6058 logon --username admin@appmesh.local --password-stdin --auth-allow-http'

  • --auth-allow-http is necessary. The issuer uses plain HTTP on the protected compose network.

  • The server certificate of the master covers 127.0.0.1, the container address, appmesh_master, and localhost.

  • Then copy the owner’s work/config/authorization.yaml to every follower and restart the follower daemons. See “Authorization data consistency” above. A fresh follower has no work/config directory. Create the directory before the first copy.

7.8.2. Web UI entry

The appmesh_ui service shares the network namespace of appmesh_master and serves HTTPS on port 443. Point a browser at https://<host>/. The UI proxies /appmesh/ to the master agent on 6060 and /auth/ to the master Dex listener. The UI image waits for the API upstream before nginx starts. If the upstream stays unreachable for 60 seconds, the container exits. The restart policy then attaches a new container to the live namespace of the master.

7.9. Verification

The verification runs in the appmesh-e2e environment against a make pack output (package root). Two instances share one host. Different port ranges separate the instances.

  1. The owner uses the default ports. The follower uses offset ports. Both instances come from the same package root.

  2. Check the follower process list for a Dex process. Check the authentication state directory. It must stay empty of identity files.

  3. Run the first-admin sign-in on the owner with the built-in password method: appm logon --username admin@appmesh.local. Use a loopback connection.

  4. Send a forwarded request from the owner Engine to the follower. Use X-Target-Host: 127.0.0.1:<follower-tcp-port>. Expect a successful response. Check the follower log for its own validation of the same bearer.

  5. Send the same token directly to the follower. Expect a successful response.

  6. Send a tampered token to both nodes. Expect rejection.

This procedure was verified in the appmesh-e2e environment against the packaged compose cluster.

7.10. Limitations

  • HTTP-protocol SDK clients append their own base port to X-Target-Host. The Go Agent handles this case: port 6060 becomes an HTTPS hop. The Engine gateway does not handle this case: it selects a TCP-msgpack forward to port 6060, and no listener serves that protocol on the port. WSS clients, TCP clients, and explicit :6059 targets work against the Engine gateway. This limitation is documented and is not changed here.

  • Browser flows (authorization code with PKCE) require browser access to the issuer origin. Off-network clients are limited to password and device flows through --auth-access-url rewriting.

  • Owner failover stays the manual fence-and-promote sequence in Security.md.