Deploying GUARDIAN Server to AWS ECS

This guide covers the current AWS deployment for Guardian. The AWS stack now uses Amazon RDS for PostgreSQL and no longer supports the legacy ECS-hosted Postgres runtime.

The deployment surface supports two stage profiles:

• DEPLOY_STAGE=dev keeps the current low-cost, fixed-capacity behavior
• DEPLOY_STAGE=prod enables ECS autoscaling, RDS storage autoscaling, RDS Proxy, larger default RDS sizing, RDS deletion protection with a final snapshot on destroy, and benchmark-oriented runtime defaults

Published Docker images

Prebuilt, versioned server images are published to the GitHub Container Registry (GHCR) at ghcr.io/openzeppelin/guardian, so you can pull a known-good image instead of building from source:

docker pull ghcr.io/openzeppelin/guardian:<version>   # e.g. v1.2.3, or latest

Images are multi-architecture (linux/amd64 + linux/arm64) and fully runtime-configurable — every setting and secret is supplied at run time, never baked in (see docs/CONFIGURATION.md):

docker run --rm -p 3000:3000 -p 50051:50051 \
  --env-file ./guardian.env \
  ghcr.io/openzeppelin/guardian:<version>

To run the published image with a Postgres backend locally, use the registry compose file (no local build):

cp .env.registry.example .env.registry          # then set POSTGRES_PASSWORD in .env.registry
GUARDIAN_VERSION=<version> docker compose --env-file .env.registry -f docker-compose.registry.yml up

The stack is driven entirely by the gitignored .env.registry (see .env.registry.example): Compose reads POSTGRES_PASSWORD / GUARDIAN_VERSION from it for interpolation (via --env-file), and the server container loads it for runtime config. The shared repo .env (AWS/deploy config) is intentionally not used here, so this example never mutates it. The repo's default docker-compose.yml (and the docker-compose.postgres.yml override) instead build the server from source for contributors; docker-compose.registry.yml pulls the published image.

Maintainers publish a version by running the Docker Publish GitHub Actions workflow one of two ways:

• On a GitHub Release. Publishing a release auto-triggers the workflow: the version and build ref come from the release tag, the build uses the postgres feature, and an existing tag is never overwritten. The build first waits for required-reviewer approval on the release environment before it pushes — approve releases that should ship a server image, and decline ones that should not (e.g. SDK-only releases that share the same vX.Y.Z tag line). A release tag that does not match vMAJOR.MINOR.PATCH[-prerelease] fails the workflow. Cutting the release as a draft (gh release create … --draft) does not trigger the workflow — review it first, then publish the draft to start the build.
• Manual dispatch. Pick the branch/tag/commit to build from, the version to tag, the build features, and whether to overwrite an existing tag — for off-release or one-off builds.

In both cases a version containing - (e.g. v1.2.3-rc.1) is treated as a pre-release and does not move the latest tag.

Published images are what the AWS Deploy workflow rolls out (next section). scripts/aws-deploy.sh still builds and pushes its own image for infrastructure changes and first-time stack setup.

Deploying a published image from GitHub Actions

The AWS Deploy workflow (.github/workflows/aws-deploy.yml) deploys a published GHCR version to a Guardian stack without local AWS credentials or Terraform state. GitHub environments are named after the Miden network they serve and map onto AWS stacks through environment variables, so stack names and infrastructure profiles stay explicit settings rather than being inferred from the environment name:

Environment	Network	Stack (STACK_NAME)	Profile	Hostname
devnet	MidenDevnet	guardian	dev	guardian-devnet.openzeppelin.com
testnet	MidenTestnet	guardian-prod	prod	guardian-testnet.openzeppelin.com

Run it from the Actions tab with:

• environment: devnet or testnet
• version: a published tag such as v1.2.3. Only devnet accepts pre-releases like v1.2.3-rc.1; every other environment is release-only.

What a run does:

1. Resolves the version tag to an immutable digest and verifies that digest against the SLSA provenance attestation signed by the Docker Publish workflow. For release-only environments the attestation must also show the build ran from the refs/tags/<version> release tag, so a manually dispatched build of another branch cannot ship under a release version. This happens before the environment's protection rules, so reviewers are only asked to approve an already-verified digest.
2. Authenticates to AWS with GitHub OIDC (bootstrap role, then role chaining to the deploy role) and checks the stack's ECR repository and ECS service exist.
3. Mirrors the verified digest into <stack>-server in ECR, tagged both <version> and latest. Moving latest keeps scripts/aws-deploy.sh plan / deploy --skip-build resolving the deployed image, so a later Terraform apply does not roll the service back.
4. Registers a new revision of the task definition the service is currently running, with only the image changed, and waits up to 20 minutes for the rollout to stabilize.

The run summary records the deployed image URI and the previous task-definition ARN. There is no automatic rollback: if a rollout fails, the summary prints the aws ecs update-service command to restore the previous revision, or re-run the workflow with the previously deployed version.

The workflow only rolls out an image. When a release also changes the task definition (new environment variables, secrets, or IAM grants in infra/), apply that release's Terraform first with scripts/aws-deploy.sh, then deploy the image. Stacks that override the default <stack>-server / <stack>-cluster resource names in Terraform (or ECR_REPO_NAME in the script) are not supported by the workflow. The guardian-evm stack is also out of scope: GHCR release images are built with the postgres feature only.

One-time setup per target (infra):

• A GitHub environment named after the network (devnet / testnet) with variables AWS_REGION, ROLE_FOR_OIDC (role trusted for GitHub OIDC), ROLE_TO_ASSUME (deploy role reached via role chaining), and STACK_NAME (guardian / guardian-prod). Add required reviewers on testnet.
• Each environment must restrict deployment branches to main. Without that, anyone able to dispatch the workflow could run an edited copy of it from a feature branch and obtain the environment's AWS OIDC identity.
• The IAM roles the workflow assumes (bootstrap role → deploy role, scoped to the deployable stacks) are managed by infra/oidc.tf on one stack; see the GitHub OIDC deploy roles runbook for setup and for adding environments or stacks.
• The ECR repository must already exist; scripts/aws-deploy.sh build creates it on a new stack.

Prerequisites

• Terraform >= 1.12 (older releases fail the plan on the null-default variable validations)
• AWS CLI configured with permissions for ECS, ECR, ELB, EC2, IAM, CloudWatch, RDS, and Secrets Manager
• Docker installed locally
• jq installed locally when deploying with GUARDIAN_SERVER_FEATURES=postgres,evm

aws sts get-caller-identity
docker info
terraform version

Quick Start

aws sso login --profile <your-profile>

set -a && source .env && set +a

# Optional: build/deploy ARM64 instead of X86_64
# export CPU_ARCHITECTURE=ARM64

# Miden network the server runs against. The server requires this at startup;
# the deploy script passes MidenTestnet unless you override it here. It must
# match the stack below: MidenDevnet for guardian-devnet, MidenTestnet for
# guardian-testnet.
export GUARDIAN_NETWORK_TYPE=MidenDevnet

# Optional: allow dashboard operators and let Terraform create the secret
# export GUARDIAN_OPERATOR_PUBLIC_KEYS_JSON='["0x<alice-falcon-public-key>","0x<bob-falcon-public-key>"]'

# Optional: use an existing dashboard operator public keys secret instead
# export GUARDIAN_OPERATOR_PUBLIC_KEYS_SECRET_ARN='arn:aws:secretsmanager:us-east-1:123456789012:secret:guardian/operators'

# Optional: enable EVM support from config/evm/chains.json
# export GUARDIAN_SERVER_FEATURES=postgres,evm
# export GUARDIAN_EVM_CHAIN_CONFIG_FILE=config/evm/chains.json
# export GUARDIAN_CORS_ALLOWED_ORIGINS=https://accounts.openzeppelin.com

# Optional: choose the deployment profile
export DEPLOY_STAGE=dev
# export DEPLOY_STAGE=prod

# Stack base name and canonical public hostname. SUBDOMAIN defaults to
# guardian, which no OZ stack uses: set guardian-devnet for the devnet stack
# or guardian-testnet for the testnet stack.
export STACK_NAME=guardian
export SUBDOMAIN=guardian-devnet

aws sts get-caller-identity
./scripts/aws-deploy.sh deploy
./scripts/aws-deploy.sh status

For a reviewable deployment, split image publishing, planning, and applying:

./scripts/aws-deploy.sh build
./scripts/aws-deploy.sh plan
./scripts/aws-deploy.sh deploy --skip-build
./scripts/aws-deploy.sh status

This builds and pushes ${ECR_REPO_NAME}:latest, plans Terraform against the immutable digest currently behind that tag, then applies using the existing ECR image without rebuilding. If you push a new image after plan, rerun plan before deploy --skip-build.

Terraform Variables

If you need to override defaults, use infra/terraform.tfvars:

aws_region = "us-east-1"

# Optional: ECS/image architecture
# cpu_architecture = "X86_64"
# cpu_architecture = "ARM64"

# Optional: derive resource names from a base stack name
# stack_name = "guardian"

# Only set this when bypassing scripts/aws-deploy.sh. The deploy script resolves
# ECR latest to an immutable digest and passes server_image_uri via -var.
# server_image_uri = "123456789012.dkr.ecr.us-east-1.amazonaws.com/guardian-server@sha256:<digest>"

# Optional: Postgres credentials (defaults derive from stack_name)
# postgres_db       = "guardian"
# postgres_user     = "guardian"
# postgres_password = "guardian_dev_password"

# Optional: managed database sizing overrides
# Stage defaults:
# - dev  -> db.t3.micro, 20 GiB allocated, no storage autoscaling ceiling
# - prod -> db.t3.medium, 50 GiB allocated, 200 GiB max allocated
# rds_instance_class = "db.t3.medium"
# rds_allocated_storage = 50
# rds_max_allocated_storage = 200

# Optional: Miden network for the server runtime
# server_network_type = "MidenDevnet"

# Optional: dashboard operator Falcon public keys managed by Terraform
# guardian_operator_public_keys = [
#   "0x<alice-falcon-public-key>",
#   "0x<bob-falcon-public-key>",
# ]

# Optional: existing dashboard operator Falcon public keys secret
# guardian_operator_public_keys_secret_arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:guardian/operators"

# Optional: hosted ECDSA ACK signer backed by AWS KMS.
# Setting this is all that is required: Terraform grants the ECS task role
# kms:Sign + kms:GetPublicKey on the key and injects GUARDIAN_ACK_ECDSA_BACKEND
# and GUARDIAN_ACK_ECDSA_KMS_KEY_ID into the server runtime. The key must be
# ECC_SECG_P256K1 / SIGN_VERIFY. On this path the ECDSA Secrets Manager secret is
# not needed; the Falcon ACK secret bootstrap is unchanged and still required in
# prod.
# guardian_ack_ecdsa_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/<key-id>"

# Optional: EVM runtime configuration
# guardian_evm_allowed_chain_ids = "1,11155111"
# guardian_evm_rpc_urls = "1=https://ethereum-rpc.publicnode.com,11155111=https://ethereum-sepolia-rpc.publicnode.com"
# guardian_evm_entrypoint_address = "0x433709009b8330fda32311df1c2afa402ed8d009"
# guardian_cors_allowed_origins = "https://accounts.openzeppelin.com"
# guardian_allowed_account_schemes = "ecdsa"   # new accounts only; existing Falcon accounts keep working

# Optional: stage/runtime capacity overrides
# deployment_stage = "prod"
# server_desired_count = 2
# server_autoscaling_enabled = true
# server_autoscaling_min_capacity = 2
# server_autoscaling_max_capacity = 6
# server_autoscaling_cpu_target = 65
# server_autoscaling_memory_target = 75
# rds_proxy_enabled = true
# rds_proxy_subnet_ids = ["subnet-xxxxxxxx", "subnet-yyyyyyyy"]
# In us-east-1, avoid subnets in us-east-1e/use1-az3 for RDS Proxy.
# guardian_rate_limit_enabled = false
# guardian_rate_burst_per_sec = 200
# guardian_rate_per_min = 5000
# guardian_db_pool_max_size = 32
# guardian_metadata_db_pool_max_size = 32

# Optional: application metrics (ADOT sidecar + CloudWatch dashboard/alarms).
# Enabled by default; see "Metrics, Dashboard, And Alarms".
# guardian_metrics_enabled = true   # server Prometheus endpoint (loopback-only)
# cloudwatch_metrics_enabled = true # ADOT sidecar + dashboard + alarms
# metrics_namespace = "Guardian/Server"
# alarm_actions = ["arn:aws:sns:us-east-1:123456789012:guardian-alerts"]
# alarm_notifications_enabled = true # managed SNS topic <stack>-alarms
# alarm_slack_workspace_id = "T0123456789" # Amazon Q chat: authorized workspace
# alarm_slack_channel_id = "C0123456789"   # e.g. #guardian-alerts-devnet
# alarm_error_rate_threshold_percent = 5
# alarm_latency_threshold_seconds = 1
# alarm_cpu_threshold_percent = 85
# alarm_memory_threshold_percent = 90
# cloudwatch_log_alarms_enabled = true # ERROR log metric filter + log-errors alarm, WARN filter with the dashboard (needs guardian_log_format = "json")
# alarm_log_error_threshold = 0

# Optional: Route 53 hosted zone ID
# route53_zone_id = "Z1234567890ABC"

# Optional: Cloudflare DNS management
# cloudflare_zone_id = "..."
# cloudflare_api_token = "..."

Database TLS verification

By default the server DATABASE_URL uses sslmode=require — the connection is encrypted but the RDS certificate is not verified. To authenticate the database, provide a CA bundle via Secrets Manager and set the rds_ca_bundle_secret_arn Terraform variable.

Migrating an already-deployed stack? Follow the staged, fail-safe procedure in runbooks/enable-db-tls.md (image-first, staging-before-prod, RDS-Proxy caveat, rollback). The rest of this section is the mechanism reference. The deployment then switches DATABASE_URL to sslmode=verify-full&sslrootcert=<mounted path>, and both the migration (libpq) and runtime (rustls) connections verify the certificate chain and hostname.

How delivery works (image stays CA-free). The published image ships no CA bundle. When rds_ca_bundle_secret_arn is set, Terraform adds a small rds-ca-initializer init container to the task: it reads the secret, writes it to a shared in-task volume, sets permissions, and exits; Fargate won't start the Guardian container until it succeeds (dependsOn: SUCCESS). The Guardian container mounts the same volume read-only and reads the bundle as a plain file. The app never calls Secrets Manager and nothing is baked into the image.

Combined CA bundle (required for RDS). Production routes DATABASE_URL through the RDS Proxy endpoint, which presents an AWS Certificate Manager certificate that chains to Amazon Trust Services roots (specifically Amazon Root CA 1) — not the Amazon RDS CA roots used by a direct instance. The secret MUST therefore contain both root sets so verify-full succeeds against either endpoint.

Size limit — do NOT use the global RDS bundle. Secrets Manager caps a secret value at 64 KiB. The global-bundle.pem (~165 KB) exceeds that and create-secret will reject it. Use your region-specific RDS bundle (a few KB) plus just Amazon Root CA 1, which keeps the combined bundle well under the cap. The Rust loader already supports multiple roots in one PEM.

Build it (mind the newline between files so the PEM blocks don't merge) and store it verbatim — plain PEM text, no encoding:

# Region-specific RDS bundle (replace us-east-1), ~4-5 KB
curl -sS https://truststore.pki.rds.amazonaws.com/us-east-1/us-east-1-bundle.pem -o rds.pem
# All Amazon Trust Services roots (~6 KB total) — more rotation-tolerant than
# pinning only the one the RDS Proxy chains to today; still well under 64 KiB.
: > ats.pem
for ca in AmazonRootCA1 AmazonRootCA2 AmazonRootCA3 AmazonRootCA4 SFSRootCAG2; do
  curl -sS "https://www.amazontrust.com/repository/${ca}.pem" >> ats.pem; echo >> ats.pem
done
{ cat rds.pem; echo; cat ats.pem; } > rds-combined-ca.pem
grep -c "BEGIN CERTIFICATE" rds-combined-ca.pem   # sanity: total root count
test "$(wc -c < rds-combined-ca.pem)" -lt 65536 || echo "WARNING: bundle exceeds the 64 KiB Secrets Manager limit"

aws secretsmanager create-secret \
  --name guardian-prod/server/rds-ca-bundle \
  --secret-string file://rds-combined-ca.pem

Why Secrets Manager for a public cert? CA roots aren't confidential, so this is a consistency choice — it reuses the same secret-injection plumbing and IAM pattern as DATABASE_URL and the ACK keys, with no new mechanism. The trade-offs are the 64 KiB cap above and ~$0.40/secret/month. If a future bundle must exceed 64 KiB, the next step is an S3 object (no size cap) or an EFS access point fetched by the same init container — the Rust/app side would not change.

Then set the ARN and apply:

rds_ca_bundle_secret_arn = "arn:aws:secretsmanager:REGION:ACCOUNT:secret:guardian-prod/server/rds-ca-bundle-XXXXXX"

The execution role is granted secretsmanager:GetSecretValue on that ARN automatically. If the bundle is missing or malformed, the init container or the server preflight fails closed at startup rather than connecting insecurely.

Rotation. Update the secret's value (it may hold old and new roots together for overlap), then force a new deployment so the init container re-reads the secret and rewrites the file. Because the task definition still points at the same secret ARN, changing only the secret value does not roll tasks on its own — force it explicitly:

aws ecs update-service --cluster <cluster> --service <service> --force-new-deployment

No image or code change is required; ensure the new roots are present before they become the only trusted ones.

Deploy

Script Commands

Command	Purpose
./scripts/aws-deploy.sh build	Build the Guardian server image and push it to ECR as latest. Does not run Terraform.
./scripts/aws-deploy.sh plan	Run terraform plan using the immutable digest currently behind ECR latest. Does not build, push, or apply.
./scripts/aws-deploy.sh deploy	Build and push the image, resolve ECR latest to an immutable digest, and run terraform apply.
./scripts/aws-deploy.sh deploy --skip-build	Resolve the existing ECR latest image to an immutable digest and run terraform apply without rebuilding.
./scripts/aws-deploy.sh bootstrap-ack-keys	Create the prod ACK key secrets in Secrets Manager. Refuses to overwrite existing secrets. With TF_VAR_guardian_ack_ecdsa_kms_key_arn set, creates only the Falcon secret (ECDSA is KMS-backed).
./scripts/aws-deploy.sh bootstrap-kms-ecdsa-key	Create the KMS ECDSA ACK signing key (ECC_SECG_P256K1 / SIGN_VERIFY) and an alias/${STACK_NAME}-ack-ecdsa alias, then print the ARN to set. Refuses to overwrite an existing alias.
./scripts/aws-deploy.sh bootstrap-dashboard-cursor-secret	Create the shared 32-byte dashboard cursor secret in Secrets Manager. Refuses to overwrite an existing secret.
./scripts/aws-deploy.sh status	Print Terraform outputs for the active STACK_NAME and DEPLOY_STAGE.
./scripts/aws-deploy.sh logs	Tail the deployed server's CloudWatch log group.
./scripts/aws-deploy.sh cleanup	Run Terraform destroy for the active STACK_NAME and DEPLOY_STAGE.

--skip-build is meaningful for deploy; plan never builds or pushes an image. Use build before plan for a new stack or whenever ECR does not yet contain ${ECR_REPO_NAME}:latest.

One-Step Deploy

./scripts/aws-deploy.sh deploy

The deploy script resolves the ECR latest tag to an immutable digest before calling Terraform, so image pushes always produce a real ECS task-definition revision instead of relying on tag reuse. It also keeps separate local Terraform state files per STACK_NAME and DEPLOY_STAGE, using infra/terraform.<stack>.<stage>.tfstate by default.

AWS deployments must include the postgres server feature. The script defaults GUARDIAN_SERVER_FEATURES to postgres; set GUARDIAN_SERVER_FEATURES=postgres,evm only when deploying the optional EVM API surface.

Reviewable Build, Plan, Apply

Use this flow when you want to inspect Terraform changes before applying them:

./scripts/aws-deploy.sh build
./scripts/aws-deploy.sh plan
./scripts/aws-deploy.sh deploy --skip-build

build creates the ECR repository if needed and pushes ${ECR_REPO_NAME}:latest. Both plan and deploy --skip-build resolve that tag to an immutable digest before invoking Terraform. Do not rebuild or push a new latest between plan and deploy --skip-build unless you intend to apply a different image; rerun plan after any rebuild.

For DEPLOY_STAGE=prod, bootstrap the ACK and dashboard cursor secrets once before the first deploy:

DEPLOY_STAGE=prod ./scripts/aws-deploy.sh bootstrap-ack-keys
DEPLOY_STAGE=prod ./scripts/aws-deploy.sh bootstrap-dashboard-cursor-secret

The normal deploy path does not create or rotate these secrets. It expects the prod Secrets Manager entries to already exist. Terraform injects the cursor secret into every ECS task as GUARDIAN_DASHBOARD_CURSOR_SECRET.

Secret names default to ${STACK_NAME}/server/ack-{falcon,ecdsa}-secret-key, so distinct stacks (e.g. guardian-prod, guardian-prod-eu) automatically resolve to distinct secrets and multiple Guardian deployments can coexist in the same AWS account. Override per stack by setting GUARDIAN_ACK_FALCON_SECRET_NAME / GUARDIAN_ACK_ECDSA_SECRET_NAME before bootstrap-ack-keys and deploy; they flow into Terraform variables and the ECS task definition's GUARDIAN_ACK_FALCON_SECRET_ID / GUARDIAN_ACK_ECDSA_SECRET_ID env vars.

The cursor secret defaults to ${STACK_NAME}/server/dashboard-cursor-secret. To use an existing secret, set GUARDIAN_DASHBOARD_CURSOR_SECRET_NAME before both bootstrap and deploy. The deploy helper always passes this resolved name explicitly, so a stale infra/terraform.tfvars value cannot make validation and deployment select different secrets. For a customer-managed KMS key, its key policy must also allow the ECS task execution role to decrypt the secret.

Prod with a KMS-backed ECDSA signer

To keep the ECDSA private key in AWS KMS (never resident in the server process) while Falcon stays in Secrets Manager, create the key first and export its ARN before the rest of the flow. The script keys off TF_VAR_guardian_ack_ecdsa_kms_key_arn (the env var, not terraform.tfvars) to skip the ECDSA Secrets Manager secret, so it must be exported before bootstrap-ack-keys and deploy:

export DEPLOY_STAGE=prod STACK_NAME=<stack>

./scripts/aws-deploy.sh bootstrap-kms-ecdsa-key                  # creates the key, prints the ARN
export TF_VAR_guardian_ack_ecdsa_kms_key_arn="arn:aws:kms:...:key/<key-id>"
./scripts/aws-deploy.sh bootstrap-ack-keys                      # Falcon only; skips ECDSA
./scripts/aws-deploy.sh deploy

Terraform then grants the ECS task role kms:Sign + kms:GetPublicKey and injects GUARDIAN_ACK_ECDSA_BACKEND=aws-kms / GUARDIAN_ACK_ECDSA_KMS_KEY_ID. See runbooks/secrets.md for key lifecycle, the immutable-spec caveat, and migrating an existing deployment (a new keypair, so a SwitchGuardian identity change).

Dashboard operator public keys use a separate optional secret. The easiest deployment path is to pass the public keys to Terraform and let it create the stack-scoped secret:

export GUARDIAN_OPERATOR_PUBLIC_KEYS_JSON='["0x<alice-falcon-public-key>","0x<bob-falcon-public-key>"]'

or in terraform.tfvars:

guardian_operator_public_keys = [
  "0x<alice-falcon-public-key>",
  "0x<bob-falcon-public-key>"
]

If you already manage the secret outside this stack, pass its ARN through GUARDIAN_OPERATOR_PUBLIC_KEYS_SECRET_ARN or guardian_operator_public_keys_secret_arn. An explicit secret ARN takes precedence over the Terraform-managed public key list.

The ECS task role is granted read access only to the configured secret ARN. The server rereads that secret during operator auth checks, so adding or removing a key in the existing secret takes effect without an application restart. When Terraform manages the secret, update the public key list and rerun deploy.

EVM deployments need the evm server feature plus server-owned chain config. By default, scripts/aws-deploy.sh derives allowed chain IDs, RPC URLs, and the shared EntryPoint address from config/evm/chains.json. It passes RPC URLs to Terraform as a stack-scoped Secrets Manager secret and the EntryPoint address as a normal ECS environment variable. To use an alternate JSON file, set GUARDIAN_EVM_CHAIN_CONFIG_FILE.

You can still override the derived values by setting GUARDIAN_EVM_ALLOWED_CHAIN_IDS, GUARDIAN_EVM_RPC_URLS, or GUARDIAN_EVM_ENTRYPOINT_ADDRESS directly, or by passing existing secret ARNs through GUARDIAN_EVM_ALLOWED_CHAIN_IDS_SECRET_ARN and GUARDIAN_EVM_RPC_URLS_SECRET_ARN.

When an EVM UI runs on a different origin, set GUARDIAN_CORS_ALLOWED_ORIGINS to a comma-separated list of exact origins. Wildcard origins are rejected. When this value is configured, the server enables credentialed CORS so browsers can include the host-only, HttpOnly guardian_evm_session cookie.

If you still have an older local state file at infra/terraform.tfstate, move it manually before using the split-state workflow:

cp infra/terraform.tfstate infra/terraform.guardian.dev.tfstate
cp infra/terraform.tfstate.backup infra/terraform.guardian.dev.tfstate.backup 2>/dev/null || true

Use --skip-build when the image already exists in ECR and you only need infra/runtime changes, or when you are applying immediately after a reviewed plan:

./scripts/aws-deploy.sh deploy --skip-build

For benchmark-oriented production deploys, prefer explicit overrides rather than changing the base prod profile in code. A typical starting point is:

set -a && source .env && set +a

export DEPLOY_STAGE=prod
export STACK_NAME=guardian-prod
export TF_VAR_server_cpu=2048
export TF_VAR_server_memory=4096
export TF_VAR_server_desired_count=3
export TF_VAR_server_autoscaling_min_capacity=3
export TF_VAR_server_autoscaling_max_capacity=10
export TF_VAR_rds_instance_class=db.r6g.large
export TF_VAR_rds_allocated_storage=100
export TF_VAR_rds_max_allocated_storage=400
export TF_VAR_rds_proxy_subnet_ids='["subnet-25c1722b","subnet-4d0eca6c"]'
export TF_VAR_guardian_db_pool_max_size=64
export TF_VAR_guardian_metadata_db_pool_max_size=64
export TF_VAR_guardian_rate_limit_enabled=false

./scripts/aws-deploy.sh deploy --skip-build

Validate

./scripts/aws-deploy.sh status
curl https://guardian-testnet.openzeppelin.com/pubkey
grpcurl -import-path crates/server/proto -proto guardian.proto -d '{}' guardian-testnet.openzeppelin.com:443 guardian.Guardian/GetPubkey

Metrics, Dashboard, And Alarms

Application metrics ship to CloudWatch by default. Two switches control this: guardian_metrics_enabled turns on the server's Prometheus endpoint, and cloudwatch_metrics_enabled deploys the ADOT sidecar, EMF log group, IAM policy, dashboard, and metric-based alarms on top of it. The export pipeline cascades off with the endpoint, so guardian_metrics_enabled = false alone turns all of that off (the log-level alarm is gated separately). Disabling only cloudwatch_metrics_enabled keeps the endpoint without publishing CloudWatch custom metrics — but note the endpoint stays loopback-only, so that mode is useful only for an alternative in-task collector you add by customizing the module; the stack exposes no knobs for a routable bind address.

• The server runs with GUARDIAN_METRICS_ENABLED=true, serving the Prometheus exposition on 127.0.0.1:9464/metrics. Fargate awsvpc containers share one network namespace, so the endpoint is loopback-only: the sidecar reaches it on 127.0.0.1 while nothing outside the task can — it is never exposed via the ALB, target groups, or security groups. An externally scraped setup would additionally require an explicit bind address, restricted security-group ingress, and GUARDIAN_METRICS_BEARER_TOKEN; this module deliberately configures none of that.
• Latency and other duration metrics are windowed: the collector delta-converts the Prometheus histograms (the awsemf exporter does not do this for histograms on its own), so CloudWatch Average over a 5-minute period reflects that period, not the process lifetime.
• An AWS Distro for OpenTelemetry (ADOT) Collector sidecar in the server task scrapes the endpoint every 60s and exports a curated selection of metrics to CloudWatch as EMF log events (log group /ecs/<service>/emf). CloudWatch materializes them as custom metrics under the metrics_namespace namespace, which is per stack: Guardian/Server for the default stack name, Guardian-Prod/Server for guardian-prod, and so on, so stacks in one account never mix metrics. The collector config is injected via AOT_CONFIG_CONTENT (infra/observability.tf); no custom image or SSM parameter is involved. Dimension sets come from Guardian's closed label sets (status, code, outcome, kind, event, pool, transport); high-cardinality labels (route, method, operation) are rolled up to keep the custom-metric count and cost bounded. No scrape bearer token is configured on this path: network isolation (the loopback-only listener inside the task) is the first defense layer described in the observability guide, and no other network path to the endpoint exists.
• The sidecar is non-essential (its exit does not stop the task) and its memory is capped at 256 MiB, which bounds its worst-case share of the shared task envelope — it is a bound, not isolation: both containers still draw from the task's server_memory total, so size that with ~256 MiB of headroom in mind. If the collector leaks it is OOM-killed at its cap, the server keeps serving, and the metrics-missing alarm fires on the fleet going dark. With more than one task, a single dead sidecar is not detected by that alarm — the remaining tasks keep the metrics alive and the fleet's Sums/Averages skew until the next deployment; per-task detection is a possible follow-up.
• Terraform creates a CloudWatch dashboard named <stack>-server (request volume, error rate, latency, proposal/delta lifecycle, canonicalization health, storage and DB-pool health, Miden RPC, ECS CPU/memory/tasks) and these alarms:

Alarm	Fires when
<stack>-http-5xx-rate	HTTP 5xx responses (500/501/502/503/504) exceed alarm_error_rate_threshold_percent (default 5%) of requests for 15 min. ALB health checks count as successful requests and dilute the rate on low-traffic multi-task fleets — treat as a sustained-fault signal
<stack>-grpc-error-rate	gRPC server-fault responses (internal, unavailable, unknown, data_loss, deadline_exceeded) exceed the same threshold for 15 min; the same health-check dilution applies
<stack>-http-latency	Average HTTP latency exceeds alarm_latency_threshold_seconds (default 1s) for 15 min. Fleet average across all routes — continuous ALB health-check probes dilute it on low-traffic stacks, so treat it as a sustained-degradation signal
<stack>-canonicalization-failures	Canonicalization passes (full, fast, or reconcile) report error or partial (some accounts failed) outcomes for 10 min
<stack>-metrics-missing	Application metrics stop arriving — the constant guardian_build_info heartbeat disappears (metrics endpoint down, sidecar dead, or scrape failing)
<stack>-metrics-refresh-failures	Slow-aggregate refresher attempts are failing; delta/proposal/account gauges are stale
<stack>-metrics-refresh-stale	The refresh timestamp stopped advancing for ≥ 10 min (hung or dead refresher — catches what the failures counter cannot)
<stack>-ecs-cpu-high / <stack>-ecs-memory-high	ECS service average CPU/memory exceeds alarm_cpu_threshold_percent (85%) / alarm_memory_threshold_percent (90%); must sit above the autoscaling targets (enforced at plan time)
<stack>-server-log-errors	More than alarm_log_error_threshold (default 0) ERROR-level server log lines per 5-minute period for two consecutive periods. Absolute count from a log metric filter, so it catches low-volume faults the rate alarms cannot; independent of the metrics pipeline. See Log-level alarms

Every alarm description ends with the stack name and console links to the <stack>-server dashboard and the server log group (streams ecs/* for the server, adot/* for the sidecar), so a notification alone is enough to start debugging.

Alarm notifications

Alarms fire both alarm_actions and ok_actions, so every target sees the recovery too. There are two ways to receive them, and they compose:

• Bring your own pipeline: list existing same-region SNS topic ARNs in alarm_actions. Nothing else is provisioned.

  alarm_actions = ["arn:aws:sns:us-east-1:123456789012:guardian-alerts"]

• Managed topic + Slack channel (infra/alerting.tf, opt-in): with alarm_notifications_enabled = true Terraform creates the SNS topic <stack>-alarms in the stack's region with a policy scoped to this account (its CloudWatch alarms via a confused-deputy condition, plus the account's own principals), and appends it to every alarm's actions (after any alarm_actions ARNs; CloudWatch caps actions at five per state, enforced at plan time). Adding alarm_slack_workspace_id and alarm_slack_channel_id also creates an Amazon Q Developer in chat applications (formerly AWS Chatbot) channel configuration <stack>-alarms-slack subscribed to that topic, with a notifications-only IAM role (<stack>-chatbot-alarms, CloudWatch read so the message carries the metric graph) capped by the CloudWatchReadOnlyAccess guardrail. Use one Slack channel per environment (for example #guardian-alerts-devnet for the guardian stack and #guardian-alerts-testnet for guardian-prod) so the channel itself carries the environment context; the alarm name, description, and links carry the stack. The Slack IDs must be set together and require alarm_notifications_enabled; the plan fails otherwise. The topic and the Slack configuration cascade off with cloudwatch_metrics_enabled, like the alarms they serve, so the metrics kill switch still needs only one variable. The topic is not KMS-encrypted: CloudWatch cannot publish to a topic encrypted with the AWS-managed SNS key, and alarm payloads contain only alarm metadata.

  The topic is plain SNS, so operators can additionally subscribe email, PagerDuty, or anything else to it out of band (alarm_sns_topic_arn output). Such subscriptions live outside Terraform state: anything that destroys the topic (disabling the flag or the metrics pipeline, aws-deploy.sh cleanup) drops them silently, and re-enabling recreates the topic without them. With the flag alone and nothing subscribed, the alarms publish into an empty topic, so add the Slack IDs (or a subscription) in the same apply.

  The Terraform resource needs AWS provider 5.61 or newer (versions.tf now requires ~> 5.61); a checkout initialised earlier must run terraform -chdir=infra init -upgrade once, otherwise plan stops on an inconsistent lock file.

  Migrating from bring-your-own: if this stack already notifies a hand-made topic with a console-created Amazon Q channel configuration, either import them or delete the console configuration and drop the old ARN from alarm_actions before applying. Leaving both in place posts every ALARM/OK twice in the channel. Do not rely on a name collision to stop you: SNS CreateTopic is idempotent, so an existing topic already named <stack>-alarms is silently adopted on apply and its access policy replaced (or the apply fails on mismatched attributes). Import it explicitly and review the plan before applying:

  terraform -chdir=infra import 'aws_sns_topic.alarms[0]' arn:aws:sns:<region>:<account>:<stack>-alarms
  terraform -chdir=infra import 'aws_chatbot_slack_channel_configuration.alarms[0]' \
    arn:aws:chatbot::<account>:chat-configuration/slack-channel/<configuration-name>
  terraform -chdir=infra import 'aws_iam_role.chatbot_alarms[0]' <channel-role-name>
  terraform -chdir=infra plan

  A topic or configuration with a different name is not preserved by importing alone: the configured names are fixed, so the plan proposes replacing the imported resource. Rename the Slack channel configuration (its name cannot be edited in place) or accept the replacement, and move any out-of-band subscriptions to the new topic afterwards.

Slack setup (once per AWS account and workspace)

Terraform cannot run the Slack OAuth flow, so authorize the workspace in the console first:

1. In the AWS console open Amazon Q Developer in chat applications (https://console.aws.amazon.com/chatbot/), choose Slack under Configure a chat client, pick the workspace, and Allow (a Slack workspace admin may need to approve the app). The API (and the Terraform resource) has no us-east-1 endpoint; the AWS provider routes the calls to a supported region itself, so nothing changes in aws_region.
2. Copy the Workspace ID (T...) from the workspace details page into alarm_slack_workspace_id. It is the same for every stack in the account.
3. Create the environment channel (e.g. #guardian-alerts-devnet), run /invite @Amazon Q in it (required for private channels, harmless for public ones), and copy its Channel ID (C..., the last path segment of Copy link) into alarm_slack_channel_id. Do not use channel names.
4. Set alarm_notifications_enabled = true together with the two IDs and apply. Repeat 3-4 per stack with that stack's own channel.

Amazon Q posts the alarm name, description (with the stack, dashboard, and log links), state change and reason, a link to the alarm, and the metric graph for both ALARM and OK transitions. When a notification never arrives, check the alarm's action history first (a denied CloudWatch → SNS publish shows there and never reaches Amazon Q), then the Amazon Q error log /aws/chatbot/<stack>-alarms-slack in us-east-1 (a service-managed log group outside Terraform state, created on the first error, with no retention policy and negligible volume); the other usual causes are the app not being invited to the channel, or a workspace or channel ID typed as a name.

Set guardian_metrics_enabled = false to turn everything off (no metrics env vars, no sidecar, no dashboard, no metric-based alarms — the CloudWatch flag cascades off with it), or only cloudwatch_metrics_enabled = false to keep the loopback-only endpoint without any CloudWatch export (see the caveat above about what that mode is useful for). The log-level alarm below is independent of both flags and stays on.

Log-level alarms

Independently of the metrics pipeline, CloudWatch Logs metric filters on the server log group (infra/log_alarms.tf) count the server's own log lines by level and publish them as custom metrics under <metrics_namespace>/Logs (the log_metrics_namespace output; kept apart from the scraped metrics so "metrics arriving in metrics_namespace" stays a pipeline health check): log_error_events (lines with level = "ERROR") and, when the dashboard is deployed, log_warn_events (level = "WARN"). They read the container's log output directly, so they need neither the metrics endpoint nor the ADOT sidecar; the ERROR filter and its alarm keep working with guardian_metrics_enabled = false, and in that mode they are the only alarm left. They cover logged faults, not process liveness: a task that panics or crash-loops at startup prints plain-text panic output, not JSON, which the filters never see (the metrics-missing and ECS alarms cover that when the pipeline is on). The alarm notifies the same effective action list as every other alarm (operator alarm_actions plus the managed SNS topic); note that the managed topic and its Slack channel cascade off with the metrics pipeline, so with guardian_metrics_enabled = false it reaches only the ARNs you pass in alarm_actions.

• The filters match the JSON level field the server emits with guardian_log_format = "json" (the default). Terraform rejects the plan if cloudwatch_log_alarms_enabled is true with a text or compact format; set cloudwatch_log_alarms_enabled = false to run those formats.
• <stack>-server-log-errors alarms on log_error_events: more than alarm_log_error_threshold (default 0) ERROR lines in each of two consecutive 5-minute periods. A persistent fault, however slow, pages within 10 minutes; one isolated line does not; a burst confined to a single period does not page on its own either (it shows on the dashboard and, if large enough, through the rate alarms). It overlaps with the rate alarms on purpose: on a low-traffic stack a few failures never move a percentage that ALB health checks dominate, but every one is an ERROR line.
• What counts as ERROR: the centralized HTTP 5xx / gRPC-internal log line, background-job and canonicalization failures, and some client-caused rejections the server logs at ERROR before mapping them to 4xx (rejected signatures or unauthorized cosigner keys in metadata/auth, invalid credentials in configure_account). A persistently misconfigured client retrying every minute therefore trips the alarm at the default threshold; if that is expected on a stack, raise alarm_log_error_threshold so a known trickle is tolerated while a burst still pages. Tightening the server's log levels is the longer-term fix.
• WARN is dashboard-only (the Server log lines by level widget) and its filter is created only alongside the dashboard, since nothing else reads it. No alarm: the server logs WARN for client-caused and self-healing conditions (retried RPC, rate limiting), so one would page for normal operation.
• A metric filter applies to the whole log group, which also carries the adot and ca-init streams. The ADOT Collector writes console-encoded lines (not JSON, lowercase level), which a JSON pattern never matches — collector faults surface through <stack>-metrics-missing — and the one-shot CA initializer prints nothing on success.
• To see what fired, query the log group in CloudWatch Logs Insights:

fields @timestamp, message, code, detail, target, span.account_id
| filter level = "ERROR"
| sort @timestamp desc
| limit 50

Verify metrics after a deploy

# Every name below is per stack; read them all from Terraform outputs.
# Steps 1-4 need the metrics pipeline (their outputs are empty with
# cloudwatch_metrics_enabled = false); step 5 is independent of it.
NS=$(terraform -chdir=infra output -raw metrics_namespace)
DASH=$(terraform -chdir=infra output -raw metrics_dashboard_name)
LOG_GROUP=$(terraform -chdir=infra output -raw server_log_group)
ALARM=$(terraform -chdir=infra output -raw metrics_missing_alarm_name)
LOG_ALARM=$(terraform -chdir=infra output -raw server_log_errors_alarm_name)
LOG_NS=$(terraform -chdir=infra output -raw log_metrics_namespace)

# 1. Metrics arriving in the namespace (allow ~2 minutes after task start)
aws cloudwatch list-metrics --namespace "$NS" --output table | head -40

# 2. Dashboard exists and populates
aws cloudwatch get-dashboard --dashboard-name "$DASH" --query DashboardName
# then open CloudWatch > Dashboards > <stack>-server in the console

# 3. Sidecar logs — scrape failures surface here as
#    "Failed to scrape Prometheus endpoint"
aws logs tail "$LOG_GROUP" --log-stream-name-prefix adot --since 15m

# 4. Exercise the alarm notification path end to end: the ALARM message
#    should appear in this stack's Slack channel (or reach every ARN in the
#    alarm_actions output) within a minute, followed by the OK message when
#    the next evaluation returns the alarm to OK automatically.
terraform -chdir=infra output alarm_actions
aws cloudwatch set-alarm-state --alarm-name "$ALARM" \
  --state-value ALARM --state-reason "notification path test"

# Nothing in Slack? Confirm the topic is on the alarm and that the publish
# succeeded (a denied publish is recorded here and never reaches Amazon Q):
aws cloudwatch describe-alarms --alarm-names "$ALARM" \
  --query 'MetricAlarms[0].[AlarmActions,OKActions]'
aws cloudwatch describe-alarm-history --alarm-name "$ALARM" \
  --history-item-type Action --max-items 4 --query 'AlarmHistoryItems[].HistorySummary'
# Then the Amazon Q error log (exists only once Amazon Q has logged an error):
SLACK_CONFIG=$(terraform -chdir=infra output -raw alarm_slack_configuration_name)
[ -n "$SLACK_CONFIG" ] && aws logs tail "/aws/chatbot/$SLACK_CONFIG" \
  --region us-east-1 --since 15m

# 5. Log metric filters are attached and counting (skipped when
#    cloudwatch_log_alarms_enabled = false: LOG_ALARM is empty then).
#    default_value = "0" publishes zeros only while log lines are being
#    ingested without matching ERROR events; a healthy but quiet service
#    (ALB health checks log nothing at the default filter) ingests no
#    lines, so an empty Datapoints list is expected and is handled by
#    treat_missing_data = "notBreaching", not a sign the filter is missing.
if [ -n "$LOG_ALARM" ]; then
  aws logs describe-metric-filters --log-group-name "$LOG_GROUP" \
    --query 'metricFilters[].{name:filterName,pattern:filterPattern}'
  aws cloudwatch get-metric-statistics --namespace "$LOG_NS" \
    --metric-name log_error_events --statistics Sum --period 300 \
    --start-time "$(( $(date +%s) - 1800 ))" --end-time "$(date +%s)"
  aws cloudwatch set-alarm-state --alarm-name "$LOG_ALARM" \
    --state-value ALARM --state-reason "notification path test"
  # returns to OK on the next evaluation (real zeros or missing data)
fi

Run step 4 on each environment's stack: a guardian (devnet) transition must land only in the devnet channel and a guardian-prod (testnet) transition only in the testnet channel.

Operations

Logs

./scripts/aws-deploy.sh logs

Status

./scripts/aws-deploy.sh status

The script reads the state file for the active STACK_NAME and DEPLOY_STAGE. The current default path is:

infra/terraform.<stack>.<stage>.tfstate

You can override that with TF_STATE_PATH if needed.

Destroy

./scripts/aws-deploy.sh cleanup

In the prod stage the RDS instance has deletion protection on and takes a final snapshot (<stack>-postgres-final) on destroy, so a prod cleanup fails until you set TF_VAR_rds_deletion_protection=false and re-apply. Restoring from the final snapshot is covered in runbooks/backup-restore.md.

ECR repositories are not managed by Terraform:

aws ecr delete-repository --repository-name guardian-server --force --region us-east-1

Resources Created

Resource	Description
ECS Cluster	Fargate cluster derived from stack_name
ECS Service	Guardian server service
Application Load Balancer	Internet-facing ALB derived from stack_name
Target Groups	HTTP target group for port 3000 and gRPC target group for port 50051
RDS	Managed PostgreSQL instance and subnet group
RDS Proxy	Managed PostgreSQL proxy in the production profile
Secrets Manager	Secret containing DATABASE_URL for the server task
Secrets Manager	Optional operator public keys secret for dashboard auth
Secrets Manager	Optional EVM allowed chain IDs and RPC URLs secrets
Secrets Manager	Secrets containing the Falcon and ECDSA ack private keys used to seed the server keystore in prod
Security Groups	ALB, server, and database security groups
CloudWatch Log Groups	Cluster execute-command logs, server logs, and the EMF metrics log group
CloudWatch Log Metric Filters	ERROR (and, with the dashboard, WARN) line counts from the server log group, published as custom metrics
IAM Role	ECS task execution and runtime roles
ADOT Sidecar	OpenTelemetry Collector container in the server task exporting Prometheus metrics to CloudWatch
CloudWatch Dashboard	<stack>-server application and ECS overview
CloudWatch Alarms	Error rate, latency, canonicalization, metrics pipeline, ECS saturation, and server log-error alarms
SNS Topic	Optional <stack>-alarms topic every alarm notifies on ALARM/OK (alarm_notifications_enabled)
Amazon Q chat configuration	Optional Slack channel configuration <stack>-alarms-slack subscribed to the alarm topic, plus its notifications-only IAM role

Outputs

Output	Description
alb_dns_name	ALB DNS name
alb_url	Full ALB URL
custom_domain_url	Canonical service URL: https with a certificate, http when Terraform manages only the DNS record
grpc_endpoint	Public gRPC endpoint when HTTPS is enabled
database_endpoint	RDS endpoint used by the server
rds_proxy_endpoint	RDS Proxy endpoint when enabled
rds_instance_class	Effective RDS instance class
rds_allocated_storage	Effective allocated RDS storage in GiB
database_url_secret_arn	Secrets Manager ARN for the server DATABASE_URL
operator_public_keys_secret_arn	Secrets Manager ARN used for dashboard operator public keys
operator_public_keys_secret_name	Terraform-managed operator public keys secret name, when created
guardian_evm_allowed_chain_ids_secret_arn	Secrets Manager ARN used for EVM allowed chain IDs
guardian_evm_rpc_urls_secret_arn	Secrets Manager ARN used for EVM RPC URLs
guardian_evm_entrypoint_address	Shared EVM EntryPoint address configured for the server
guardian_cors_allowed_origins	Explicit CORS origins configured for the server
guardian_allowed_account_schemes	Signature schemes new accounts may register with (GUARDIAN_ALLOWED_ACCOUNT_SCHEMES); empty keeps every scheme
ack_falcon_secret_name	Secrets Manager name for the Falcon ack key
ack_ecdsa_secret_name	Secrets Manager name for the ECDSA ack key
dashboard_cursor_secret_name	Secrets Manager name for the shared dashboard cursor key
ecs_cluster_arn	ECS cluster ARN
server_service_arn	Server ECS service ARN
metrics_namespace	CloudWatch namespace receiving Guardian application metrics
metrics_dashboard_name	CloudWatch dashboard name
metrics_emf_log_group	Log group the ADOT sidecar writes EMF metric events into
metrics_missing_alarm_name	Name of the metrics-pipeline heartbeat alarm for this stack
alarm_actions	Effective ARNs notified on alarm/ok transitions (operator alarm_actions plus the managed topic)
alarm_sns_topic_arn	ARN of the managed alarm SNS topic, empty when not enabled
alarm_slack_configuration_name	Name of the Amazon Q Slack channel configuration (error log group /aws/chatbot/<name>, us-east-1), empty when not configured
alarm_slack_configuration_arn	ARN of the Amazon Q Slack channel configuration, empty when not configured
cloudwatch_log_alarms_enabled	Whether the server log group's ERROR metric filter and the log-errors alarm are deployed
log_metrics_namespace	CloudWatch namespace receiving the log-level metric-filter counts (<metrics_namespace>/Logs)
server_log_errors_alarm_name	Name of the alarm on ERROR-level server log lines for this stack

Stage Profiles

Dev

• single ECS task
• no ECS autoscaling
• direct ECS to RDS connection
• no RDS Proxy
• conservative Guardian runtime limits

Prod

• higher ECS desired count
• ECS service autoscaling
• larger default RDS instance class and base storage
• RDS storage autoscaling
• RDS Proxy between ECS and RDS
• higher Guardian runtime rate-limit and DB-pool defaults for benchmark traffic

Horizontal scaling (multiple replicas)

The prod profile runs 2–6 tasks behind the ALB. Because it sets GUARDIAN_ENV=prod and the Postgres backend, the server runs shared coordination (sessions, login challenges, and the canonicalization lease live in Postgres) — so any request lands on any replica and canonicalization runs on exactly one replica at a time. Terraform also sets GUARDIAN_MAX_REPLICAS from effective_guardian_max_replicas (the greater of desired count and autoscaling max, 6 by default) so global HTTP and dashboard commitment rate limits are partitioned across the steady-state fleet. A rolling deployment may allow up to server_deployment_maximum_percent / 100 times the configured aggregate limit (2× by default). The default dashboard share is 5 requests per minute on a keep-alive-pinned replica. In prod, Terraform requires the pre-created dashboard cursor secret and injects the same value into every task, so dashboard pagination works across replicas. The server itself still warns and uses an ephemeral key when run without the variable outside this managed prod profile. Watch the per-replica GUARDIAN_DB_POOL_MAX_SIZE against Postgres max_connections (RDS Proxy absorbs most of this). Full operator guidance: runbooks/horizontal-scaling.md.

HTTPS And gRPC

HTTPS is enabled when acm_certificate_arn is set. DNS can be managed through Cloudflare, Route 53, or both depending on which variables are provided.

When HTTPS is enabled, the ALB routes standard HTTPS requests to the server HTTP port 3000 and gRPC requests for /guardian.Guardian/* to the server gRPC port 50051. The public gRPC endpoint uses the same hostname on port 443.

On Apple Silicon hosts, CPU_ARCHITECTURE=X86_64 builds are slower because Docker builds linux/amd64 images under emulation. Switching to ARM64 avoids that local emulation cost, but it also changes the ECS task runtime architecture.

Migrating An Existing ECS-Postgres Stack

The current Terraform configuration is RDS-only. There is no supported dual-mode deployment that keeps the old ECS Postgres service alive after apply.

Use this cutover flow for an existing stack:

1. Capture the current stack state:

   ./scripts/aws-deploy.sh status

2. Create a logical PostgreSQL backup from the existing ECS-hosted database before applying the updated stack.
3. Apply the updated RDS-backed Terraform stack:

   ./scripts/aws-deploy.sh deploy --skip-build

4. Restore the backup into the new RDS database.
5. Validate the public service:

   ./scripts/aws-deploy.sh status
   curl https://<host>/pubkey
   grpcurl -import-path crates/server/proto -proto guardian.proto -d '{}' <host>:443 guardian.Guardian/GetPubkey

6. Confirm the old Postgres ECS service and Cloud Map database-discovery resources are gone from AWS before treating the cutover as complete.

Troubleshooting

• If the server task fails during startup, check ./scripts/aws-deploy.sh logs first and confirm the reported database_endpoint matches the expected RDS host.
• If a prod deploy fails before Terraform starts, confirm the fixed prod ACK secrets exist by running ./scripts/aws-deploy.sh bootstrap-ack-keys once and then retrying the deploy.
• If RDS subnet-group creation fails, verify the selected subnets cover at least two subnets for the database deployment.
• If gRPC works against the ALB directly but fails on the public hostname, check Cloudflare gRPC settings on the zone.
• If the metrics-missing alarm fires or the dashboard is empty, tail the sidecar stream (aws logs tail /ecs/<service> --log-stream-name-prefix adot) — scrape failures appear as Failed to scrape Prometheus endpoint, and export failures reference awsemf.

Legacy Script

The legacy deployment logic has been replaced by the Terraform-backed scripts/aws-deploy.sh.