Operations: environment variables, admin container, and SRE runbooks --- # Admin Container > Admin container tools for checking database migration status, rotating encryption keys, and running diagnostics. Source: The admin container provides tools for key rotation, migration status checks, and other administrative tasks in production environments. > **Note**: Migrations are **auto-applied on server startup**. The admin container’s `migrate` command is primarily for checking status or running migrations separately in special cases. ## Building ```bash docker build --target admin -f docker/Dockerfile.unified -t everruns-admin . ``` ## Commands | Command | Description | | -------------- | ------------------------------- | | `migrate` | Run pending database migrations | | `migrate-info` | Show migration status | | `reencrypt` | Re-encrypt secrets with new key | | `shell` | Interactive shell for debugging | | `help` | Show usage information | ## Usage ### Check Migration Status Use this before deployments to verify migration state: ```bash docker run --rm \ -e DATABASE_URL="postgres://user:pass@host:5432/db" \ everruns-admin migrate-info ``` ### Run Migrations Manually Migrations auto-apply on server startup. Use this only for: * Running migrations without starting the server * Debugging migration issues (with `--no-migrations` on server) ```bash docker run --rm \ -e DATABASE_URL="postgres://user:pass@host:5432/db" \ everruns-admin migrate ``` ### Re-encrypt Secrets (Dry Run) ```bash docker run --rm \ -e DATABASE_URL="postgres://user:pass@host:5432/db" \ -e SECRETS_ENCRYPTION_KEY="kek-v2:..." \ -e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \ everruns-admin reencrypt --dry-run ``` ### Re-encrypt Secrets (Execute) ```bash docker run --rm \ -e DATABASE_URL="postgres://user:pass@host:5432/db" \ -e SECRETS_ENCRYPTION_KEY="kek-v2:..." \ -e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \ everruns-admin reencrypt --batch-size 50 ``` ## Environment Variables | Variable | Required | Description | | --------------------------------- | ------------- | ---------------------------- | | `DATABASE_URL` | Yes | PostgreSQL connection string | | `SECRETS_ENCRYPTION_KEY` | For reencrypt | Primary encryption key | | `SECRETS_ENCRYPTION_KEY_PREVIOUS` | For rotation | Previous encryption key | | `RUST_LOG` | No | Log level (default: info) | ## TLS/SSL Connections The admin container supports TLS connections to PostgreSQL. Use the `sslmode` parameter in your connection string: ```bash DATABASE_URL="postgres://user:pass@host:5432/db?sslmode=require" ``` ## Migration Troubleshooting ### Migration Fails on Startup If the server won’t start due to a migration error: 1. Check server logs for the specific SQL error 2. Fix the migration file 3. Rebuild and redeploy ### Bad Migration Deployed If a migration succeeded but caused issues, use **forward-fix**: ```bash # Create a new migration that fixes the problem sqlx migrate add -r fix_bad_migration # Edit the migration, then redeploy ``` ### Emergency: Manual Database Fix For emergencies where you need to manually fix the database: ```bash # Start server without auto-migrations everruns-server --no-migrations # Connect and fix manually psql -h host -U everruns -d everruns > -- Fix schema issues > DELETE FROM _sqlx_migrations WHERE version = 006; -- If needed # Restart server normally everruns-server ``` ## Production Deployment The admin container can be run as a one-off task in any container orchestration platform: * **Kubernetes**: Use a Job or run via `kubectl run` * **ECS**: Use `aws ecs run-task` with command override * **Docker Compose**: Use `docker compose run` * **Nomad**: Use a batch job --- # Environment Variables > Every Everruns environment variable: database connections, authentication, encryption, and development mode. Source: ## DEV\_MODE Enable development mode with in-memory storage. No PostgreSQL required. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | `false` | **Example:** ```bash # Start in dev mode (no database required) DEV_MODE=true ./target/debug/everruns-server # Or with 1 DEV_MODE=1 ./target/debug/everruns-server ``` **Notes:** * When enabled, uses in-memory storage instead of PostgreSQL * All data is lost when the server stops * gRPC server and worker communication are disabled * Stale task reclamation is disabled * Useful for quick local development and testing * Not suitable for production or multi-instance deployments **Limitations in dev mode:** * No persistence (data is lost on restart) * No worker support (all execution happens in-process) * No distributed tracing of worker activities * Single-instance only ## DEPLOYMENT\_GRADE Deployment environment grade. Controls which features and capabilities are available. | Property | Value | | ------------ | ------------------------------------ | | **Required** | No | | **Default** | `prod` (or `dev` if `DEV_MODE=true`) | **Valid values:** | Grade | Description | | --------- | ----------------------------------------------- | | `dev` | Development - all experimental features enabled | | `poc` | Proof of concept / demo environment | | `preview` | Preview/staging environment | | `prod` | Production - only stable features | **Example:** ```bash # Run in development mode with experimental features DEPLOYMENT_GRADE=dev ./target/debug/everruns-server # Production mode (default) DEPLOYMENT_GRADE=prod ./target/debug/everruns-server ``` **Notes:** * If not set, falls back to `DEV_MODE`: if `DEV_MODE=true`, uses `dev`; otherwise uses `prod` * Experimental capabilities (e.g., Docker Container) are only available in `dev` grade * Experimental seed agents (e.g., Python Coder) are only created in `dev` grade * Use `dev` for local development and testing experimental features * Use `prod` for production deployments ## API\_PREFIX Path prefix for REST API routes. | Property | Value | | ------------ | ------ | | **Required** | No | | **Default** | `/api` | **Example:** ```bash # Routes at /api/v1/agents API_PREFIX=/api ``` **Notes:** * `/health`, `/api-doc/openapi.json`, `/mcp`, `/.well-known/*`, `/oauth/*`, and `/cli/login-success` stay at the server root * `/mcp` is always mounted and authenticated; there is no `FEATURE_MCP_ENDPOINT` deployment variable or organization feature toggle * REST API routes including auth (`/v1/auth/*`) are mounted under this prefix * OAuth callback URLs use `AUTH_BASE_URL`, which defaults to `PUBLIC_APP_URL` plus `API_PREFIX` when unset * Override only if you need a non-`/api` REST prefix behind a reverse proxy or gateway ## PUBLIC\_APP\_URL Public browser origin for the Everruns app. In single-origin deployments, set this once and the server derives `FRONTEND_URL` and `AUTH_BASE_URL` from it. | Property | Value | | ------------ | ----------------------- | | **Required** | No | | **Default** | `http://localhost:9300` | **Example:** ```bash PUBLIC_APP_URL=https://everruns.example.com ``` **Notes:** * `FRONTEND_URL` defaults to `PUBLIC_APP_URL` * `AUTH_BASE_URL` defaults to `PUBLIC_APP_URL` plus `API_PREFIX` (for example, `https://everruns.example.com/api`) * Set `FRONTEND_URL` only when browser redirects must land on a different origin * Set `AUTH_BASE_URL` only when OAuth callbacks use a different public API base ## AUTH\_LOGIN\_ORIGIN Trusted browser origin that hosts the login page. Set this when an OSS-UI-based app delegates `/login` to a central identity origin. | Property | Value | | ------------ | --------------------------------------- | | **Required** | No | | **Default** | Not set (same-origin relative `/login`) | **Example:** ```bash AUTH_LOGIN_ORIGIN=https://id.example.com ``` **Notes:** * Supply only an HTTP(S) origin, with no credentials, path, query, or fragment * Set the same value in the server and UI runtime environments * The value is trusted deployment configuration; request/query input cannot override it * `return_to` remains a relative path and is still sanitized against open redirects * Configured absolute login redirects use full-page navigation ## CORS\_ALLOWED\_ORIGINS Comma-separated list of allowed origins for cross-origin requests. Only needed when the UI is served from a different domain than the API. | Property | Value | | ------------ | ----------------------- | | **Required** | No | | **Default** | Not set (CORS disabled) | **Example:** ```bash # Allow requests from a different frontend origin CORS_ALLOWED_ORIGINS=https://app.example.com # Multiple origins CORS_ALLOWED_ORIGINS=https://app.example.com,https://admin.example.com ``` **Notes:** * Not needed for local development (Caddy reverse proxy keeps UI and backend on one origin) * Not needed in production if using a reverse proxy on the same domain * If set, credentials are allowed (`Access-Control-Allow-Credentials: true`) * Wildcard (`*`) is not supported when using credentials ## HTTP\_ADDR Bind address for the server HTTP API. | Property | Value | | ------------ | -------------- | | **Required** | No | | **Default** | `0.0.0.0:9000` | **Example:** ```bash HTTP_ADDR=0.0.0.0:9000 ``` **Notes:** * `ADDR` is supported as a legacy alias * Container images already default to `0.0.0.0:9000`; most deployments do not need to set this ## VALKEY\_URL Connection URL for Valkey (Redis-compatible) used for distributed rate limiting across control-plane instances. | Property | Value | | ------------ | --------------------------------------------------- | | **Required** | No | | **Default** | Not set (uses per-instance in-memory rate limiting) | **Example:** ```bash # Local Valkey VALKEY_URL=redis://localhost:6379 # With authentication VALKEY_URL=redis://user:password@valkey.example.com:6379 # TLS (managed cloud service) VALKEY_URL=rediss://user:password@valkey.example.com:6380 ``` **Notes:** * When not set, rate limiting falls back to in-memory governor (per-instance, no coordination) * With N instances behind a load balancer, per-instance rate limiting allows N× the intended budget per IP, set `VALKEY_URL` for coordinated limits * Accepts `redis://`, `rediss://` (TLS), `valkey://`, `valkeys://` (TLS) schemes * Fail-open: if Valkey is unreachable, requests are allowed (availability over strictness) * Only used by control-plane (server); workers don’t need this variable * Uses sliding-window counters via Lua scripts for atomic rate limit checks ## DATABASE\_UNPOOLED\_URL Direct PostgreSQL connection URL used only for session-scoped `LISTEN/NOTIFY` listeners. | Property | Value | | ------------ | --------------------------------------------------------------------- | | **Required** | No | | **Default** | Not set (listeners reuse `DATABASE_URL` if it is a direct connection) | **Example:** ```bash # Query traffic through a pooler, listeners through a direct endpoint DATABASE_URL=postgres://app:secret@ep-foo-pooler.us-east-1.aws.neon.tech/everruns?sslmode=require DATABASE_UNPOOLED_URL=postgres://app:secret@ep-foo.us-east-1.aws.neon.tech/everruns?sslmode=require ``` **Notes:** * Use this when `DATABASE_URL` points at Neon `-pooler`, PgBouncer, or another proxy that does not preserve session-scoped `LISTEN/NOTIFY` semantics. * Listener paths include PostgreSQL-backed event wakeups, notification SSE, and PG task notification fallback when NATS is unavailable. * If `DATABASE_URL` or `DATABASE_UNPOOLED_URL` appears to point at a pooled/proxied endpoint, startup now fails fast with guidance to set a direct listener URL. * Ordinary query traffic still uses `DATABASE_URL`. ## Object Storage (S3-compatible blob backend) Optional backend that offloads workspace-file and image *content bytes* to an S3-compatible object store while keeping all metadata in PostgreSQL. Everruns remains the proxy for every read/write, no presigned URLs are handed to clients or workers. See [knowledge/runtime-resources/object-storage.md](https://github.com/everruns/everruns/blob/main/knowledge/runtime-resources/object-storage.md). | Variable | Required | Default | Description | | ------------------------------------- | --------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `STORAGE_BLOB_BACKEND` | No | `db` | `db` keeps bytes inline in PostgreSQL (current behavior); `s3` offloads to object storage. | | `STORAGE_S3_BUCKET` | When `s3` | , | Target bucket name. | | `STORAGE_S3_REGION` | No | , | Bucket region / region label. | | `STORAGE_S3_ENDPOINT` | No | , | Custom endpoint for S3-compatible stores (SeaweedFS, R2). Unset for AWS S3. | | `STORAGE_S3_ACCESS_KEY_ID` | No | , | Static access key. Omit to use the AWS credential chain (IAM role/instance). | | `STORAGE_S3_SECRET_ACCESS_KEY` | No | , | Static secret key. | | `STORAGE_S3_PREFIX` | No | (empty) | Key prefix isolating multiple deployments within one bucket. | | `STORAGE_S3_ALLOW_HTTP` | No | `false` | Allow plaintext HTTP (local/dev only, e.g. SeaweedFS over HTTP). | | `STORAGE_S3_FORCE_PATH_STYLE` | No | `true` | Use path-style requests (required by SeaweedFS; harmless on AWS S3). | | `STORAGE_BLOB_GC_INTERVAL_SECONDS` | No | `21600` (6h) | Interval between blob GC sweeps that reclaim orphaned objects. `0` disables GC. Only effective with the `s3` backend (inline `db` storage has no orphans). | | `STORAGE_BLOB_GC_GRACE_SECONDS` | No | `86400` (24h) | Safety grace period; orphaned objects younger than this are never deleted (avoids racing in-flight creates). | | `STORAGE_BLOB_GC_MAX_DELETES_PER_RUN` | No | `10000` | Per-sweep deletion cap to bound work; remaining orphans are reclaimed next sweep. | | `STORAGE_BLOB_GC_MAX_LIST_PER_RUN` | No | `100000` | Per-sweep cap on objects listed per prefix, bounding GC memory; larger buckets are reconciled across sweeps in key-order windows. | **Example (local SeaweedFS via `just seaweedfs`):** ```bash STORAGE_BLOB_BACKEND=s3 STORAGE_S3_BUCKET=everruns-dev STORAGE_S3_ENDPOINT=http://127.0.0.1:8333 STORAGE_S3_REGION=us-east-1 STORAGE_S3_ACCESS_KEY_ID=everruns STORAGE_S3_SECRET_ACCESS_KEY=everruns-secret STORAGE_S3_ALLOW_HTTP=true ``` Any S3-compatible store works (AWS S3, SeaweedFS, R2); only the endpoint and credentials differ. **Notes:** * The backend is selected per process at startup; a deployment runs entirely on `db` or `s3`. Enabling `s3` offloads newly written content; pre-existing inline content is still served transparently. * Tenant isolation is by object key (`workspaces/{workspace_id}/…`, `images/org-{org_id}/…`) plus the existing org/workspace authorization layer. ## NATS\_URL Connection URL for NATS with JetStream, used for push-based event delivery and task notifications. | Property | Value | | ------------ | ------------------------------------------------------------------------------------------- | | **Required** | No | | **Default** | Not set (uses PG NOTIFY for task notifications, in-memory broadcast for SSE event delivery) | **Example:** ```bash # Local NATS NATS_URL=nats://localhost:4222 # Cluster NATS_URL=nats://nats1:4222,nats://nats2:4222,nats://nats3:4222 # Server with `authorization { users: [...] }` NATS_URL=nats://control:s3cret@nats:4222 ``` **Notes:** * When not set, the system behaves exactly as before, all events persist to PG, SSE polls PG, task notifications use PG NOTIFY. Zero behavioral change. * When set, enables two features: * **Ephemeral event delivery**: delta events (`output.message.delta`, `reason.thinking.delta`, `tool.output.delta`, `llm.generation`) skip PostgreSQL and flow only through NATS JetStream. SSE streams subscribe to NATS instead of polling PG. * **Task notifications**: `task.available.{activity_type}` subjects replace PG NOTIFY for push-based worker notification. Lower latency (\~1ms vs \~30ms), supports multi-instance deployments. * When NATS event delivery is active, the server skips the legacy PostgreSQL event listener used only for SSE wakeups. * NATS JetStream must be enabled on the server (`--jetstream` flag) * Credentials embedded in the URL (`nats://user:password@host`) are sent as user/password auth. Reserved characters in the password (`/`, `@`, `:`, `%`) are accepted as-is, so a generated secret can be pasted unmodified; percent-encoded passwords are decoded and also work * Fail-graceful: if NATS connection fails at startup, falls back to PG NOTIFY + in-memory delivery with a warning that includes the connection error * Only used by control-plane (server); workers communicate via gRPC and don’t need NATS access * Default port: 4222 (or `PORT_PREFIX22` with `PORT_PREFIX`) * `just start-all` automatically starts NATS and exports `NATS_URL` if `nats-server` is installed ## LLM Provider API Keys LLM provider API keys (OpenAI, Anthropic, Gemini) are primarily stored encrypted in the database and managed via the Settings > Providers UI. | Property | Value | | ----------------------- | ---------------------------------------------- | | **Storage** | Database (encrypted with AES-256-GCM) | | **Configuration** | Settings > Providers UI or `/v1/providers` API | | **Supported Providers** | OpenAI, Anthropic, Google Gemini | **Required for encryption:** The `SECRETS_ENCRYPTION_KEY` environment variable must be set for the control-plane API to encrypt/decrypt API keys. Workers receive decrypted API keys via gRPC and do not need this variable. ```bash # Generate a new key python3 -c "import os, base64; print('kek-v1:' + base64.b64encode(os.urandom(32)).decode())" # Set in environment (control-plane only) SECRETS_ENCRYPTION_KEY=kek-v1:your-generated-key-here ``` ### Default API Keys (Development Convenience) For development, you can set default API keys via environment variables on the **control-plane only**. These are used as fallbacks when providers don’t have keys configured in the database. | Variable | Description | | --------------------------- | --------------------------------------------- | | `DEFAULT_OPENAI_API_KEY` | Fallback API key for OpenAI providers | | `DEFAULT_ANTHROPIC_API_KEY` | Fallback API key for Anthropic providers | | `DEFAULT_GEMINI_API_KEY` | Fallback API key for Google Gemini providers | | `DEFAULT_META_API_KEY` | Fallback API key for Meta Model API providers | **Example:** ```bash # Set in .env or environment (control-plane only) DEFAULT_OPENAI_API_KEY=sk-... DEFAULT_ANTHROPIC_API_KEY=sk-ant-... DEFAULT_GEMINI_API_KEY=AIza... DEFAULT_META_API_KEY=... ``` **Notes:** * These variables are only used by the control-plane, not workers * Workers receive API keys via gRPC from the control-plane * Database-stored keys always take priority over environment variables * These are intended for development convenience, not production use * The `just start-all` command automatically sets these from `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, and `GEMINI_API_KEY` if present * If no API key is configured for a provider, LLM calls will fail and users will see an error message in the chat: “I encountered an error while processing your request. Please try again later.” ## System Model Keys Two deployment-owned models sit outside the provider system above. Neither is selectable by an agent, neither is stored in the database, and neither is reachable from session or agent configuration — they are host services the platform uses for its own internal work. | Variable | Powers | Unset means | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | `UTILITY_OPENAI_API_KEY` | Agent Analyze/Health checks, and guardrail checks with `engine: "utility_llm"` (the default), called directly against OpenAI | Those checks are skipped; Analyze and Health are unavailable | | `UTILITY_OPENROUTER_API_KEY` | The same work, routed through OpenRouter instead. Setting it selects OpenRouter; it wins when both keys are set, and the startup log says so | The utility LLM falls back to `UTILITY_OPENAI_API_KEY` | | `UTILITY_LLM_MODEL` | The model the utility LLM calls on whichever backend was selected | Defaults to `gpt-6-luna` on OpenAI, `openai/gpt-6-luna` on OpenRouter | | `UTILITY_TYPESAFE_API_KEY` | Guardrail checks with `engine: "jev"` | Those checks are skipped with a warning and the turn proceeds | The keys are read from the process environment at startup. Missing keys **fail open**: a guardrail whose engine is not configured never blocks, so a missing key weakens policy rather than wedging traffic. Check the startup logs if a configured guardrail appears to do nothing. There is no separate provider variable: the utility LLM’s backend is whichever key you supply. An OpenRouter model id is namespaced by its upstream provider, so override `UTILITY_LLM_MODEL` with an id that backend accepts (`anthropic/claude-sonnet-4.5`, not `claude-sonnet-4.5`). ```bash # Control-plane and workers both read these. UTILITY_OPENAI_API_KEY=sk-... # ...or route the utility LLM through OpenRouter instead: UTILITY_OPENROUTER_API_KEY=sk-or-... UTILITY_LLM_MODEL=openai/gpt-6-luna UTILITY_TYPESAFE_API_KEY=ts-... ``` Agents can also be given the TypeSafe capability directly, which is a **separate** credential: a per-user connection configured in Settings > Connections, never this deployment key. See [TypeSafe](https://docs.everruns.com/integrations/typesafe/) and [Guardrails](https://docs.everruns.com/capabilities/guardrails/). ## System Email Delivery System email delivery is an internal service used by product and operational flows. It is not an agent capability, public API, or UI setting. | Variable | Required | Default | Description | | --------------------- | ------------------------------------- | ------------------------ | ---------------------------------------------------------------- | | `EMAIL_PROVIDER` | Yes, when sending email in production | unset / disabled | Email provider. Supported values: `disabled`, `resend` | | `RESEND_API_KEY` | Yes, when `EMAIL_PROVIDER=resend` | unset | Resend API key | | `RESEND_API_BASE_URL` | No | `https://api.resend.com` | Resend API base URL override for tests or controlled deployments | **Example:** ```bash EMAIL_PROVIDER=resend RESEND_API_KEY=re_... ``` **Notes:** * Set these on the control-plane process that performs system email sends. * The sender is fixed in code as `Everruns `. * The Resend account must have `everruns.com` verified and enabled for sending. ## UI API Proxy Architecture The UI makes all REST API requests (including SSE) to `/api/*` paths. The backend serves those routes under `/api` directly. Root-level backend routes like `/oauth/*`, `/mcp`, and `/.well-known/*` bypass the UI and are proxied straight to the backend. **Local Development:** * Caddy on `:9300` routes `/api/*`, `/oauth/*`, `/mcp`, and `/.well-known/*` to backend at `:9301` * Example: `/api/v1/agents` → `http://localhost:9301/api/v1/agents` * Example: `/oauth/authorize?...` → `http://localhost:9301/oauth/authorize?...` * Example: `/mcp` → `http://localhost:9301/mcp` * Example: `/.well-known/oauth-authorization-server` → `http://localhost:9301/.well-known/oauth-authorization-server` * SSE streaming works via `flush_interval -1` in Caddy config * No CORS needed (same-origin through Caddy) **Production:** * Configure your reverse proxy (nginx, Caddy, etc.) to route `/api/*`, `/oauth/*`, `/mcp`, and `/.well-known/*` to the API server * Disable response buffering for SSE endpoints * Example Caddy config: see `local/Caddyfile` ## SSE Streaming Configuration | Variable | Default | Description | | ----------------------------- | ------- | ------------------------------------------------------------------ | | `SSE_REALTIME_CYCLE_SECS` | `300` | Connection cycle interval for session event streams (seconds) | | `SSE_MONITORING_CYCLE_SECS` | `600` | Connection cycle interval for durable monitoring streams (seconds) | | `SSE_HEARTBEAT_INTERVAL_SECS` | `30` | Interval between heartbeat comments on all SSE streams (seconds) | | `SSE_GLOBAL_MAX` | `10000` | Maximum total SSE connections across all users | | `SSE_PER_SESSION_MAX` | `12` | Maximum SSE connections per session | | `SSE_PER_ORG_MAX` | `1000` | Maximum SSE connections per organization | **Notes:** * Heartbeat comments (`: heartbeat\n\n`) are sent on all SSE streams to detect stale connections * The heartbeat interval must be less than the SDK read timeout (default: 60s) with safety margin * Connection cycling prevents stale connections through proxies and load balancers * When running behind HTTP/1.1 proxies, increase `SSE_REALTIME_CYCLE_SECS` to reduce reconnection frequency ## Worker gRPC Configuration ### SERVER\_GRPC\_ADDRESS Address of the server gRPC endpoint for worker communication. | Property | Value | | ------------ | ---------------- | | **Required** | No (worker only) | | **Default** | `127.0.0.1:9001` | **Example:** ```bash SERVER_GRPC_ADDRESS=127.0.0.1:9001 ``` **Notes:** * Workers communicate with the server via gRPC for all database operations * `WORKER_GRPC_ADDRESS` is supported as a legacy alias * The server exposes both HTTP (default `9000`) and gRPC (default `9001`) interfaces * Workers are stateless and do not connect directly to the database ### WORKER\_GRPC\_AUTH\_TOKEN Bearer token for authenticating worker gRPC connections to the control-plane. | Property | Value | | ------------ | ------------------------------- | | **Required** | Yes (production); No (dev mode) | | **Default** | Unset (auth disabled) | **Example:** ```bash WORKER_GRPC_AUTH_TOKEN=your-secret-token ``` **Notes:** * Must be set on both the server and all workers (same value) * When unset, gRPC auth is disabled (acceptable for local development only) * Server panics on startup if unset when not in dev mode ### SERVER\_GRPC\_BIND\_ADDR Bind address for the server-side gRPC listener. | Property | Value | | ------------ | ---------------- | | **Required** | No (server only) | | **Default** | `0.0.0.0:9001` | **Example:** ```bash SERVER_GRPC_BIND_ADDR=0.0.0.0:9001 ``` **Notes:** * `WORKER_GRPC_ADDR` is supported as a legacy alias ### WORKER\_GRPC\_CONNECT\_TIMEOUT Timeout in seconds for worker initial connection to control-plane gRPC. | Property | Value | | ------------ | ---------------- | | **Required** | No (worker only) | | **Default** | `30` | **Example:** ```bash WORKER_GRPC_CONNECT_TIMEOUT=60 ``` ### WORKER\_GRPC\_TLS\_CERT Path to PEM-encoded certificate file. On the server, this is the gRPC server certificate. On the worker, this is the client certificate presented during mTLS handshake. | Property | Value | | ------------ | ---------------------- | | **Required** | No | | **Default** | Not set (TLS disabled) | **Example:** ```bash WORKER_GRPC_TLS_CERT=/etc/everruns/grpc-cert.pem ``` **Notes:** * Must be set together with `WORKER_GRPC_TLS_KEY` * Server: enables TLS on the gRPC listener when both cert and key are set * Worker: presents client certificate to the server when both cert and key are set (requires `WORKER_GRPC_TLS_CA_CERT`) ### WORKER\_GRPC\_TLS\_KEY Path to PEM-encoded private key file corresponding to `WORKER_GRPC_TLS_CERT`. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | Not set | **Example:** ```bash WORKER_GRPC_TLS_KEY=/etc/everruns/grpc-key.pem ``` ### WORKER\_GRPC\_TLS\_CA\_CERT Path to PEM-encoded CA certificate bundle for verifying the remote peer. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | Not set | **Example:** ```bash WORKER_GRPC_TLS_CA_CERT=/etc/everruns/grpc-ca.pem ``` **Notes:** * Server: when set, requires workers to present valid client certificates signed by this CA (mutual TLS) * Worker: when set, verifies the server’s certificate against this CA and switches to `https://` transport * For full mTLS, set on both server and worker alongside their respective cert/key pairs ### WORKER\_GRPC\_TLS\_DOMAIN Override the expected server domain name for TLS certificate verification (worker only). | Property | Value | | ------------ | ------------------------------------------- | | **Required** | No | | **Default** | Derived from `SERVER_GRPC_ADDRESS` hostname | **Example:** ```bash WORKER_GRPC_TLS_DOMAIN=control-plane.internal ``` **Notes:** * Useful when the server certificate CN/SAN differs from the connection hostname (e.g., connecting via IP but cert has a DNS name) ## OpenTelemetry Configuration Everruns supports distributed tracing via OpenTelemetry with OTLP export. Agent traces follow the OpenTelemetry [Gen-AI agent and inference conventions](https://github.com/open-telemetry/semantic-conventions-genai/tree/main/docs/gen-ai) and, on the same spans, the [OpenInference conventions](https://arize-ai.github.io/openinference/spec/semantic_conventions.html) read by Arize Phoenix, so one OTLP endpoint serves both families of backends. ### OTEL\_EXPORTER\_OTLP\_ENDPOINT OTLP endpoint for trace export (e.g., Grafana Tempo, Arize Phoenix, Datadog, or any OTLP-compatible backend). | Property | Value | | ------------ | -------------------------- | | **Required** | No | | **Default** | Not set (tracing disabled) | **Example:** ```bash # For a local OTLP collector or Phoenix OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 # For production Tempo OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.monitoring:4318 ``` **Notes:** * When set, traces are exported via OTLP over HTTP/protobuf (use the backend’s HTTP port, typically 4318, not the gRPC port 4317) * Point this at the base endpoint; the `/v1/traces` path is appended for you, and a full signal URL is used as given * Connect to any OTLP-compatible backend for trace visualization * See [OpenTelemetry](https://docs.everruns.com/observability/opentelemetry/) for the span model and attributes * Without this variable, only console logging is enabled ### OTEL\_SERVICE\_NAME Service name for traces. | Property | Value | | ------------ | --------------------------------------------------- | | **Required** | No | | **Default** | `everruns-server` (API), `everruns-worker` (Worker) | **Example:** ```bash OTEL_SERVICE_NAME=everruns-prod-api ``` ### OTEL\_SERVICE\_VERSION Service version for traces. | Property | Value | | ------------ | --------------------- | | **Required** | No | | **Default** | Cargo package version | ### OTEL\_ENVIRONMENT Deployment environment label. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | Not set | **Example:** ```bash OTEL_ENVIRONMENT=production ``` ### OTEL\_RECORD\_CONTENT Enable recording of LLM input/output content in traces. **Warning:** May contain sensitive data. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | `false` | **Example:** ```bash # Standard OTel env var (preferred) OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true # Legacy alias (also works) OTEL_RECORD_CONTENT=true ``` **Notes:** * When enabled, the chat span records `gen_ai.system_instructions`, `gen_ai.input.messages`, `gen_ai.output.messages`, and `gen_ai.tool.definitions` (plus the OpenInference `input.value`, `output.value`, and flattened `llm.input_messages.*`); tool spans record `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result`; the turn root records the input message and final answer; the thinking span records the reasoning text * Disabled by default for privacy and data size concerns * Only enable in development or when debugging specific issues ### EVERRUNS\_TRACE\_CONVENTIONS Which attribute vocabularies agent spans carry. | Property | Value | | ------------ | ---------------------- | | **Required** | No | | **Default** | `gen_ai,openinference` | **Example:** ```bash # Only the OpenTelemetry Gen-AI attributes (Tempo, Jaeger, Datadog, Langfuse) EVERRUNS_TRACE_CONVENTIONS=gen_ai # Only the OpenInference attributes (Arize Phoenix) EVERRUNS_TRACE_CONVENTIONS=openinference ``` **Notes:** * Span names, kinds, and hierarchy are the same under either vocabulary; only attributes differ * Unknown values are ignored, and an empty selection falls back to both ## Local Development with OpenTelemetry To visualize traces locally, point `OTEL_EXPORTER_OTLP_ENDPOINT` at any OTLP-compatible collector: ```bash # Set OTLP endpoint for API and Worker export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 # Start services just start-all ``` To see traces in Arize Phoenix, run Phoenix locally and point the same variable at its OTLP/HTTP port (`http://localhost:6006`); spans render as AGENT, LLM, and TOOL spans with token counts and, when content capture is on, the messages. ### Gen-AI Trace Structure Traces follow the agentic execution lifecycle with 13 event types; every span starts and ends at the timestamp of the event it records: ```plaintext invoke_agent {agent name} (root span, INTERNAL) ├── reason (LLM reasoning phase) │ └── chat {model} (LLM API call, CLIENT) │ └── thinking (extended thinking, if enabled) ├── act (tool execution phase) │ ├── execute_tool {name} │ └── execute_tool {name} ├── reason (iteration 2) │ └── chat {model} └── ... ``` ### Gen-AI Trace Attributes Spans carry the OpenTelemetry Gen-AI attributes and the OpenInference attributes side by side (see `EVERRUNS_TRACE_CONVENTIONS`). The most useful ones: | Attribute | Span Types | Description | | -------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `gen_ai.operation.name` | invoke\_agent, chat, execute\_tool | `invoke_agent`, `chat`, or `execute_tool` | | `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.agent.description` | invoke\_agent | Agent the turn runs as | | `gen_ai.conversation.id` / `session.id` | All | Session identifier | | `gen_ai.provider.name` / `llm.provider` | chat | Provider (`openai`, `anthropic`, `gcp.gemini`, `aws.bedrock`, …) | | `gen_ai.request.model`, `gen_ai.response.model` / `llm.model_name` | chat | Model name | | `gen_ai.response.id` | chat | Provider response identifier | | `gen_ai.response.finish_reasons` | chat | Why generation stopped (string array) | | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` / `llm.token_count.*` | chat, invoke\_agent | Token usage (turn root carries the cumulative total) | | `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_write.input_tokens` | chat, invoke\_agent | Prompt-cache tokens | | `gen_ai.request.temperature`, `gen_ai.request.max_tokens`, `gen_ai.request.reasoning.level`, `gen_ai.request.stream` / `llm.invocation_parameters` | chat | Request parameters | | `gen_ai.response.time_to_first_chunk` | chat | Streaming latency in seconds | | `gen_ai.conversation.compacted` | chat | Context was compacted before the call | | `llm.cost.total` / `everruns.usage.cost_usd` | chat, invoke\_agent | Cost in USD when known | | `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.type`, `gen_ai.tool.description` / `tool.name`, `tool.description` | execute\_tool | Tool identity | | `openinference.span.kind` | All | `AGENT`, `LLM`, `TOOL`, or `CHAIN` | | `error.type` | All | Low-cardinality error class on failure (error code, HTTP status, `timeout`, or `_OTHER`); the span status carries the message | | `everruns.phase` | reason, act, thinking | Phase span marker | | `everruns.turn.id`, `everruns.exec.id` | All | Everruns correlation ids | | `everruns.turn.iterations`, `everruns.turn.tool_call_count`, `everruns.turn.llm_call_count` | invoke\_agent | Turn counters | | `everruns.tool.status` | execute\_tool | `success`, `error`, `timeout`, or `cancelled` | ## Braintrust Integration Everruns supports sending turn, reasoning, tool, and session lifecycle events to [Braintrust](https://www.braintrust.dev/) for observability, evaluation, and logging. For setup instructions and configuration details, see the [Braintrust Integration Guide](https://docs.everruns.com/observability/braintrust/). | Variable | Required | Default | Description | | -------------------------------- | -------- | ------------------------------- | --------------------------------------------------------------- | | `BRAINTRUST_ENABLED` | No | enabled when API key is present | Explicit Braintrust on/off switch | | `BRAINTRUST_API_KEY` | Yes | - | API key from Braintrust settings | | `BRAINTRUST_PROJECT_NAME` | No | `My Project` | Project name for organizing traces | | `BRAINTRUST_PROJECT_ID` | No | - | Direct project UUID (skips name lookup) | | `BRAINTRUST_API_URL` | No | `https://api.braintrust.dev` | API base URL | | `BRAINTRUST_QUEUE_CAPACITY` | No | `1024` | Buffered event capacity before new exports are dropped | | `BRAINTRUST_MAX_BATCH_SIZE` | No | `50` | Max events per Braintrust insert call | | `BRAINTRUST_FLUSH_INTERVAL_MS` | No | `500` | Max delay before flushing a partial batch | | `BRAINTRUST_REQUEST_TIMEOUT_MS` | No | `10000` | Per-request timeout for Braintrust insert calls | | `BRAINTRUST_MAX_RETRIES` | No | `3` | Retries for `429`, `5xx`, and timeout/connect failures | | `BRAINTRUST_RETRY_BASE_DELAY_MS` | No | `250` | Initial retry backoff | | `BRAINTRUST_RETRY_MAX_DELAY_MS` | No | `5000` | Retry backoff cap | | `BRAINTRUST_RECORD_CONTENT` | No | `false` | Export raw turn and LLM text content | | `BRAINTRUST_RECORD_THINKING` | No | `none` | Extended thinking export mode: `none`, `summary`, `full` | | `BRAINTRUST_TOOL_ARGS_MODE` | No | `redacted` | Tool argument export mode: `full`, `redacted`, `none` | | `BRAINTRUST_TOOL_RESULTS_MODE` | No | `summary` | Tool result export mode: `full`, `summary`, `redacted`, `none` | | `BRAINTRUST_DEBUG_PAYLOADS` | No | `false` | Print full outbound Braintrust payload JSON to local debug logs | --- # Authentication Configuration Runbook > Configure authentication modes, personal access tokens, OAuth providers, JWT secrets, token lifetimes, and sign-up controls. Source: ## Overview This runbook covers configuring and managing authentication for Everruns. ## Authentication Modes ### 1. No Authentication (Development) Use for local development when authentication isn’t needed: ```bash export AUTH_MODE=none ``` All requests will be allowed with full admin access. ### 2. Admin Mode (Simple Development) Use for local development with basic access control: ```bash export AUTH_MODE=admin export AUTH_ADMIN_EMAIL=admin@example.com export AUTH_ADMIN_PASSWORD=your-secure-password export AUTH_JWT_SECRET=$(openssl rand -hex 32) ``` Only the admin user can authenticate. ### 3. Full Authentication (Production) Use for production deployments: ```bash export AUTH_MODE=full export PUBLIC_APP_URL=https://your-domain.com export AUTH_JWT_SECRET=$(openssl rand -hex 32) # Optional: Configure OAuth export AUTH_GOOGLE_CLIENT_ID=your-google-client-id export AUTH_GOOGLE_CLIENT_SECRET=your-google-client-secret export AUTH_GITHUB_CLIENT_ID=your-github-client-id export AUTH_GITHUB_CLIENT_SECRET=your-github-client-secret ``` ## Environment Variables Reference ### Core Settings | Variable | Required | Description | | ------------------- | -------------- | ---------------------------------------------------------------------------------- | | `AUTH_MODE` | No | `none`, `admin`, or `full` (default: `none`) | | `PUBLIC_APP_URL` | For OAuth | Public app origin used to derive auth callback URLs | | `AUTH_BASE_URL` | No | Override callback base URL when it differs from `PUBLIC_APP_URL` + `API_PREFIX` | | `AUTH_LOGIN_ORIGIN` | No | Trusted remote origin hosting `/login`; set identically for server and UI runtimes | | `AUTH_JWT_SECRET` | For admin/full | JWT signing secret (min 32 chars recommended) | ### Admin Mode Settings | Variable | Required | Description | | --------------------- | ---------------- | ------------------- | | `AUTH_ADMIN_EMAIL` | Yes (admin mode) | Admin user email | | `AUTH_ADMIN_PASSWORD` | Yes (admin mode) | Admin user password | ### JWT Settings | Variable | Required | Description | | --------------------------------- | -------- | ---------------------------------------------------- | | `AUTH_JWT_ACCESS_TOKEN_LIFETIME` | No | Access token lifetime in seconds (default: 900) | | `AUTH_JWT_REFRESH_TOKEN_LIFETIME` | No | Refresh token lifetime in seconds (default: 2592000) | ### Feature Toggles | Variable | Required | Description | | ----------------------- | -------- | ------------------------------------------ | | `AUTH_DISABLE_PASSWORD` | No | Set to `true` to disable password login | | `AUTH_DISABLE_SIGNUP` | No | Set to `true` to disable user registration | ### Google OAuth | Variable | Required | Description | | ----------------------------- | ---------------- | ------------------------------------- | | `AUTH_GOOGLE_CLIENT_ID` | For Google OAuth | Google OAuth client ID | | `AUTH_GOOGLE_CLIENT_SECRET` | For Google OAuth | Google OAuth client secret | | `AUTH_GOOGLE_REDIRECT_URI` | No | Custom redirect URI | | `AUTH_GOOGLE_ALLOWED_DOMAINS` | No | Comma-separated allowed email domains | ### GitHub OAuth | Variable | Required | Description | | --------------------------- | ---------------- | -------------------------- | | `AUTH_GITHUB_CLIENT_ID` | For GitHub OAuth | GitHub OAuth client ID | | `AUTH_GITHUB_CLIENT_SECRET` | For GitHub OAuth | GitHub OAuth client secret | | `AUTH_GITHUB_REDIRECT_URI` | No | Custom redirect URI | ## Common Tasks ### Generate JWT Secret ```bash # Using OpenSSL openssl rand -hex 32 # Using Python python3 -c "import secrets; print(secrets.token_hex(32))" ``` ### Verify Authentication is Working ```bash # Check auth config endpoint curl http://localhost:9300/api/v1/auth/config # Should return: # {"mode":"none","password_auth_enabled":false,"oauth_providers":[],"signup_enabled":false} # For admin mode: curl -X POST http://localhost:9300/api/v1/auth/login \ -H "Content-Type: application/json" \ -d '{"email":"admin@example.com","password":"your-password"}' ``` ### Create Personal Access Token ```bash # Login first to get access token TOKEN=$(curl -s -X POST http://localhost:9300/api/v1/auth/login \ -H "Content-Type: application/json" \ -d '{"email":"user@example.com","password":"password"}' | jq -r '.access_token') # Create personal access token curl -X POST http://localhost:9300/api/v1/auth/personal-access-tokens \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"name":"my-token"}' # Response includes the full token - save it, it's shown only once ``` ### Revoke Personal Access Token ```bash curl -X DELETE http://localhost:9300/api/v1/auth/personal-access-tokens/{token_id} \ -H "Authorization: Bearer $TOKEN" ``` ### Force Logout User Delete their refresh tokens from database: ```sql DELETE FROM refresh_tokens WHERE user_id = 'user-uuid-here'; ``` ## Troubleshooting ### ”Authentication required” when AUTH\_MODE=none * Verify `AUTH_MODE` environment variable is set correctly * Restart the server after changing environment variables ### JWT Validation Fails * Ensure `AUTH_JWT_SECRET` hasn’t changed * Check token hasn’t expired * Verify the token is for the correct environment ### OAuth Redirect Fails * Verify `AUTH_BASE_URL` matches the OAuth app configuration, or that `PUBLIC_APP_URL` derives the expected `{PUBLIC_APP_URL}/api` base * Check that redirect URI in provider matches `{AUTH_BASE_URL}/v1/auth/callback/{provider}` or `{PUBLIC_APP_URL}/api/v1/auth/callback/{provider}` * If you set `AUTH_BASE_URL`, ensure it already includes your REST API prefix (default: `/api`) * Ensure client ID and secret are correct ### Password Login Returns Unauthorized * In admin mode: check `AUTH_ADMIN_EMAIL` and `AUTH_ADMIN_PASSWORD` match * In full mode with password disabled: check `AUTH_DISABLE_PASSWORD` isn’t set * Verify user exists and password is correct ## Database Migration Authentication tables are included in the base schema and **auto-applied on server startup**. No manual migration step is required. To check migration status: ```bash # Via admin container docker run --rm -e DATABASE_URL="$DATABASE_URL" everruns-admin migrate-info ``` ## Health Check The `/health` endpoint shows current auth mode: ```bash curl http://localhost:9300/health # {"status":"ok","version":"0.2.0","auth_mode":"None"} ``` ## Agent Discovery Endpoints The server publishes two public documents so an AI agent can work out how to authenticate before it has any credentials: | Path | Contents | | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `/auth.md` | How to obtain credentials: OAuth 2.1 with dynamic client registration for the MCP endpoint, and personal access tokens for the REST API | | `/.well-known/mcp/server-card.json` | MCP Server Card (SEP-1649): server identity, transport, capabilities, and where its OAuth metadata lives | Both are generated from the running configuration rather than hardcoded, so a self-hosted deployment describes itself: * URLs come from `AUTH_BASE_URL` / `BASE_URL` (or `PUBLIC_APP_URL` plus `API_PREFIX`), the same value used for OAuth callbacks and the MCP resource binding. Set them to the public origin, not an internal container address, or the documents will advertise URLs an agent cannot reach. * Content follows `AUTH_MODE`. Under `AUTH_MODE=none` both documents state that no credentials are required instead of describing an OAuth flow that is not enforced. ### Reverse proxy configuration (required) `/auth.md` sits at the server root, so a deployment that puts the UI on `/` must route this one path to the server explicitly. Without the rule the request falls through to the UI and returns its 404, and the endpoint is unreachable even though the server serves it. The bundled proxies (`local/Caddyfile`, `infra/railway/caddy/Caddyfile`, and `examples/docker-compose-full.yaml`) already include it. For a custom proxy, add `/auth.md` wherever `/.well-known/*` is routed: ```caddyfile handle /.well-known/* { reverse_proxy server:9000 } handle /auth.md { reverse_proxy server:9000 } ``` nginx: ```nginx location = /auth.md { proxy_pass http://server:9000; } ``` Verify after deploying: ```bash curl -fsS https://your-host/auth.md | head -1 # expect "# auth.md" curl -fsS https://your-host/.well-known/mcp/server-card.json ``` A response of `text/html` rather than `text/markdown` means the UI answered and the proxy rule is missing. ## Security Best Practices 1. **Never commit secrets**: Use environment variables or secret management 2. **Rotate JWT secret**: Change `AUTH_JWT_SECRET` periodically (invalidates all tokens) 3. **Use HTTPS**: Always use HTTPS in production for OAuth callbacks 4. **Limit OAuth domains**: Use `AUTH_GOOGLE_ALLOWED_DOMAINS` to restrict access 5. **Monitor personal access token usage**: Track `last_used_at` for suspicious activity 6. **Set token expiration**: Use shorter `AUTH_JWT_ACCESS_TOKEN_LIFETIME` for higher security --- # Durable Execution Engine Setup > Run Everruns with the PostgreSQL-backed durable execution engine: database setup, migrations, and worker configuration. Source: This guide explains how to run Everruns with the custom PostgreSQL-backed durable execution engine. ## Overview The durable execution engine is a PostgreSQL-backed workflow orchestration system that provides: * Event-sourced workflows with automatic retries * Distributed task queue with backpressure support * Circuit breakers and dead letter queues * No additional infrastructure required (uses existing PostgreSQL) ## Quick Start ### 1. Prerequisites * PostgreSQL running and accessible * `DATABASE_URL` environment variable set * Migrations applied (includes durable tables) ### 2. Start API in Durable Mode ```bash # Set runner mode to durable export RUNNER_MODE=durable export DATABASE_URL="postgres://postgres:postgres@localhost/everruns" # Start the API server cargo run -p everruns-server ``` You should see: ```plaintext Using Durable execution engine runner (PostgreSQL-backed) ``` ### 3. Start Durable Worker In a separate terminal: ```bash # Workers only need gRPC address - NO DATABASE_URL required! export SERVER_GRPC_ADDRESS="127.0.0.1:9001" # Start the task worker cargo run -p everruns-worker --bin durable-worker ``` **Important:** Workers communicate with the control-plane via gRPC and do not require direct database access. This improves security and simplifies deployment. Or programmatically: ```rust use everruns_worker::{TaskWorkerConfig, WorkerAppBuilder}; #[tokio::main] async fn main() -> anyhow::Result<()> { WorkerAppBuilder::new(TaskWorkerConfig::from_env()) .run() .await } ``` ## Configuration ### Environment Variables | Variable | Description | Default | | ------------------------ | -------------------------------------------------------- | ---------------- | | `RUNNER_MODE` | Runner mode (durable only) | `durable` | | `DATABASE_URL` | PostgreSQL connection URL | Required | | `SERVER_GRPC_ADDRESS` | Server gRPC address (`WORKER_GRPC_ADDRESS` legacy alias) | `127.0.0.1:9001` | | `WORKER_GRPC_AUTH_TOKEN` | Bearer token for gRPC auth | Unset (disabled) | | `WORKER_ID` | Unique worker identifier | Auto-generated | | `MAX_CONCURRENT_TASKS` | Max tasks per worker | `1000` | ### Database Tables The durable engine uses these tables (created by migration 002\_durable\_execution): * `durable_workflow_instances` - Workflow state and metadata * `durable_workflow_events` - Event sourcing log * `durable_task_queue` - Distributed task queue * `durable_dead_letter_queue` - Failed tasks for manual inspection * `durable_workers` - Worker registration and heartbeats * `durable_signals` - Workflow signals (cancel, custom) * `durable_circuit_breaker_state` - Circuit breaker states ## Testing ### Unit Tests (No Dependencies) ```bash cargo test -p everruns-durable --lib ``` Expected: 91+ tests passing ### Integration Tests (Requires PostgreSQL) ```bash # Create test database psql -U postgres -c "CREATE DATABASE everruns_test;" # Run migrations (required for tests - server auto-migrates but tests don't start server) DATABASE_URL="postgres://postgres:postgres@localhost/everruns_test" \ sqlx migrate run --source crates/server/migrations # Run integration tests DATABASE_URL="postgres://postgres:postgres@localhost/everruns_test" \ cargo test -p everruns-durable --test postgres_integration_test -- --test-threads=1 ``` Expected: 17 tests passing > **Note**: In production, migrations are auto-applied when `everruns-server` starts. For tests, we run migrations manually since tests don’t start the server. ## Workflow Lifecycle 1. **Message Created**: User sends message via API 2. **Workflow Started**: `DurableRunner` creates workflow and enqueues `process_input` task 3. **Input Processing**: Worker claims task, processes input, enqueues `reason` task 4. **LLM Reasoning**: Worker executes LLM call, may enqueue `act` tasks for tools 5. **Completion**: Workflow marked as `completed` after final response ## Monitoring ### Check Active Workflows ```sql SELECT id, workflow_type, status, created_at FROM durable_workflow_instances WHERE status IN ('pending', 'running') ORDER BY created_at DESC; ``` ### Check Pending Tasks ```sql SELECT id, workflow_id, activity_type, status, attempt FROM durable_task_queue WHERE status = 'pending' ORDER BY created_at; ``` ### Check Dead Letter Queue ```sql SELECT id, workflow_id, activity_type, last_error, dead_at FROM durable_dead_letter_queue ORDER BY dead_at DESC; ``` ### Check Worker Status ```sql SELECT id, status, current_load, last_heartbeat_at FROM durable_workers WHERE status = 'active'; ``` ## Crash Recovery The durable execution engine provides automatic crash recovery through: ### Worker Heartbeats Workers send heartbeats every 10 seconds while executing tasks. If a worker crashes: 1. The task remains in `claimed` status with stale `heartbeat_at` 2. Control-plane background task detects stale tasks (30s threshold) 3. Stale tasks are automatically reset to `pending` status 4. Another worker can claim and retry the task ### Stale Task Reclamation The control-plane runs a background task (every 10s) that: * Finds tasks with `status = 'claimed'` and `heartbeat_at` older than 30s * Resets them to `pending` status * Logs reclaimed task IDs for monitoring ```sql -- View tasks that may need reclamation SELECT id, workflow_id, activity_type, claimed_by, heartbeat_at FROM durable_task_queue WHERE status = 'claimed' AND heartbeat_at < NOW() - INTERVAL '30 seconds'; ``` ## Troubleshooting ### Worker Not Processing Tasks 1. Check worker is running and connected to correct `SERVER_GRPC_ADDRESS` 2. Verify `activity_types` match task types in queue 3. Check worker heartbeat in `durable_workers` table ### Workflows Stuck in Running 1. Check for claimed tasks that haven’t completed 2. Look for errors in worker logs 3. Check DLQ for failed tasks 4. Wait for stale task reclamation (30s threshold) ### Task Retries Exhausted Tasks moved to DLQ after exhausting retries: ```sql -- View DLQ entries SELECT * FROM durable_dead_letter_queue ORDER BY dead_at DESC; -- Requeue a task UPDATE durable_dead_letter_queue SET requeued_at = NOW() WHERE id = ''; ``` ## Implementation Status | Phase | Status | Description | | --------- | --------------- | -------------------------------------------------------- | | Phase 1-4 | ✅ Complete | Core abstractions, persistence, reliability, worker pool | | Phase 5 | 🔄 Planned | Observability & Metrics (OpenTelemetry integration) | | Phase 6 | 🔄 Planned | Scale Testing (1000+ concurrent workers) | | Phase 7 | ✅ Core Complete | gRPC-based worker integration, crash recovery | The durable execution engine is production-ready for single-instance deployments. --- # Encryption Key Rotation > Rotate the secrets encryption key (KEK): key deployment, data re-encryption, and old key removal. Source: This runbook describes how to rotate the secrets encryption key (KEK) used to encrypt sensitive data in the database. ## Overview Everruns uses envelope encryption with versioned keys. Key rotation is a multi-phase process: 1. **Deploy new key** alongside old key 2. **Re-encrypt data** from old key to new key 3. **Remove old key** after all data is migrated ## Prerequisites * Access to secrets management (environment config or secrets manager) * Ability to run the admin container in production * Ability to deploy application updates * The `reencrypt-secrets` CLI tool (available in the admin container) ## Rotation Procedure ### Phase 1: Generate New Key Generate a new encryption key with an incremented version: ```bash # Generate new key (increment version number from current) python3 -c "import os, base64; print('kek-v2:' + base64.b64encode(os.urandom(32)).decode())" ``` Store the output securely. Example output: ```plaintext kek-v2:xR7qW2mN9pL4kJ8vB3tY6fE1hG5sD0cA9uI7oP2nM6w= ``` ### Phase 2: Deploy with Both Keys Update environment configuration: ```bash # Current key becomes the new one SECRETS_ENCRYPTION_KEY=kek-v2:xR7qW2mN9pL4kJ8vB3tY6fE1hG5sD0cA9uI7oP2nM6w= # Previous key is preserved for decryption SECRETS_ENCRYPTION_KEY_PREVIOUS=kek-v1:8B3uCQ4Znx45hl5nB+PKVriRrj/KtEVM+wBZ2VGa9vY= ``` Deploy the application with both keys configured. At this point: * **New encryptions** use `kek-v2` * **Existing data** encrypted with `kek-v1` is still decryptable ### Phase 3: Re-encrypt Existing Data Use the `reencrypt-secrets` CLI tool to migrate all data to the new key. #### Step 1: Dry Run (Preview Changes) First, run in dry-run mode to see what would be re-encrypted: ```bash docker run --rm \ -e DATABASE_URL="$DATABASE_URL" \ -e SECRETS_ENCRYPTION_KEY="kek-v2:..." \ -e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \ everruns-admin reencrypt --dry-run ``` Example output: ```plaintext 2024-01-15T10:00:00Z INFO Encryption service initialized. Primary key: kek-v2 2024-01-15T10:00:00Z INFO Available keys: ["kek-v2", "kek-v1"] 2024-01-15T10:00:00Z INFO Connected to database 2024-01-15T10:00:00Z INFO Processing table: llm_providers 2024-01-15T10:00:01Z INFO Would re-encrypt llm_providers.api_key_encrypted (id=..., current_key=kek-v1) 2024-01-15T10:00:01Z INFO DRY RUN: Would re-encrypt 42 of 100 records ``` #### Step 2: Execute Re-encryption Once satisfied with the dry run, execute the actual re-encryption: ```bash docker run --rm \ -e DATABASE_URL="$DATABASE_URL" \ -e SECRETS_ENCRYPTION_KEY="kek-v2:..." \ -e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \ everruns-admin reencrypt --batch-size 50 ``` #### CLI Options ```plaintext USAGE: reencrypt-secrets [OPTIONS] OPTIONS: -n, --dry-run Show what would be changed without making changes -b, --batch-size Process N records at a time (default: 100) -t, --table Only process specified table (default: all) -h, --help Show this help message ``` ### Phase 4: Verify Migration Confirm all data has been migrated by running another dry run: ```bash docker run --rm \ -e DATABASE_URL="$DATABASE_URL" \ -e SECRETS_ENCRYPTION_KEY="kek-v2:..." \ -e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \ everruns-admin reencrypt --dry-run ``` Expected output: ```plaintext 2024-01-15T11:00:00Z INFO DRY RUN: Would re-encrypt 0 of 100 records ``` You can also verify directly in the database: ```sql -- Check for any remaining records with old key SELECT COUNT(*) FROM llm_providers WHERE api_key_encrypted::text LIKE '%"key_id":"kek-v1"%'; -- Should return 0 ``` ### Phase 5: Remove Old Key Once verified, remove the old key from configuration: ```bash # Remove the previous key SECRETS_ENCRYPTION_KEY=kek-v2:xR7qW2mN9pL4kJ8vB3tY6fE1hG5sD0cA9uI7oP2nM6w= # SECRETS_ENCRYPTION_KEY_PREVIOUS= (remove or leave empty) ``` Deploy the updated configuration. **Important**: Keep the old key archived securely for disaster recovery. You may need it if backup restoration is required. ## Rollback Procedure If issues occur during rotation: ### During Phase 2-3 (Both Keys Active) No rollback needed - both keys work. Simply stop the re-encryption CLI if causing issues. ### After Phase 5 (Old Key Removed) If old key was removed but some data wasn’t migrated: 1. Re-add the old key as `SECRETS_ENCRYPTION_KEY_PREVIOUS` 2. Deploy 3. Run the re-encryption CLI again 4. Verify again before removing ## Monitoring During rotation, monitor: * **CLI Progress Output**: The tool logs progress every 1000 records * **API Error Rates**: Watch for decryption failures in application logs * **Database Load**: Ensure re-encryption isn’t causing performance issues ## Emergency: Compromised Key If a key is suspected compromised: 1. **Immediately** generate new key and deploy with both keys 2. Run re-encryption CLI with **highest priority**: ```bash docker run --rm \ -e DATABASE_URL="$DATABASE_URL" \ -e SECRETS_ENCRYPTION_KEY="kek-v2:..." \ -e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \ everruns-admin reencrypt ``` 3. Remove compromised key as soon as all data is migrated 4. Rotate any credentials that may have been exposed ## Key Storage Best Practices * Store keys in a secrets manager (AWS Secrets Manager, HashiCorp Vault, etc.) * Enable audit logging for key access * Rotate keys on a regular schedule (e.g., annually) * Keep previous key archived for disaster recovery (separate secure storage) * Never commit keys to source control