This is the full developer documentation for Everruns
---
# Everruns Documentation
> Tutorials, how-to guides, reference, and explanation for building and operating Everruns agents.
Source: {
vec![AgentBlueprint {
id: "repo_scout",
name: "Repo Scout",
description: "Search repositories for code, files, and issues. \
Read-only agent for codebase exploration.",
model: BlueprintModel::Fixed("claude-haiku-4-5-20251001".to_string()),
system_prompt: REPO_SCOUT_PROMPT,
tools: vec![Box::new(SearchCodeTool), Box::new(ReadFileTool)],
max_turns: Some(15),
config_schema: Some(json_schema_for::()),
}]
}
```
The `description` is what a parent agent reads when deciding whether to delegate, so write it as a routing decision: what the blueprint is for, and when to pick it.
`model` chooses one of three strategies. `Fixed` pins a model the host cannot override, which suits specialist work with a known cost/quality target. `Default` names a model the validated config may override. `Inherit` takes the parent’s model, for work that genuinely needs the parent’s characteristics.
Tools listed here are **private**. They are instantiated only for the blueprint’s own child session and never appear in the host agent’s tool list.
## Define configuration as a type
Derive the config schema from a Rust struct rather than writing JSON by hand. The struct is the single source of truth: field set, bounds, defaults, and descriptions all reach the spawning agent from one place.
```rust
use everruns_capability::json_schema_for;
use everruns_capability::schemars::JsonSchema;
use serde::{Deserialize, Serialize};
/// The same ceiling the search tools apply to their own arguments.
const MAX_REPOS: u32 = 50;
/// Configuration for the repository scout.
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields, default)]
#[schemars(crate = "everruns_capability::schemars")]
pub struct RepoScoutConfig {
/// Maximum number of repositories to scan.
#[schemars(range(min = 1, max = MAX_REPOS))]
pub max_repos: u32,
}
impl Default for RepoScoutConfig {
fn default() -> Self {
Self { max_repos: 10 }
}
}
```
Three habits keep the contract honest:
* **Express each bound once.** Write it as a constant shared by the schema attribute and the code that enforces it at runtime, so a schema edit cannot drift from the clamp it describes.
* **Write doc comments for the caller.** They become the schema descriptions the spawning agent reads. Implementation notes belong in ordinary `//` comments.
* **Close the object** with `deny_unknown_fields` unless the blueprint genuinely accepts open configuration.
Requiring configuration is a matter of leaving a field without a default: fields serde cannot default become `required` in the derived schema, and the spawn path rejects a call that omits them.
## What the spawn path enforces
Configuration is validated against the derived schema **before** a child session exists, so a declared bound constrains the host rather than merely advising the model. A spawn is rejected when:
* a value violates the schema — out of range, wrong type, missing a required property;
* the config carries a key the schema does not define (with `deny_unknown_fields`);
* the blueprint declares no schema at all but config was supplied.
The error returned to the calling agent names the violations and includes the schema, so a model can usually correct its own call and retry.
Validation governs what a **host** may configure. It is not a substitute for a tool checking its own arguments: keep the runtime clamps in the blueprint’s tools, so a misbehaving child agent stays inside the same envelope.
Configuration can only select behavior the blueprint intentionally exposes. It cannot replace the system prompt, inject tools, bypass model policy, or expand capability permissions.
## Delegation and lifetime
A spawned blueprint session is a real durable session. It takes part in the same message, event, task, workspace, cancellation, and recovery infrastructure as any other sub-agent; the difference is only how its runtime is assembled. The session persists the blueprint identity and its validated config, so a worker can reconstruct the same runtime after a retry or handoff.
The blueprint itself stays a stateless template. Follow-ups address the durable session, not the blueprint.
## Worked examples
Two blueprints ship in-tree and are worth reading as references:
* [`integrations/github/src/lib.rs`](https://github.com/everruns/everruns/blob/main/integrations/github/src/lib.rs) — the minimal case: one config field, a pattern constant shared with the runtime check that enforces it.
* [`integrations/openrouter/src/model_scout.rs`](https://github.com/everruns/everruns/blob/main/integrations/openrouter/src/model_scout.rs) — the fuller case: numeric bounds tied to runtime constants, a nested config type, and a spend budget.
## Related
* [Sub-agents](https://docs.everruns.com/capabilities/sub-agents/) — the delegation tool blueprints are invoked through
* [GitHub Scout](https://docs.everruns.com/capabilities/github-scout/) — a shipped blueprint from the caller’s side
---
# Budgets
> Cap session spending in dollars, tokens, or custom credits, with soft pause thresholds and automatic enforcement.
Source: { ... }
// L180: fn validate(output: &Output) -> bool { ... }
// L220: struct Config { ... }
// L235: impl Config { ... }
// L236: fn load(path: &str) -> Result { ... }
// L270: fn merge(&mut self, other: &Config) { ... }
// L310: #[cfg(test)] mod tests { ... }
```
### Supported Languages
| Language | Extensions |
| ---------- | ----------------------------- |
| Rust | `.rs` |
| TypeScript | `.ts`, `.tsx`, `.mts` |
| JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` |
| Python | `.py` |
### What Gets Outlined
* **Functions/methods** — name, parameters, return type
* **Structs/classes/enums** — name
* **Impl blocks / trait implementations** — with nested methods
* **Modules** — including `#[cfg(test)]` detection
* **Interfaces and type aliases** (TypeScript)
For unsupported languages, no outline is generated — the response still works, just without structural context for unread portions.
---
# Request Signing
> How Everruns signs outbound HTTP requests with Ed25519 signatures per RFC 9421, enabling target servers to verify bot identity
Source: /.well-known/http-message-signatures-directory
```
This returns a [JSON Web Key Set (JWKS)](https://www.rfc-editor.org/rfc/rfc7517#section-5) containing the bot’s Ed25519 public keys. The target server uses the `kid` field to match the key against the `keyid` in the incoming signature.
## How It Works in Everruns
Request signing is implemented as a server-wide feature. When enabled, **every outbound HTTP request** made by the `web_fetch` tool is signed.
### Signing (outbound)
The signing pipeline is handled by [fetchkit](https://github.com/everruns/fetchkit), the library powering the `web_fetch` capability:
1. Agent calls `web_fetch` with a URL
2. fetchkit builds the HTTP request
3. If bot-auth is configured, fetchkit signs the request:
* Covers `@authority` and optionally `signature-agent`
* Generates a random nonce
* Sets `created` and `expires` timestamps
* Signs with Ed25519, attaches `Signature` and `Signature-Input` headers
4. If signing fails (clock error, etc.), the request proceeds unsigned with a warning logged, signing never blocks requests
5. The request is sent to the target server
### Key directory (inbound)
Everruns serves the public key at `/.well-known/http-message-signatures-directory`. This endpoint:
* Is **public** (no authentication required)
* Returns a JWKS containing the server’s Ed25519 public key
* Derives the key at startup from the same seed used for signing
Example response:
```json
{
"keys": [
{
"kty": "OKP",
"crv": "Ed25519",
"x": "base64url-encoded-public-key",
"kid": "JWK-thumbprint-matching-keyid-in-signatures"
}
]
}
```
### What target servers see
A signed request arrives with three additional headers:
| Header | Purpose |
| ----------------- | --------------------------------------------------------------------------- |
| `Signature` | The Ed25519 signature over the covered components |
| `Signature-Input` | Describes what was signed: components, timestamps, key ID, algorithm, nonce |
| `Signature-Agent` | FQDN where the bot’s public keys can be discovered (optional) |
Target servers that support web-bot-auth can:
1. Extract the `keyid` from `Signature-Input`
2. Fetch the public key from the `Signature-Agent` FQDN’s well-known endpoint
3. Verify the signature
4. Apply access policies based on the verified identity
Servers that don’t support it simply ignore the extra headers.
## Verifying Signatures (Server Side)
If you operate a server that receives requests from Everruns agents, here’s how to verify them.
### Verification steps
1. **Check the tag**: parse `Signature-Input` and confirm `tag="web-bot-auth"`. Ignore signatures with other tags.
2. **Check timestamps**: reject if `created` is in the future or `expires` is in the past. A 5-minute clock skew tolerance is reasonable.
3. **Fetch the public key**: extract the `Signature-Agent` FQDN and fetch `https:///.well-known/http-message-signatures-directory`. Find the key matching the `keyid` from `Signature-Input`. Cache the JWKS (keys rotate infrequently).
4. **Reconstruct the signature base**: build the canonical representation per [RFC 9421 Section 2.5](https://www.rfc-editor.org/rfc/rfc9421#section-2.5) using the covered components listed in `Signature-Input`.
5. **Verify**: use Ed25519 to verify the signature against the reconstructed base and the fetched public key.
### Python example
```python
import base64
import hashlib
import time
import httpx
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
def verify_bot_auth(request) -> bool:
"""Verify a web-bot-auth signature on an incoming request."""
# 1. Parse Signature-Input header
sig_input = request.headers.get("signature-input", "")
if 'tag="web-bot-auth"' not in sig_input:
return False # not a bot-auth signature
# Extract parameters from sig_input
# sig=("@authority" "signature-agent");created=...;expires=...;keyid="...";...
params = parse_signature_params(sig_input)
# 2. Check timestamps
now = int(time.time())
if params["created"] > now + 300 or params["expires"] < now:
return False # expired or future-dated
# 3. Fetch public key from Signature-Agent FQDN
agent_fqdn = request.headers.get("signature-agent", "")
jwks_url = f"https://{agent_fqdn}/.well-known/http-message-signatures-directory"
jwks = httpx.get(jwks_url).json()
key_data = next(k for k in jwks["keys"] if k.get("kid") == params["keyid"])
public_key_bytes = base64.urlsafe_b64decode(key_data["x"] + "==")
public_key = Ed25519PublicKey.from_public_bytes(public_key_bytes)
# 4. Reconstruct signature base (RFC 9421 Section 2.5)
# Covered components are listed in parentheses in Signature-Input
sig_base = build_signature_base(request, params)
# 5. Verify
signature = base64.b64decode(
request.headers["signature"].split(":")[1] # sig=:base64:
)
try:
public_key.verify(signature, sig_base.encode())
return True
except Exception:
return False
```
### Node.js example
```javascript
import { createPublicKey, verify } from "node:crypto";
async function verifyBotAuth(request) {
const sigInput = request.headers["signature-input"] || "";
if (!sigInput.includes('tag="web-bot-auth"')) return false;
const params = parseSignatureParams(sigInput);
// Check timestamps (5-minute tolerance)
const now = Math.floor(Date.now() / 1000);
if (params.created > now + 300 || params.expires < now) return false;
// Fetch public key
const fqdn = request.headers["signature-agent"];
const res = await fetch(
`https://${fqdn}/.well-known/http-message-signatures-directory`
);
const jwks = await res.json();
const jwk = jwks.keys.find((k) => k.kid === params.keyid);
const key = createPublicKey({ key: jwk, format: "jwk" });
// Reconstruct signature base and verify
const sigBase = buildSignatureBase(request, params);
const signature = Buffer.from(
request.headers["signature"].split(":")[1],
"base64"
);
return verify(null, Buffer.from(sigBase), key, signature);
}
```
> **Note:** The `parseSignatureParams` and `buildSignatureBase` helpers follow the structured fields parsing rules from [RFC 8941](https://www.rfc-editor.org/rfc/rfc8941) and the signature base construction from [RFC 9421 Section 2.5](https://www.rfc-editor.org/rfc/rfc9421#section-2.5). Libraries like [httpbis-message-signatures](https://pypi.org/project/httpbis-message-signatures/) (Python) and [@httpbis/message-signatures](https://www.npmjs.com/package/@httpbis/message-signatures) (Node.js) handle both.
## Configuration
Request signing is configured via environment variables. Set them before starting the server.
### Environment variables
| Variable | Required | Default | Description |
| --------------------------- | -------- | ------- | -------------------------------------- |
| `BOT_AUTH_SIGNING_KEY_SEED` | yes | , | Base64url-encoded 32-byte Ed25519 seed |
| `BOT_AUTH_AGENT_FQDN` | no | , | FQDN for the `Signature-Agent` header |
| `BOT_AUTH_VALIDITY_SECS` | no | `300` | Signature validity window in seconds |
When `BOT_AUTH_SIGNING_KEY_SEED` is not set, signing is disabled and no crypto code runs at request time.
### Generate a signing key
```bash
python3 -c "import os, base64; print(base64.urlsafe_b64encode(os.urandom(32)).rstrip(b'=').decode())"
```
### Enable signing
```bash
export BOT_AUTH_SIGNING_KEY_SEED="your-base64url-seed-here"
export BOT_AUTH_AGENT_FQDN="bot.yourcompany.com"
```
Then start the server. All `web_fetch` requests will be signed, and the public key will be available at `https://bot.yourcompany.com/.well-known/http-message-signatures-directory`.
### Verify it’s working
```bash
# Check the key directory endpoint
curl -s http://localhost:9301/.well-known/http-message-signatures-directory | jq .
```
## Standards
| Standard | Role |
| ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [RFC 9421, HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421) | Core signing mechanism, how to sign and verify HTTP requests |
| [RFC 8941, Structured Field Values](https://www.rfc-editor.org/rfc/rfc8941) | Encoding format for `Signature` and `Signature-Input` headers |
| [RFC 7638, JWK Thumbprint](https://www.rfc-editor.org/rfc/rfc7638) | How the `keyid` is computed from the public key |
| [RFC 7517, JSON Web Key (JWK)](https://www.rfc-editor.org/rfc/rfc7517) | Format of the keys in the JWKS response |
| [RFC 8037, Ed25519 in JOSE](https://www.rfc-editor.org/rfc/rfc8037) | Ed25519 key representation in JWK format |
| [draft-meunier-web-bot-auth-architecture](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture) | Bot-specific profile of RFC 9421 (algorithm, tag, covered components) |
| [draft-meunier-http-message-signatures-directory](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory) | Well-known endpoint for public key discovery |
## See Also
* [fetchkit](https://github.com/everruns/fetchkit), The library implementing the signing client
---
# Tool Output Pipeline
> How Everruns processes tool results from execution to the model, verbosity budgets, distillation, persistence, hard limits, and where the full output lives
Source: /outputs/
│ • Persist Output (exec tools → VFS) {tool_call_id}.stdout
│ • Output Hard Limit (64 KiB ceiling) {tool_call_id}.stderr
▼ ▲
inline result → stored in the session, │
shown to the model next turn ──────────────────────┘
(read_file recovers the full original)
```
Two ideas run through the whole pipeline:
1. **Storage stays lossless.** Whatever the model sees inline, the full output is written to the session filesystem (the *destination*). The inline view always carries a pointer back to it.
2. **Each stage shrinks, none deletes.** Truncation, distillation, and masking only change the *view*. The agent can always `read_file` the persisted original.
## Stage 1, Verbosity budget (exec tools)
Exec and sandbox tools (`bash`, `*_exec`, sandboxed shells) clean their output (strip ANSI, collapse carriage returns) and apply a **verbosity budget** before returning. The mode is configurable per call; the default is `auto`:
* **Success (`exit_code == 0`)** → collapse to a compact summary (\~512 bytes), because the full log is persisted (Stage 3) and the agent rarely needs it inline.
* **Failure (non-zero exit)** → keep a larger diagnostic window (\~8 KiB) so the error stays debuggable in-loop.
The full pre-truncation output is stashed on the result as `raw_output` for the persistence hook to consume. Non-exec tools (MCP, web fetch, client tools) do **not** have a verbosity budget, that gap is what Stage 2 exists for.
## Stage 2, Tool Output Distillation (non-exec tools)
[Tool Output Distillation](https://docs.everruns.com/capabilities/) targets the tools Stage 1 doesn’t: **MCP tools and `web_fetch`**, whose results otherwise enter history verbatim. It runs as a capability hook, so it executes *before* the final hooks.
For a large non-exec result, distillation produces a compact, **content-aware** inline view:
| Output shape | What you get inline |
| ---------------- | ---------------------------------------------------------------------------- |
| Large JSON array | Schema-preserving sample: the first few rows + `[… N more items elided …]` |
| Long string | Head + tail window (both ends preserved), with a byte-elision marker |
| Unified diff | A diffstat-style summary: file + hunk headers and `+added / -removed` counts |
| Nested object | Each oversized field distilled; small fields untouched |
Before it replaces anything, distillation **persists the full original** to the session filesystem (same destination as Stage 3) and injects a recovery pointer. If persistence fails, or the session has no filesystem, it restores the verbatim output rather than leave a lossy result the agent can’t recover. **Reversibility is never sacrificed.**
Distillation is on by default in the **generic harness**. Every transform is deterministic, so identical output distills identically and the model provider’s prompt cache keeps hitting across turns.
## Stage 3, Persistence and the hard limit
Two infrastructure hooks always run last, in order:
1. **Persist Output**: for tools that declare the `persist_output` hint (exec/sandbox), writes the full `raw_output` to the session VFS. When content is absent from the inline result, it adds a recovery pointer; complete inline results keep the retained file internal and do not invite a redundant read. It **skips** if a result already carries `output_files` (e.g. distillation already persisted it), so the two never double-write.
2. **Output Hard Limit**: a final, unremovable 64 KiB ceiling. By the time it runs, the result has usually already been budgeted or distilled, so it rarely fires; it’s a backstop against pathological cases.
## The destination, where full output lives
Everything the pipeline elides is recoverable from the **session filesystem**:
```plaintext
/outputs/{tool_call_id}.stdout ← full standard output
/outputs/{tool_call_id}.stderr ← full standard error (when present)
```
When the inline result omits persisted content, it carries the recovery pointer in `output_files` and `full_output`, plus a human-readable note telling the model to use `read_file` (with `offset`/`limit`) for the missing detail. Complete inline output has no model-facing pointer. Persisted streams are capped at 1 MiB each. Deleting the session cascades and removes them.
This is the key to aggressive shrinking: because the original is one `read_file` away, the inline view can be small without the agent losing the ability to drill in.
## How this relates to compaction
The pipeline above operates on **individual tool results at capture time**. [Context Compaction](https://docs.everruns.com/advanced/compaction/) operates **later**, across the whole conversation, when it approaches the context window, masking or summarizing older messages at serialization time.
They compose cleanly:
* The pipeline keeps each result lean as it’s produced.
* Compaction further masks older results when the *accumulated* history grows too large.
* [Infinity Context](https://docs.everruns.com/capabilities/) adds a `query_history` tool to retrieve older *messages* that scrolled out of the window.
Together: the pipeline controls per-result size, compaction controls total-history size, and both keep the full record recoverable.
## Summary
| Stage | Applies to | Effect | Destination of full output |
| ----------------- | -------------------------- | -------------------------------------------------------- | --------------------------------------------- |
| Verbosity budget | exec/sandbox | Compact summary on success, diagnostic window on failure | `raw_output` → persisted in Stage 3 |
| Distillation | MCP / web fetch / non-exec | Content-aware compact view | `/outputs/{id}.stdout` |
| Persist Output | `persist_output` tools | Lossless write + pointer | `/outputs/{id}.{stdout,stderr}` |
| Output Hard Limit | all | 64 KiB ceiling backstop | (already persisted) |
The agent always sees a lean view and can always recover the full original with `read_file`.
---
# Built-ins Overview
> Built-in harness types and capabilities that ship with Everruns. Harnesses define session environments; capabilities add tools and behaviors.
Source: ",
"capabilities": ["web_fetch"]
}'
```
The agent’s own capabilities are added on top of the empty harness. In this example, only `web_fetch` would be available.
## See Also
* [Generic Harness](https://docs.everruns.com/built-ins/harnesses/generic/), recommended default with core capabilities
* [Harnesses feature guide](https://docs.everruns.com/features/harnesses/), harness selection and API management
---
# Data Analyst Harness
> Data analysis harness with SQL databases, persistent memory, interactive charts, and a structured analysis pipeline inspired by OpenAI's Dash.
Source: ` XML tags, broadest scope first, behind a trust framing header
4. Injected as the leading user-role message — model-visible, re-resolved every turn, below system instructions in precedence
## Config
```json
{
"files": ["AGENTS.md", "CLAUDE.md"]
}
```
`files` is optional. When omitted, Everruns reads only `/workspace/AGENTS.md`.
## Notes
* Default file name: `AGENTS.md` (plain Markdown, max 32 KiB per file, 128 KiB total per turn)
* Hierarchy: root to working directory; deeper files win, siblings out of scope
* Re-resolved every turn, edits take effect immediately
* Missing configured files are ignored (no error)
* Works with [File System](https://docs.everruns.com/capabilities/file-system/) tools to update instructions dynamically
## See Also
* [AGENTS.md feature guide](https://docs.everruns.com/features/agent-instructions/), detailed documentation
* [File System](https://docs.everruns.com/capabilities/file-system/), manage the AGENTS.md file
* [Agent Skills](https://docs.everruns.com/capabilities/agent-skills/), another way to inject specialized instructions
* [Capabilities Overview](https://docs.everruns.com/capabilities/)
---
# Agent Skills
> Discover and activate portable skill packages from the session workspace at runtime.
Source: --help`, and many also support ` --version`.
* Common builtins: `cd`, `ls`, `cat`, `echo`, `grep`, `head`, `tail`, `sed`, `find`, plus shell features like pipes, redirections, and command substitution. `grep` is backed by the session’s indexed search.
## See Also
* [Bashkit project](https://bashkit.sh), the interpreter powering this capability
* [File System](https://docs.everruns.com/capabilities/file-system/), file operations on the same workspace
* [Sub Agents](https://docs.everruns.com/capabilities/sub-agents/), background and parallel execution
* [Capabilities Overview](https://docs.everruns.com/capabilities/)
---
# Browserless
> Headless browser automation through Browserless for screenshots, DOM reading, scraping, and page interaction.
Source: ` code (e.g. `guardrail.regex`, `guardrail.llm_judge`, `guardrail.moderation`). Clients localize copy from the code rather than the human-readable text. (The separate [Prompt Canary Guardrail](https://docs.everruns.com/capabilities/prompt-canary-guardrail/) capability uses its own `system_prompt_leak` code.)
## Endpoints
| Method | Path | Description |
| ------ | -------------------------------------- | -------------------------------------------------------- |
| `GET` | `/v1/capabilities/guardrails/examples` | List adoptable gallery presets with trust metadata |
| `POST` | `/v1/capabilities/guardrails/dry-run` | Evaluate a config against sample text; nothing persisted |
Both are gated by the same `capability.view` policy as other capability reads.
## Related
* [Prompt Canary Guardrail](https://docs.everruns.com/capabilities/prompt-canary-guardrail/), a narrow streaming-output guardrail for naive system-prompt leakage.
* [Agent Checks](https://docs.everruns.com/features/agent-checks/), advisory, config-*time* review of an agent’s setup. Distinct from guardrails, which enforce at *runtime*.
---
# Host Shell
> Run bash commands on the machine hosting the agent, bounded by a kernel policy (Landlock and seccomp on Linux, Seatbelt on macOS).
Source: "}` | run that binary |
| `{"reexec_self": [""]}` | re-exec this binary with those leading arguments |
`everruns-host` ships `everruns-sandbox-exec` under the same feature, but cargo does not build a dependency’s binaries, so a single-binary host will not find one beside it. Such a host routes the arguments into `everruns_host::containment::worker::run_from_args` from its own `main` and selects `reexec_self`. See `examples/host-shell-agent`.
## See Also
* [Bashkit Shell](https://docs.everruns.com/capabilities/bashkit-shell/), the sandboxed interpreter
* [File System](https://docs.everruns.com/capabilities/file-system/), file operations on the same workspace
* [Daytona](https://docs.everruns.com/capabilities/daytona/) and [E2B](https://docs.everruns.com/capabilities/e2b/), real binaries on someone else’s machine
* [Capabilities Overview](https://docs.everruns.com/capabilities/)
---
# Infinity Context
> Trim live prompt history and query older messages on demand, so conversation length is not bounded by the context window.
Source: ` form accepted by Agent capability configuration. Capture JSON results and use `jq` to pass dependent IDs or capability references to later commands in the same script. No separate MCP attachment operation is needed.
## Scope and authorization
Platform tools are always bound to the current session’s organization. Their schemas do not accept `organization_id`, and an injected override is rejected. The server resolves the session’s human owner for every distributed call and applies that caller’s normal command permissions. Attaching this capability does not grant authority the owner does not already have.
## Autonomous workflows
For recurring autonomous work, create an Agent and an Agent Trigger. Do not use a schedule on the Platform Chat session: that would wake the management chat, not provision an independently owned worker workflow.
Credentials are not transferred from Platform Chat session secrets into a new Agent. Configure integrations through their supported Agent-scoped credential or connection flow; do not paste credentials into command scripts.
## Platform documentation
When the build embeds the product documentation, this capability mounts it at `/workspace/docs` as a read-only virtual filesystem, served from memory with no database writes per session. Agents browse it with the standard file tools (`read_file`, `list_directory`, `grep`) or with `cat`, `ls`, and `grep` through Bashkit Shell.
Key sections:
* `/workspace/docs/getting-started/`, introduction, concepts, architecture
* `/workspace/docs/features/`, SDK, CLI, UI, events, harnesses, capabilities
* `/workspace/docs/capabilities/`, per-capability reference
* `/workspace/docs/integrations/`, external integrations (Slack, Daytona, etc.)
* `/workspace/docs/advanced/`, budgets, compaction, embedding, network access
* `/workspace/docs/sre/`, environment variables, runbooks
## See also
* [Platform Chat harness](https://docs.everruns.com/built-ins/harnesses/platform-chat/)
* [MCP](https://docs.everruns.com/features/mcp/)
* [Agent Triggers](https://docs.everruns.com/features/agent-triggers/)
* [Platform Management](https://docs.everruns.com/capabilities/platform-management/), the removed predecessor
---
# Platform Management (removed)
> Removed capability. Its management tools are superseded by the catalog-backed Platform capability.
Source: .”* When (and only when) a continuation was actually scheduled, it appends *“We’ll continue work automatically once it resets.”*, so the copy never promises a resumption that will not happen. Without the capability, the same error stays generic and no continuation is scheduled.
## Configuration
### Default
```json
{
"capabilities": ["usage_limit_auto_continue"]
}
```
Defaults: continuation fires **120 seconds** after the reported reset with the prompt **“Continue tasks”**.
### Custom delay and prompt
```json
{
"capabilities": [
{
"ref": "usage_limit_auto_continue",
"config": {
"delay_seconds": 300,
"prompt": "Resume the migration where you left off."
}
}
]
}
```
| Field | Type | Default | Description |
| --------------- | ----------------- | ------------------ | ---------------------------------------------------------------------------------------------------------- |
| `delay_seconds` | integer (0–86400) | `120` | How long to wait after the reset before resuming. A small buffer avoids racing the provider’s reset clock. |
| `prompt` | string | `"Continue tasks"` | Message injected to resume work when the limit resets. |
## When To Enable
Use this capability for agents running long, autonomous, or scheduled work on provider plans that enforce usage-window limits, where a multi-hour stall would otherwise require a human to manually restart the session.
## Limitations
* Requires a provider whose error reports a concrete reset time; without one the error copy stays generic and no continuation is scheduled.
* The continuation is delivered through the session schedule machinery, so it requires both a session schedule store and a schedule poller in the deployment. The hosted platform provides both. The default embedded runtime provides neither, so this capability is **not** part of the runtime-safe preset, an embedder must wire a `SessionScheduleStore` (e.g. via a schedule-store factory) and run a poller for it to take effect; otherwise its error hook is a no-op and the error copy stays generic.
## See Also
* [Schedules](https://docs.everruns.com/capabilities/session-schedules/), the scheduling machinery the continuation reuses
* [Capabilities](https://docs.everruns.com/features/capabilities/), the capability model
---
# User Hooks
> Run user-authored shell commands at lifecycle and tool events. Block, mutate, or audit agent actions from outside the model.
Source: "}}` and the turn proceeds with the rewritten text. For example, to prepend a house style reminder:
```jsonc
{
"id": "prepend_style_note",
"event": "user_prompt_submit",
"executor": {
"type": "bash",
"command": "echo \"$EVERRUNS_HOOK_PAYLOAD_JSON\" | jq -c '{decision:\"mutate\",patch:{message:(\"[reminder: follow the house style guide]\\n\" + .data.message)}}'"
},
"on_error": "warn"
}
```
### Log every completed turn
`turn_end` is advisory, its decision is ignored, so use it for side effects like metrics or audit trails. The payload’s `data.success` reports whether the turn finished cleanly.
```json
{
"capabilities": [
{ "ref": "session_file_system" },
{ "ref": "bashkit_shell" },
{
"ref": "user_hooks",
"config": {
"hooks": [
{
"id": "log_turn_end",
"event": "turn_end",
"executor": {
"type": "bash",
"command": "echo \"$EVERRUNS_HOOK_PAYLOAD_JSON\" | jq -r '.ts + \" turn \" + .turn_id + \" success=\" + (.data.success | tostring)' >> /workspace/.turn-log; echo '{}'"
},
"timeout_ms": 3000,
"on_error": "warn",
"description": "Append one line per completed turn to /workspace/.turn-log"
}
]
}
}
]
}
```
## Observability
Today, hook decisions and errors are recorded in the server logs via `tracing`: blocks, ignored post-hook blocks, mutations, `on_error` outcomes, and `disabled_contributions` mutings are logged with the resolved `hook_id` and the `tool_call_id`.
**Planned (deferred):** structured `hook.invoked` / `hook.completed` / `hook.blocked` / `hook.warning` events emitted through the same observability pipeline as tool events (Braintrust, OTel). These are not emitted yet, see the spec’s observability section.
## Event firing
All six events fire:
* **`pre_tool_use`**: before each tool call; can block or mutate the call.
* **`post_tool_use`**: after each tool call; can mutate the result.
* **`session_start`**: after a session is created (mounts + initial files in place); advisory.
* **`session_end`**: when a session is deleted, before VFS eviction; advisory.
* **`user_prompt_submit`**: on the first reason iteration, before the LLM is consulted; can block (aborts the turn with a user-facing message) or mutate the user message text.
* **`turn_end`**: when a turn reaches a terminal outcome; advisory.
Note on `user_prompt_submit` blocking: the API persists the user message and runs the turn asynchronously, so a block does not reject the HTTP request, it aborts the turn. The session shows a `turn.failed` outcome carrying the hook’s `user_message`.
## See also
* [`knowledge/runtime-resources/user-hooks.md`](https://github.com/everruns/everruns/blob/main/knowledge/runtime-resources/user-hooks.md), full contract
* [`knowledge/execution/capabilities.md`](https://github.com/everruns/everruns/blob/main/knowledge/execution/capabilities.md), capability framework
* [`knowledge/security/threat-model.md`](https://github.com/everruns/everruns/blob/main/knowledge/security/threat-model.md), TM-HOOK entries
* [Bashkit Shell](https://docs.everruns.com/capabilities/bashkit-shell/), the sandbox that runs hook commands
---
# Web Fetch
> Fetch a URL and convert its HTML to markdown.
Source: ` sections as the first user message — never as system prompt.
* If a configured file doesn’t exist, the agent operates normally.
## Prompt order
Every turn the model sees, top-to-bottom:
1. **System prompt**: harness safety instructions, tool guidance, role (cache-stable).
2. **Conversation context**: the resolved `AGENTS.md` hierarchy, broadest scope first — model-visible, re-resolved every turn, but below system instructions in precedence.
3. **Conversation history and the latest user message**.
Explicit user instructions win over project files on conflict; system instructions always win over both.
## Limits
* Content cap: **32 KiB per file** (excess truncated with a warning), plus a **128 KiB total budget** per turn binding hierarchy depth — deeper files past the budget are omitted with an explicit note.
* Plain Markdown, no required sections.
* At most 16 instruction files resolved per turn.
## Compatibility
Everruns keeps `AGENTS.md` as the default. Configure the capability with `files` to read additional tool-specific files:
```json
{
"files": ["AGENTS.md", "CLAUDE.md"]
}
```
## Do something
* [Use AGENTS.md for project instructions](https://docs.everruns.com/how-to/use-agents-md/), full guide with examples.
* [Equip an agent with tools](https://docs.everruns.com/how-to/equip-agents-with-tools/), adding capabilities generally.
## See also
* [AGENTS.md capability reference](https://docs.everruns.com/capabilities/agent-instructions/), config schema.
---
# Agent Triggers
> Run an agent proactively on a recurring schedule, choose session reuse, test it immediately, and inspect recent outcomes.
Source: ` (default 300), `--no-stream` to queue without waiting.
## Output formats
Every command accepts `-o` / `--output`:
```bash
everruns agents list -o json
everruns agents list -o yaml
```
`--quiet` suppresses headers and prints only the essential identifier, useful for capturing IDs in shell variables.
## See also
* [Automate with the CLI](https://docs.everruns.com/how-to/automate-with-the-cli/), `jq`, quiet mode, scripting patterns.
* [Define agents as files](https://docs.everruns.com/how-to/define-agents-as-files/), file formats for `-f`.
* [SDK](https://docs.everruns.com/features/sdk/), the programmatic equivalent.
## Agent composition
Discover and manage the plugins, skills, and knowledge bases available to agents:
```bash
# Plugins
everruns plugins list
everruns plugins get
everruns plugins install
everruns plugins uninstall
# Skills
everruns skills list
everruns skills get
everruns skills create ./SKILL.md
everruns skills delete
# Knowledge bases
everruns knowledge-bases list
everruns knowledge-bases get
everruns knowledge-bases create "Product docs" --description "Published product documentation"
everruns knowledge-bases delete
```
Resource identifiers are URL-encoded before requests are sent. Use the global `--output json` or `--output yaml` option for machine-readable discovery output.
Skill creation reads the supplied Markdown file and sends its contents to Everruns. Knowledge document ingestion and assigning composition resources to an agent are not yet exposed by the CLI.
---
# Evals
> Define, run, and track behavioral tests for Agents. Each case runs a real session and is scored.
Source: `.
2. **Activation** (under 5,000 tokens) — full SKILL.md instructions load when the agent calls `activate_skill`.
3. **Resources** — bundled files mount at `/skills/{name}/` and are read with the normal filesystem tools.
This keeps thousands of skills available to an agent without burning context on the ones it doesn’t need.
## Two ways to use skills
| Source | Where they live | When to use |
| ---------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------- |
| Workspace skills | `/.agents/skills/` in the session VFS | Project-specific skills; per-session experimentation |
| Registry skills | Organization-wide via the [Skills Registry](https://docs.everruns.com/features/skills-registry/) | Shared across agents; managed via API |
The built-in `skills` capability handles workspace discovery. Registry skills appear in the capability system as virtual capabilities (`skill:{uuid}`).
## Do something
* [Package an agent skill](https://docs.everruns.com/how-to/package-a-skill/) — author a SKILL.md, bundle scripts.
* [Publish a skill to the registry](https://docs.everruns.com/how-to/publish-a-skill-to-the-registry/) — share it.
## See also
* [Agent Skills capability reference](https://docs.everruns.com/capabilities/agent-skills/) — tools, dependencies, config.
* [Skills Registry](https://docs.everruns.com/features/skills-registry/) — the org-wide registry.
* [SKILL.md format spec](https://agentskills.io/) — upstream open spec.
---
# Skills Registry
> API endpoints for managing organization-wide skill packages, create from SKILL.md or ZIP, validate, assign to agents as virtual capabilities.
Source: ,
}
struct Lookup;
#[capability::async_trait]
impl capability::Handler for Lookup {
type Input = LookupInput;
type Output = Record;
type Error = capability::Error;
fn name(&self) -> &str { "lookup_record" }
fn description(&self) -> &str { "Look up one record by exact id." }
fn hints(&self) -> capability::Hints {
capability::Hints::default()
.readonly(true)
.idempotent(true)
}
async fn execute(
&self,
input: Self::Input,
context: capability::Context,
) -> Result {
context.progress("Looking up the record").await;
if input.id != "rec_42" {
return Err(capability::Error::user(
"record_not_found",
"No record has that id",
).details(capability::serde_json::json!({ "id": input.id })));
}
Ok(Record {
id: input.id,
score: 0.98,
labels: vec!["verified".into()],
})
}
}
let records = capability::Definition::new(
"records",
"Records",
"Application-owned record lookup.",
)
.instructions("Use exact record ids and do not infer missing records.")
.metadata(capability::serde_json::json!({ "owner": "risk" }))
.tool(Lookup);
let agent = Agent::builder()
.instructions("Answer with verified record data.")
.provider(OpenAI::from_env()?)
.model("gpt-5.6-terra")
.capability(records)
.build()?;
```
Both input and output types must satisfy the compile-time protocol bounds. The generated schemas are available through `Definition::tools()[..].spec()` for tests, documentation, or a host catalog. Results are serialized directly to JSON, so structs, arrays, numbers, booleans, and null do not pass through a string conversion.
## Errors
Return `capability::Error::user(code, message)` for an expected domain failure. Add bounded JSON details when they help the model recover. The code, message, and details travel through the model-visible tool-error channel.
Return `capability::Error::internal(code, message)` for diagnostic details that are unsafe to show to the model, such as network internals or implementation bugs. The engine logs the diagnostic and gives the model a generic error; internal details do not cross the model boundary. Never include credential values or other secrets in any error because host logs may retain internal diagnostics.
Custom application error enums can implement `Into` and be used as `Handler::Error`.
## Context, progress, and cancellation
`capability::Context` exposes only stable lifecycle data:
* opaque session and workspace ids;
* the resolved locale, when present;
* best-effort correlated `tool.progress` events;
* a call-scoped cancellation signal.
Observe progress through `Session::events()` and `SessionEventKind::ToolProgress`. Everruns does not currently expose a custom capability result-streaming protocol; return one typed result when execution finishes.
Normal awaited work needs no cancellation branch. Cancelling a turn drops the handler future. Clone `context.cancellation()` only into child tasks, processes, or watchers that might otherwise survive after `execute` is dropped. The signal fires on cancellation and on every other call completion path.
## Security boundary
The advanced SPI intentionally does not export provider credentials, stores, tenant or organization objects, registries, payment authority, filesystem backends, or other host services. Pass application-owned clients or state into your handler struct when constructing the definition. A handler is trusted application code and retains whatever process authority those values provide; `Hints` describe behavior but do not enforce authorization, egress, or approval policy. Apply authorization, egress policy, timeouts, and input bounds at those application boundaries, and never place secrets in capability or tool metadata.
For a complete provider-backed program, run the [`advanced_capability` example](https://github.com/everruns/everruns/blob/main/crates/everruns/examples/advanced_capability.rs).
Some built-ins expect the application to supply behavior rather than configuration. `ask_user` is one: it needs someone to answer, so `AgentBuilder::ask_user` takes a responder instead of a config value. See [Answer agent questions](https://docs.everruns.com/framework/ask-user/).
---
# Agents
> Describe an agent with instructions, a model, tools, files, integrations, and an optional workspace.
Source: ` is retained. It does not survive a process restart. A host that needs durable recovery supplies a `WorkBackend`.
* **Scheduling:** `Immediate` work is claimable now; `At(SystemTime)` work is not claimable early. Both are one-shot requests. The host owns polling; recurring calendars and schedule runners stay host concerns. There is no hidden scheduler or database in the default build.
* **Delivery:** task and wake claims are leased and at least once. If a process stops before settlement or acknowledgment, the item is claimable after the lease expires. Each retry has a new token and attempt; stale attempts cannot settle newer work.
* **Idempotency:** task keys and direct-wake keys each have a session-scoped namespace. Repeating the same request returns the original task or wake. Reusing a key for different input fails with `IdempotencyConflict`. Workers should also deduplicate external side effects on the stable task id or submission key.
* **Cancellation:** pending work cancels immediately. Running work records cancellation intent; the worker checks the latest task snapshot, stops cooperatively, then reports `TaskOutcome::Canceled`. A task may still succeed if it passes its safe cancellation point first.
* **Wakes:** applications can request an immediate wake directly. A task can also create one atomic completion wake with `WakePolicy::OnCompletion`. Wakes use the same lease/retry/acknowledgment model as tasks.
Durable platform scheduling, retention, distributed polling, and multi-host coordination belong in the host’s `WorkBackend`; they are not enabled by the offline Framework default. The in-memory backend retains accepted payloads until it is dropped and does not enforce admission quotas, so hosted providers must apply tenant authorization, payload limits, quotas, and retention at their own boundary.
---
# Canonical Framework events
> Observe a complete agent turn through a bounded typed/raw bridge while keeping durability, live delivery, and derived history distinct.
Source: {
let session_id = request.session_id();
let current_high = self.high_watermark(session_id);
let (after, snapshot) = match request.cursor() {
None => (0, current_high),
Some(cursor) => {
if cursor.session_id() != session_id {
return Err(EventLogError::CrossSessionCursor {
detail: "cursor belongs to another session".into(),
});
}
match cursor.snapshot_high_watermark() {
Some(snapshot) if snapshot > current_high => {
return Err(EventLogError::ExpiredCursor {
detail: "cursor snapshot is not available".into(),
});
}
// Pinned continuation, then the polling form.
Some(snapshot) => (cursor.after_sequence(), snapshot),
None => (cursor.after_sequence(), current_high),
}
}
};
let limit = request.limit().get();
let mut events = self.events_in(session_id, after, snapshot, limit + 1);
let has_more = events.len() > limit;
if has_more {
events.pop();
}
let next_cursor = has_more
.then(|| {
let last = events.last().and_then(|event: &Event| event.sequence).unwrap_or(after);
EventCursor::continuation(session_id, last, snapshot)
})
.transpose()?;
EventPage::new(events, next_cursor, snapshot)
}
}
#[async_trait]
impl EventLog for MyEventLog {
async fn append(&self, request: EventRequest) -> Result {
if request.is_ephemeral() {
return Err(EventLogError::InvalidAppend {
detail: "ephemeral events are sink-only".into(),
});
}
// The log owns identity: assign the event id and the next per-session
// sequence, persist, and return the finalized canonical envelope.
let sequence = self.next_sequence(request.session_id);
let event = request.into_event(EventId::new(), sequence);
self.persist(&event)?;
Ok(event)
}
fn durability(&self) -> EventDurability {
EventDurability::CrashDurable
}
}
```
The contract an implementation must uphold:
* an accepted append owns id and sequence assignment and returns the finalized canonical `Event`, visible to the next read of that session;
* durable sequences are unique and strictly increasing per session, and need not be contiguous, gaps are expected when a reader projects an append-only physical log into a filtered logical event sequence;
* a continuation stays pinned to the first page’s high-watermark and cannot observe concurrent appends; a poll cursor can;
* cursor/session mismatches and inconsistent positions return the typed `EventLogError` variants above rather than panicking;
* the log is append-only. There is no truncate, rewind, or mutation contract, and `EventHistory` remains a read-only projection rather than a second writable message store.
`crates/everruns/tests/fixtures/external-consumer/event-log` in the repository is a complete out-of-workspace implementation exercised by repository CI.
## Ordering and bounded delivery
A subscriber receives events in channel arrival order and each session has its own stream. The canonical `sequence` field is a replay position, not a live delivery counter: durable events carry `Some(sequence)` and live-only ephemeral events such as streaming deltas carry no sequence. Persisted sequences increase monotonically per session and may have gaps. Ephemeral events do not consume replay positions.
The live stream has a bounded buffer and never applies backpressure to the agent turn. A dropped or slow subscriber cannot stall model or tool execution. If a subscriber falls behind, `recv()` and `try_recv()` return `EventStreamError::Lagged { missed }`; loss is never hidden. The next receive can continue from the oldest retained event, but the renderer must treat its live projection as incomplete.
Streaming deltas are provisional and sink-only. Completed assistant/tool events are authoritative, and an output-replacement event means accumulated text for that message must be discarded. Durable events reach the live sink only after their log append commits; ephemeral events go directly to the sink and never enter history. After live lag, a Framework application can rebuild its persisted transcript with bounded [`Session::history()` pages](https://docs.everruns.com/framework/session-history/). That projection excludes ephemeral deltas by design. Applications that need raw durable envelopes rather than derived messages can provide and read an `EventLog` through the advanced `everruns-host` SPI; neither recovery path relies on the in-process subscriber.
## Cancellation and failure
Pass a `CancellationToken` through `RunOptions` to stop a turn. Cancellation produces both a `Turn` with `TurnStopReason::Cancelled` and a correlated `turn.cancelled` event carrying the same `turn_id`.
Runtime failures remain available through `Session::run()`’s outcome/error semantics and the event stream. Subscribe before running and continue draining the stream after the run resolves to retain the terminal failure event and its full structured payload.
The complete runnable example is [`canonical_events.rs`](https://github.com/everruns/everruns/blob/main/crates/everruns/examples/canonical_events.rs):
```bash
cargo run -p everruns --example canonical_events
```
---
# Portable and hosted capabilities
> Understand which capabilities run in the Framework and which require the Everruns Platform host.
Source: {
Agent::builder()
.instructions("Use the company model gateway.")
.provider(Provider::new("company-gateway", driver))
.model("assistant-v2")
.build()
}
```
A driver implements the streaming chat-completion contract. It receives the resolved endpoint, model-facing messages (`everruns::llm::Message`), and call configuration, and returns an `LlmResponseStream`. Exact trait methods and event shapes live in the [`everruns::ChatDriver` API reference](https://docs.rs/everruns/latest/everruns/trait.ChatDriver.html).
Keep credential lookup and refresh in trusted host/provider configuration. Model ids must remain safe to log, compare, store, and pass across application boundaries. Provider errors should preserve useful decisions without including secrets.
A driver registered through a `DriverDescriptor` also declares which environment variables it reads, on its own credential fields. Declaring is inert — the driver never reads them — and it is what lets a caller resolve the provider from the environment without any central name mapping. See [Credentials](https://docs.everruns.com/framework/credentials/).
Use focused provider crates when they already implement the protocol you need. Custom backends and provider registry topology belong to [low-level host composition](https://docs.everruns.com/framework/custom-backends/), not ordinary model selection.
---
# Direct Decisions
> Ask Decisions for a number instead of prose, without building an agent.
Source: > {
let decisions = Decisions::new("jev-latest", TypeSafeAI::from_env()?);
let text = "CONGRATULATIONS! You've WON $1,000,000. Click here to claim your prize now!";
let spam = decisions.probability("Is this message spam?", text).await?;
println!("spam: {spam:.2}");
if spam > 0.9 {
println!("quarantined");
}
Ok(())
}
```
```text
spam: 0.98
quarantined
```
One number, and your own `> 0.9` decides — the model reports how likely, not what to do. The same call answers `0.03` for “Standup moved to 10am.” and `0.74` for a bare “Claim your prize now!”; the middling ones are what a threshold is for.
`--features typesafe` adds `TypeSafeAI` and the `Jev` capability. Without it `everruns` names no vendor: `Decisions::new` takes any `DecisionsService`.
## Three primitives
A decision asks one or more questions about the same state. Each is one of three shapes:
```rust
use everruns::Decisions;
let answers = decisions
.about("I've been on hold for two hours and my card was charged twice.")
.noul("urgent", "Does this convey urgency?")
.score(
"severity",
"How severe is the problem the writer describes?",
[
"A minor annoyance",
"A real problem with their account",
"Serious harm requiring immediate action",
],
)
.choice(
"queue",
"Which team should handle this message?",
["billing", "technical", "sales"],
)
.send()
.await?;
let urgent: f64 = answers.probability("urgent")?;
let queue: &str = answers.selected("queue")?;
```
* **`noul`** — whether something holds, as the probability of yes. A value near 0.5 means yes and no are near-equally likely, not “medium”. ([Noul](https://docs.typesafe.ai/primitives/noul))
* **`choice`** — exactly one option from your set, with the distribution behind it. Needs at least two options. ([Choice](https://docs.typesafe.ai/primitives/choice))
* **`score`** — a position along levels you define, lowest first. Needs at least two levels. ([Score](https://docs.typesafe.ai/primitives/score))
The three are System One’s own, so TypeSafe’s [Primitives](https://docs.typesafe.ai/primitives) documents what each answer means and how to choose between them, and [State](https://docs.typesafe.ai/concepts/state) covers what to put in the `about(...)` value. Everruns names them the same way rather than inventing synonyms.
Questions in one call are answered **in parallel inside a single request**, so asking five costs one round trip, not five. TypeSafe calls leaning on that [speculative fan-out](https://docs.typesafe.ai/patterns/fan-out): ask the questions you *might* need, and let your code decide which ones mattered.
## Ids are yours; instructions are the model’s
The id labels the answer for your code and is never sent to the model. A question whose meaning lives in its id asks nothing:
```rust
// Wrong: the model never sees "is_the_joke_funny".
.noul("is_the_joke_funny", "?")
// Right: the instructions carry the question.
.noul("funny", "Would a general audience laugh at this joke?")
```
The same applies to score levels. Describe concrete situations — “A minor annoyance” reads on its own where “2 out of 5” does not.
## Read the tail, not the average
For “is there any serious hit here” rules, read the probability mass at or above a level rather than the weighted score. Something probably fine but possibly awful must not average into fine:
```rust
// Not: answers.score("severity")? > 1.5
let serious = answers.tail("severity", 2)?;
if serious > 0.3 {
println!("escalated");
}
```
## Errors
`DecisionsError` separates configuration mistakes from service failures, the same split [`CompletionError`](https://docs.everruns.com/framework/direct-model-calls/#errors) makes:
* `MissingService` — the decisions was built without a service to reach.
* `NoQuestions` — the decision was sent with nothing to ask.
* `Unconfigured` — the service exists but the deployment never configured its credential, so it would answer nothing.
* `NoSuchAnswer(id)` — you read an id that was not asked, or read an answer as the wrong shape (a `score` as a probability).
* `Call(..)` — the service call failed, carrying the `AgentLoopError`.
The first three are caught before any request leaves the process. `Unconfigured` is worth handling separately: a guardrail treats it as fail-open, but a direct caller usually wants to know the number never arrived rather than read a confident-looking default.
## Going lower
`Decision` is a thin value-first layer over `DecisionsService`, which is public. Applications that already hold a service — or implement their own, over a different vendor or a local model — can call it directly with `everruns`’s `DecisionRequest`, `DecisionQuestion`, and `DecisionAnswer` re-exports:
```rust
use everruns::{DecisionQuestion, DecisionRequest, DecisionsService};
let outcome = service
.evaluate(
DecisionRequest::new("Claim your prize now!")
.ask("spam", DecisionQuestion::noul("Is this message spam?")),
)
.await?;
```
That surface is the contract itself: every question type and the full `DecisionOutcome`, including usage, with nothing defaulted for you. Implementing `DecisionsService` is also how a different decisions — another vendor, or a fine-tuned local model — plugs into the same `Decisions`, guardrails included.
## Choosing a model
The model is named up front, the way [`Model::new`](https://docs.everruns.com/framework/direct-model-calls/) names one: the service is transport, and the model is the thing that answers. There is no default to inherit without noticing, because a threshold calibrated against one version is not evidence about the next.
Ids are the provider’s own, so they are spelled the way the vendor spells them. `jev-latest` is TypeSafe’s alias for the current Jev, so it tracks whatever the current version is; an exact id like `jev-1.13.0` pins one, so a vendor update cannot move your thresholds under you. Bare `jev` is not an id the API knows — nothing here rewrites what you pass.
Ask for the alias and read back what answered, which is the id to pin once a threshold is calibrated:
```rust
let version = answers.model(); // "jev-1.13.0" for a "jev-latest" request
```
A single call can name a different model with the same method on the request builder, and it wins for that call:
```rust
let answers = decisions
.about("...")
.noul("urgent", "Does this convey urgency?")
.model("jev-1.13.0")
.send()
.await?;
```
A deployment that must pin a model does so by never exposing the knob in the config an agent writes — not by the type being unable to carry one, because there will be other decision services and other models.
A deployment running the Everruns platform configures a separate `UTILITY_TYPESAFE_API_KEY` for its [guardrails](https://docs.everruns.com/capabilities/guardrails/) — a different account from the one an embedding application holds.
## Giving an agent the decisions
Everything above is agentless: your code asks, your code decides. The other half is letting an *agent* classify as part of its own work — checking a claim against a source before citing it, rating a draft before sending it.
`Jev` is the same decisions as a capability, so an agent gets it as a tool:
```rust
use everruns::{Agent, Engine, Jev, Model, OpenAI};
let agent = Agent::builder()
.name("reviewer")
.instructions(
"You review copy. When asked how something reads, measure it with \
jev_decision and report the numbers rather than judging by eye.",
)
.model(Model::new("gpt-5.6-terra", OpenAI::from_env()?))
.capability(Jev::from_env()?)
.build()?;
let session = Engine::new().create(agent);
let turn = session
.run("Rate this subject line for pushiness: 'Act now before it is too late'")
.await?;
```
The agent calls `jev_decision`, writing its own questions about whatever it is looking at, and gets the same calibrated numbers back. It is the identical tool the hosted [TypeSafe integration](https://docs.everruns.com/integrations/typesafe/) gives platform agents — same name, same schema — so behavior matches whether you embed the Framework or run on Everruns.
Which one to reach for:
| | you decide | the agent decides |
| ------------ | -------------------------------------- | --------------------------------- |
| **who asks** | your code writes the questions | the model writes the questions |
| **use** | `Decisions` | the `Jev` capability |
| **good for** | a policy check, a routing rule, a gate | verification inside a longer task |
## What stays with an agent
A decision owns no session, no history, and no workspace, and runs no tools. Reach for an [agent](https://docs.everruns.com/framework/agents/) as soon as the work needs any of those. Typed output guarantees the interface, not the truth: validate thresholds against your own data and consequences.
## Testing without a credential
`Decisions::simulated` returns a fixed number from an in-process stub. It is a **test double**, not a local decisions: it runs no inference, reads nothing from the state you pass it, and is not a way to classify without a provider. It exists so tests and examples can assert on the code around a decision without a network call or an API key.
Real work always goes through a decisions service — `TypeSafeAI` above, or your own `DecisionsService`.
```rust
use everruns::Decisions;
let p = Decisions::simulated(0.93)
.probability("Does this convey urgency?", "Two hours on hold.")
.await?;
assert!(p > 0.9);
```
Because the answer is fixed, a simulated decision proves your threshold logic runs — never that a real decisions would return that number.
The runnable version of this page is [`direct_decisions.rs`](https://github.com/everruns/everruns/blob/main/crates/everruns/examples/direct_decisions.rs). It uses the stub by default so it runs with no key; pass `--live` (with `--features typesafe` and `TYPESAFE_API_KEY` set) to send the same questions to a real decisions. [`agent_decisions.rs`](https://github.com/everruns/everruns/blob/main/crates/everruns/examples/agent_decisions.rs) does the same for the agent path.
---
# Direct Model Calls
> Call a model once through the Framework's provider edge, without building an agent.
Source: {
Agent::builder()
.name("bashkit-repo-agent")
.instructions(include_str!("resources/instructions.md"))
.provider(OpenAI::new(api_key))
.model(MODEL)
.max_iterations(12)
.workspace(workspace)
.workspace_policy(WorkspacePolicy::read_write())
.capability(BashkitShell::new())
.build()
}
```
## Run and verify
The shared observer displays a bounded shell timeline and waits for a successful turn. The host then verifies the mounted files itself.
```rust
let agent = agent::build(api_key, &workspace)?;
let engine = Engine::new();
let session = engine.create(agent);
demo::run(&session, &request).await?;
verify_release(&workspace, &release_date)?;
```
Use `session.send_and_wait(&request).await?` when a live tool timeline is not needed.
## How Bashkit behaves
Bashkit interprets shell scripts in-process against `/workspace`. The model gets no host filesystem outside that mount, network, Git credentials, or subprocess execution. Commands, loops, output, and script size are bounded. Repository text is treated as untrusted data rather than agent instructions.
## Validate it
```bash
cargo test -p everruns-bashkit-repo-agent
bash examples/bashkit-repo-agent/demo/record.sh --check
```
These checks validate argument handling, fixture state, host assertions, and the live-demo contract without provider credentials. They do not grade model quality.
## Demo and recording
The screencast types the release task into the interactive binary, displays real `bashkit_shell` calls, and ends with host-side checks over the mutated files. It does not replay prepared output. Read the [captured transcript](https://github.com/everruns/everruns/blob/main/examples/bashkit-repo-agent/demo/transcript.txt) at your own pace.
With VHS, ffmpeg, a VHS-compatible browser, and funded OpenAI credentials:
```bash
bash examples/bashkit-repo-agent/demo/record.sh
```
The script uses `OPENAI_API_KEY` when exported, otherwise Doppler project `everruns-dev`, config `dev`. It updates the GIF and transcript only after the model turn and host verification succeed.
## Adapt it safely
Replace the fixture, request acceptance criteria, and verifier together. Keep the mount disposable and narrow, start read-only unless mutation is required, and verify consequential claims outside the model/tool boundary.
## Boundaries
This demonstrates one sandboxed repository mutation, not a general coding agent. It cannot fetch dependencies, run native programs, commit, or push. The Framework session is in-memory.
## Source map
`src/main.rs`: input, session, and verification flow; `src/agent.rs`: agent definition; `src/fixture.rs`: fixture materialization and assertions; `src/resources/`: prompt and sample repository; `demo/`: live VHS recording and transcript. `examples/demo-support::shell` provides terminal presentation.
---
# Coding Review Agent
> Tool-based code inspection, a tightly scoped execution tool, and distinguishing test failure from tool failure.
Source: {
Agent::builder()
.name("everruns-support-agent")
.instructions(include_str!("resources/instructions.md"))
.provider(everruns_anthropic::provider("anthropic", api_key))
.model(MODEL)
.max_iterations(12)
.tool(tools::search_docs())
.tool(tools::read_doc())
.build()
}
```
## Send, observe, and wait
The Framework interaction stays small in `main.rs`. The shared demo helper subscribes before sending, shows bounded tool previews, waits for completion, and rejects unsuccessful turns. Use `session.send_and_wait(&question).await?` when a live tool timeline is unnecessary.
```rust
let agent = agent::build(api_key)?;
let engine = Engine::new();
let session = engine.create(agent);
println!("MODEL: {}", agent::MODEL);
demo::run(&session, &question).await?;
```
This engine is in-memory. The agent can explain durable sessions from its corpus, but the example itself does not persist its session.
## How the tools work
`search_docs` ranks matches against five bundled pages and returns page IDs, public URLs, and matching excerpts. `read_doc` accepts only those page IDs and returns the complete snapshot. It never accepts an arbitrary filesystem path.
## Validate the behavior
```bash
cargo test -p everruns-framework-support-agent
bash examples/everruns-support-agent/demo/record.sh --check
```
Tests cover content-based search, empty and unmatched queries, complete citable reads, arbitrary-path rejection, and interactive input validation. They do not grade the model’s answer; compare a live response with the expected outcome above.
CI runs the offline checks without provider credentials. Live model behavior is evaluated separately.
## Demo and recording
The screencast types a question into the same interactive binary shown above, then displays the actual Opus tool calls and answer. VHS hides most provider wait time but does not replace the model or tools with scripted output. Read the [captured transcript](https://github.com/everruns/everruns/blob/main/examples/everruns-support-agent/demo/transcript.txt) at your own pace.
With credentials exported and VHS, ffmpeg, and a VHS-compatible browser installed:
```bash
bash examples/everruns-support-agent/demo/record.sh
```
The script uses an exported `ANTHROPIC_API_KEY` when present, otherwise Doppler project `everruns-dev`, config `dev`. It updates `demo/demo.gif` and `demo/transcript.txt` only after a successful turn.
## Adapt it
Replace the bundled pages with a versioned documentation index while retaining separate search and read operations. Attach source/version metadata, restrict reads to authorized documents, and treat retrieved text as untrusted evidence rather than instructions.
## Boundaries
The corpus is a five-page snapshot from 2026-09-08, not a live search of docs.everruns.com. Provenance is recorded in `src/resources/docs/README.md`, and the snapshot can lag current APIs.
## Source map
`src/main.rs`: input and session execution; `src/agent.rs`: agent definition; `src/tools.rs`: bounded documentation retrieval; `src/resources/`: prompt and documentation corpus; `demo/`: live VHS recording, transcript, and recording script. `examples/demo-support` handles shared terminal presentation.
---
# Foreman
> A decisions supervising a coding agent it never has to stop.
Source: {
Agent::builder()
.name("support-agent")
.instructions(include_str!("resources/instructions.md"))
.provider(OpenAI::new(api_key))
.model(MODEL)
.max_iterations(12)
.tool(tools::lookup_customer())
.tool(tools::read_support_policy())
.build()
}
```
## Send, observe, and wait
The Framework interaction stays readable in `main.rs`. The shared demo helper subscribes before sending, filters events to this turn, shows bounded tool previews, waits for completion, and rejects unsuccessful turns. It changes presentation only; use `session.send_and_wait(question).await?` when you do not need the live tool timeline.
```rust
let agent = agent::build(api_key)?;
let engine = Engine::new();
let session = engine.create(agent);
println!("MODEL: {}", agent::MODEL);
demo::run(&session, question).await?;
```
This engine is in-memory. It does not demonstrate durable session storage; the Everruns Support example can explain that API, but does not itself persist its session.
## How the tools work
`lookup_customer` reads one of three fictional records from `src/resources/customers.json`. It returns facts, not a prewritten recommendation. `read_support_policy` returns the recovery rules from `src/resources/policy.md`. The model combines the two; no tool disables MFA or changes a real account.
## Validate the behavior
```bash
cargo test -p everruns-support-agent
bash examples/support-agent/demo/record.sh --check
```
Tests cover the distinct account states and rejection of unknown IDs. They do not grade the model’s recommendation: compare a live response with the expected outcomes above.
CI runs these offline checks without provider credentials. Live model behavior is evaluated separately; passing tests is not proof of answer quality.
## Demo and recording
The screencast runs the same `cargo run -q -p everruns-support-agent` command shown above. VHS hides most provider wait time but does not replace the model or tools with scripted output. Read the [captured transcript](https://github.com/everruns/everruns/blob/main/examples/support-agent/demo/transcript.txt) at your own pace.
With credentials exported and VHS, ffmpeg, and a VHS-compatible browser installed:
```bash
bash examples/support-agent/demo/record.sh
```
The recording script uses an exported `OPENAI_API_KEY` when present, otherwise Doppler project `everruns-dev`, config `dev`. It runs the real command inside VHS and updates `demo/demo.gif` plus `demo/transcript.txt` only after a successful turn. Public/demo data is safe here, but adapting tools may expose private data.
## Adapt it
Replace the fixture lookup with your authorized customer-data service. Keep policy separate from account facts, scope lookups to the authenticated customer, and return only fields needed for the support decision. Add human approval before any account mutation.
## Boundaries
All customers, policy rules, and support.example.com URLs are fictional. This is a read-only support exercise, not a live help desk.
## Source map
`src/main.rs`: input and session execution; `src/agent.rs`: agent definition; `src/tools.rs`: bounded account lookup and policy tool; `src/resources/`: prompt and bundled support data; `demo/`: live VHS recording, transcript, and recording script. `examples/demo-support` handles shared terminal presentation.
---
# Lifecycle Hooks
> Run typed application handlers at agent, turn, tool, and completion boundaries.
Source: {
Agent::builder()
.instructions("Use the configured provider.")
.provider(Provider::new("acme", driver))
.model("assistant-v1")
.build()
}
```
For a complete driver boundary, see [Custom providers](https://docs.everruns.com/framework/custom-providers/).
To call a model once without building an agent, see [Direct model calls](https://docs.everruns.com/framework/direct-model-calls/). To ask a provider which models it offers, and what each one supports, see [Model catalogs](https://docs.everruns.com/framework/model-catalogs/).
## Simulated models, for tests
```rust
use everruns::{Agent, Model};
let agent = Agent::builder()
.instructions("Answer deterministically.")
.model(Model::simulated("fixed response"))
.build()?;
```
`Model::simulated` is backed by the focused `everruns-llmsim` crate. It is a **test double that runs no inference** — it replays canned responses so tests can assert on agent behavior without a network call or an API key. It is not a local model and not a way to run Everruns without a provider. Depend on the crate directly when building a low-level host or scripting multi-turn provider behavior; ordinary Framework applications need only `everruns`.
For real work, pick a provider from [Supported providers](https://docs.everruns.com/framework/supported-providers/).
---
# Persistence
> Choose volatile memory, crash-durable local state, or the distributed durable Platform.
Source: > {
let agent = Agent::builder()
.instructions("You are a concise assistant.")
.provider(OpenAI::from_env()?)
.model("gpt-5.6-terra")
.build()?;
let engine = Engine::new();
let session = engine.create(agent);
let turn = session.send_and_wait("Explain durable execution in one sentence.").await?;
println!("{}", turn.response);
Ok(())
}
```
The model id stays credential-free: `.model("…")` names the model, and `.provider(…)` supplies the driver, endpoint, and key separately. `OpenAI::from_env` reads `OPENAI_API_KEY` and redacts it from debug output.
`send_and_wait` is the request/response convenience; use `send` when the application needs to stream output or steer a turn while it runs.
## Give it a tool
An agent becomes useful when it can act. Annotate a function and hand it over:
```rust
use everruns::{Agent, OpenAI};
#[everruns::tool]
/// Look up the current stock level for a SKU.
async fn stock_level(sku: String) -> Result {
Ok(inventory_lookup(&sku).await)
}
let agent = Agent::builder()
.instructions("Answer inventory questions. Use the tool rather than guessing.")
.provider(OpenAI::from_env()?)
.model("gpt-5.6-terra")
.tool(stock_level())
.build()?;
```
The macro derives the JSON schema from the signature, so the model sees typed arguments and the code stays ordinary Rust. See [Tools and macros](https://docs.everruns.com/framework/tools-and-macros/).
## Where to go next
* [Agents](https://docs.everruns.com/framework/agents/) — instructions, files, workspaces, and MCP.
* [Tools and macros](https://docs.everruns.com/framework/tools-and-macros/) — typed function tools.
* [Sessions](https://docs.everruns.com/framework/sessions/) — multi-turn state and history.
* [Framework architecture](https://docs.everruns.com/framework/architecture/) — how the pieces fit.
## Testing without a provider
Once you are building for real, you will want tests that do not call a model. `Model::simulated` replays a canned response through the same model/provider path:
```rust
use everruns::{Agent, Engine, Model};
let agent = Agent::builder()
.instructions("You are a concise assistant.")
.model(Model::simulated("Everruns is ready."))
.build()?;
let turn = Engine::new().create(agent).send_and_wait("Are you ready?").await?;
assert_eq!(turn.response, "Everruns is ready.");
```
It is a test double, not a local model: it runs no inference, so it proves your wiring rather than any model behavior. Use it in tests and CI, not as a way to run Everruns without a provider. See [Testing and simulation](https://docs.everruns.com/framework/testing-and-simulation/).
---
# Session History and Resume
> Read bounded conversation history and continue Framework sessions without importing host or runtime storage APIs.
Source: {
Ok(left + right)
}
let agent = Agent::builder()
.instructions("Use the add tool for arithmetic.")
.provider(OpenAI::from_env()?)
.model("gpt-5.6-terra")
.tool(add())
.build()?;
```
Use `#[everruns::tool(name = "…", description = "…")]` to override metadata, or `#[tool(rename = "…")]` on a parameter to change its model-facing name. Functions must be async, non-generic, and have plain named parameters.
The published `everruns-macros` package is an implementation crate. Its source lives at `crates/macros`, but applications should use the re-exported `everruns::tool` macro and should not depend on `everruns-macros` directly.
## Dynamic handlers
`FunctionTool::new` is available when a tool schema is determined at runtime:
```rust
use everruns::FunctionTool;
use serde_json::json;
let echo = FunctionTool::new(
"echo",
"Return the supplied text.",
json!({
"type": "object",
"properties": { "text": { "type": "string" } },
"required": ["text"]
}),
|args: serde_json::Value| async move {
Ok::<_, String>(args["text"].clone())
},
);
```
Prefer the macro for normal typed application tools. Use the dynamic form for schemas obtained from configuration or another protocol.
---
# Workspace Security
> Configure safe read and write scopes for in-process Everruns agents.
Source: > {
let policy = WorkspacePolicy::builder()
.allow_read("/")
.allow_write("generated")
.deny_write("generated/locked")
.allow_hidden(".github")
.build()?;
Ok(Agent::builder()
.instructions("Work only inside the configured workspace.")
.provider(OpenAI::from_env()?)
.model("gpt-5.6-terra")
.workspace(root)
.workspace_policy(policy)
.build()?)
}
```
## Defaults
`WorkspacePolicy::default()` and `WorkspacePolicy::read_only()` use the same secure baseline:
| Operation | Default |
| ---------------------------------------- | ------- |
| Read ordinary workspace files | Allowed |
| Write, create, or delete | Denied |
| Read or write hidden paths | Denied |
| Read or write common credential paths | Denied |
| Read framework-managed `.agents` content | Allowed |
| Recursively delete a directory | Denied |
`WorkspacePolicy::read_write()` is an explicit opt-in to ordinary writes. It does not expose additional hidden or sensitive paths and does not enable recursive deletion. It also keeps common dependency and build directories such as `node_modules` and `target` non-writable at every depth. For narrower access, start with `WorkspacePolicy::builder()`, which has no readable or writable scopes until you add them.
A custom builder does not inherit the default `.agents` exception. If the agent needs workspace-provided instructions or skills, add both a readable scope and a narrow `allow_hidden(".agents")` opt-in.
Protected path names are defense in depth, not content-based secret scanning. Keep credentials outside the mounted workspace, add explicit deny scopes for application-specific secret locations, and never place credentials in `.agents` content.
## Matching and precedence
Scopes are literal path prefixes, not globs, and compare ASCII letters without case sensitivity so a deny cannot be bypassed on a case-insensitive backend. `generated` therefore includes `generated/report.md` but not `other/generated/report.md`.
* A deny scope always wins over an allow scope.
* `deny_write_component` rejects an exact directory or file name at every depth; `deny_write` rejects one rooted path subtree.
* Hidden paths need `allow_hidden` for the narrow path that should be visible.
* Common credential paths need the stronger `allow_sensitive` opt-in, which also permits hidden components inside that specific scope.
* `compose` is restrictive: every composed policy must allow the operation. A library can add constraints without accidentally broadening the application’s policy.
Trusted starter files are installed before model access is enforced. This lets an application seed a read-only file even under a non-writable policy. Later reads, writes, and deletes of that file still go through the policy, so seeding a hidden file does not automatically expose it.
## Paths and containment
Policy paths live in one portable workspace namespace. These spellings identify the same file:
```text
src/lib.rs
/src/lib.rs
/workspace/src/lib.rs
```
Traversal (`..`), NUL bytes, and backslash-separated paths fail closed. Host absolute paths are backend-specific and are not portable policy scopes; use `/workspace/...` in application configuration and model instructions.
The policy layer controls visibility and mutation. The selected filesystem backend remains responsible for mapping workspace paths to storage. The local host backend canonicalizes its root, keeps resolved paths contained, and rejects symlinks in existing path components before every operation. An absolute path outside the configured root cannot expose that host file.
The policy governs capabilities that use Everruns’ session filesystem. A custom tool that calls `std::fs`, launches a shell, or uses another storage API does not pass through this boundary. Apply equivalent restrictions to those tools or run them in a sandbox.
## Symlinks and races
The built-in local backend rejects a symlink introduced after the workspace was configured because it rechecks components on every operation. This blocks normal traversal and symlink-swap attempts between operations.
It is not an OS sandbox. A malicious process running as the same operating system user can race a final path check and filesystem syscall. If local processes are mutually untrusted, use an isolated sandbox/filesystem backend or operating-system isolation. Do not use `WorkspacePolicy` as a substitute for that process boundary.
## Backend extension
The in-process host applies the policy after resolving the platform’s filesystem factory. In-memory, local-disk, database, and custom backends all receive the same policy checks. Backend authors still own containment, symlink-safe I/O, quotas, durability, and atomic update guarantees for their storage system.
Directory listings and grep are enforced at the same boundary as direct reads. Denied files are not opened by policy grep, and denied names, match counts, and byte totals are not returned. Recursive deletes inspect descendants through the backend before deletion, so opting into recursion does not override a deny or protected descendant. Backends with mutable external state must still treat that preflight-to-delete window as a race boundary.
`WorkspaceRootSet` additional roots are named mounts inside one selected head; they are not independent heads and carry no fork/reopen lifecycle. Likewise, `WorkspacePolicy` is path authorization, not a compute sandbox. See [Workspaces and Environments](https://docs.everruns.com/framework/workspaces-and-environments/) for the identity and lifecycle model.
---
# Workspaces and Environments
> Isolate writable project heads, bind them permanently to sessions, and reopen them safely.
Source:
# Optional: Add API keys here to skip UI configuration
DEFAULT_OPENAI_API_KEY=sk-...
DEFAULT_ANTHROPIC_API_KEY=sk-ant-...
DEFAULT_GEMINI_API_KEY=AIza...
```
### 4. Start Services
```bash
docker compose pull # Fetch latest images
docker compose up -d
```
The published compose file defaults to app entry point on `9300`. If that port is busy, override before startup:
```bash
EXAMPLE_PROXY_PORT=10300 docker compose up -d
```
This starts:
* PostgreSQL database
* Control plane API
* 3 worker instances
* Next.js UI
* Caddy reverse proxy
### 5. Access the Platform
| Service | URL |
| ------------------ | -------------------------------------------------------------- |
| **Web UI** | |
| **API** | |
| **OAuth Metadata** | |
| **Health Check** | |
## Configuration
### Run Multiple Copies
If you want multiple Everruns compose stacks on the same machine, set both a Compose project name and host-port overrides:
```bash
COMPOSE_PROJECT_NAME=everruns-demo-2 \
EXAMPLE_PROXY_PORT=10300 \
docker compose up -d
```
### Configure LLM Provider
If you didn’t set LLM API keys (`DEFAULT_OPENAI_API_KEY`, `DEFAULT_ANTHROPIC_API_KEY`, or `DEFAULT_GEMINI_API_KEY`) in your `.env` file, configure via UI:
1. Open
2. Navigate to **Settings** > **Providers**
3. Add your OpenAI or Anthropic API key
4. Save and verify connection
### Create Your First Agent
1. Go to **Agents** in the UI
2. Click **Create Agent**
3. Set a name and system prompt
4. Select your configured LLM provider
5. Save the agent
### Start a Session
```bash
# Create a session (agent_id in request body)
curl -X POST http://localhost:9300/api/v1/sessions \
-H "Content-Type: application/json" \
-d '{"agent_id": "{agent_id}"}'
# Send a message
curl -X POST http://localhost:9300/api/v1/sessions/{session_id}/messages \
-H "Content-Type: application/json" \
-d '{"message": {"role": "user", "content": [{"type": "text", "text": "Hello!"}]}}'
```
## Scaling Workers
Add more workers by scaling the worker services:
```bash
# Scale to 5 workers
docker compose up -d --scale worker-1=1 --scale worker-2=1 --scale worker-3=3
```
Or modify `docker-compose.yaml` to add more worker services.
## Monitoring
### View Logs
```bash
# All services
docker compose logs -f
# Specific service
docker compose logs -f server
docker compose logs -f worker-1
```
### Distributed Tracing
Set `OTEL_EXPORTER_OTLP_ENDPOINT` to export traces to any OTLP-compatible backend (Grafana Tempo, Datadog, etc.).
## Stopping Services
```bash
# Stop all services
docker compose down
# Stop and remove volumes (deletes data)
docker compose down -v
```
## Troubleshooting
### Database Connection Issues
If services fail to connect to PostgreSQL:
```bash
# Check postgres health
docker compose ps postgres
# View postgres logs
docker compose logs postgres
```
### Migration Failures
Migrations are auto-applied when the API server starts. If migrations fail:
```bash
# Check API logs for migration errors
docker compose logs api
# Restart API to retry migrations
docker compose restart api
```
To check migration status before deployment:
```bash
docker compose exec api everruns-admin migrate-info
```
### Worker Not Processing
Verify workers can reach the control plane:
```bash
# Check worker logs
docker compose logs worker-1
# Verify gRPC connection
docker compose exec worker-1 /bin/sh -c "echo" || echo "Cannot exec (distroless image)"
```
## Next Steps
* [API Reference](https://docs.everruns.com/api/) - Full API documentation
* [Capabilities](https://docs.everruns.com/features/capabilities/) - Extend agent functionality
* [Environment Variables](https://docs.everruns.com/sre/environment-variables/) - Advanced configuration
---
# Introduction
> What Everruns is, what it provides, and where to go next.
Source: .txt` — one topic at a time, mirroring the sidebar: [framework](https://docs.everruns.com/_llms-txt/framework.txt), [getting-started](https://docs.everruns.com/_llms-txt/getting-started.txt), [built-ins](https://docs.everruns.com/_llms-txt/built-ins.txt), [guides](https://docs.everruns.com/_llms-txt/guides.txt), [integrations](https://docs.everruns.com/_llms-txt/integrations.txt), [explanation](https://docs.everruns.com/_llms-txt/explanation.txt), [reference](https://docs.everruns.com/_llms-txt/reference.txt), [operations](https://docs.everruns.com/_llms-txt/operations.txt). Prefer a set over the complete text.
* [`/api/openapi.json`](https://docs.everruns.com/api/openapi.json) — the REST API as OpenAPI 3.0. The text sets carry prose only, so take endpoint shapes from here.
Every page in those files begins with its title, its description, and a `Source:` line holding the canonical URL. Cite that URL, not the text file.
---
# How-to guides
> Task-oriented recipes for common Everruns workflows, equipping agents with tools, streaming events, deploying to channels, and operating production agents.
Source: ` positional disables implicit `agent.toml` selection so you can update a different agent from the same directory.
## See also
* [CLI reference](https://docs.everruns.com/features/cli/), full command surface.
* [Equip an agent with tools](https://docs.everruns.com/how-to/equip-agents-with-tools/), choosing capabilities.
* [Use AGENTS.md for project instructions](https://docs.everruns.com/how-to/use-agents-md/), adding per-project guidance.
---
# Enforce a budget
> Cap LLM spend per session or agent with USD, token, or credit-denominated budgets, soft pause thresholds, and stacked limits.
Source: () else { ... };
let store = &store.0;
```
If you implement a custom host, install them the way `everruns-host` does:
```rust
extensions.insert(Arc::new(SessionSqlDbStoreExt(store)));
extensions.insert(Arc::new(SessionMutatorExt(mutator)));
```
## Capabilities and implementations
| what | 0.18 home |
| ------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Knowledge Bases and Indexes, Memories, delegation, subagents, background and scheduled work, user hooks, citations, model scouting, platform management | `everruns_platform::capabilities::` |
| Session info and session storage | `everruns_host::session_services::capabilities::` (also re-exported by platform) |
| Session SQL database and session sandbox | `everruns_platform::capabilities::` |
| `spawn_background` and its runtime, event sink, admission permits, reattach | `everruns_platform::background_run::` |
| Portable built-ins, human intent, infinity context, skills, UI prompts, compaction, tool search | `everruns_builtins::` |
| `everruns_openui::{PromptOptions, default_library, generate_prompt}` | `everruns_builtins::openui::{PromptOptions, default_library, generate_prompt}` |
| `everruns_a2ui::{PromptOptions, default_catalog, generate_prompt}` | `everruns_builtins::a2ui::{PromptOptions, default_catalog, generate_prompt}` |
| OpenRouter workspace, model scout, and provider-executed server tools | `everruns_integrations_openrouter::` |
| Filesystem, shell, web fetch, Lua | `everruns_integrations_*` |
| MCP adapter | `everruns_mcp::` |
| In-process HTTP egress transport | `everruns_host::DirectEgressService` |
| Telemetry init, exporter event listeners, `CompositeEventListener` | `everruns_host::observability::` (feature `observability`) |
| `llmsim` driver, configs, scripted turns, registry helpers, host-builder extension | `everruns_llmsim::` |
| in-memory agentic loop, writable test doubles, fixture capabilities | `everruns_test_support::` |
| `everruns_core::in_memory::{InMemoryAgentStore, InMemoryHarnessStore, InMemorySessionStore, InMemoryProviderStore}` | `everruns_host::{InMemoryAgentStore, InMemoryHarnessStore, InMemorySessionStore, InMemoryProviderStore}` |
| `everruns_core::in_memory::{InMemoryMessageRetriever, InMemoryEventEmitter}` | `everruns_test_support::{InMemoryMessageRetriever, InMemoryEventEmitter}` for isolated deterministic tests |
Product presets compose these explicitly. Core registries are now empty by default: use `everruns_host::runtime_capability_registry()` for the Framework preset or `everruns_platform::capabilities::hosted_capability_registry()` for the hosted product catalog.
Hosted conversation history has no writable message-store replacement. Append canonical events through `everruns_host::EventLog` / `HostEventEmitter` and read messages through `EventHistory`. This avoids message/event dual writes and keeps resume and replay behavior identical across in-memory and durable hosts.
`everruns-test-support` continues to re-export its 0.17 simulator paths during the 0.18 migration, so existing test suites can upgrade without an immediate import rewrite. Treat that as a migration bridge: production code, new tests, and low-level hosts should depend on `everruns-llmsim` directly. The application-facing `everruns::Model::simulated` and `Model::simulated_with_config` APIs are unchanged.
## Features
| 0.17 | 0.18 |
| -------------------------------------- | ------------------------------------------------------------------------- |
| `everruns-core/sqlx` | removed, use `everruns-provider` with `features = ["sqlx"]` |
| `everruns-core/embedded-platform-docs` | removed, it gated nothing; use `everruns-platform/embedded-platform-docs` |
| `everruns-platform/sqlx` | removed, it forwarded to core’s and nothing enabled it |
| `everruns-core/llm-tests` | removed, use the `everruns-llm-tests` package for live provider tests |
`everruns-core` now has an empty default feature set. OpenAPI derives remain available only with `features = ["openapi"]`; structural outlines remain available only with `features = ["tree-sitter-outlines"]`. Neither subtree is present in a default core build.
Concrete provider protocol and utility-model implementations also moved to their effectful owners:
| 0.17 (`everruns_core::`) | 0.18 |
| ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| `OpenAIProtocolChatDriver`, `openai_protocol` | `everruns_provider::` |
| `OpenResponsesProtocolChatDriver`, `openresponses_protocol` | `everruns_provider::` |
| `driver_helpers`, `stream_reconnect` | `everruns_provider::` |
| `OpenAiUtilityLlmService` (now `ProviderUtilityLlmService`), `SystemUtilityLlmConfig`, `UTILITY_OPENAI_API_KEY_ENV` | `everruns_host::` with `features = ["utility-llm"]` |
Core no longer initializes Rustls. Provider HTTP clients install the workspace crypto provider when they are first constructed, while server, worker, and CLI startup owners install it eagerly. Custom binaries that combine TLS stacks can depend on `everruns-provider` with `features = ["tls-aws-lc-rs"]` and call `everruns_provider::install_default_crypto_provider()` once during startup; the call is idempotent and safe under concurrent initialization.
## Provider and typed-ID imports
Provider-owned modules are no longer compatibility-exported by `everruns-core`. Low-level consumers must add `everruns-provider` directly. This keeps credentials and concrete driver assembly out of the neutral kernel and makes the dependency owner visible in `Cargo.toml`.
There are two common compiler-error shapes:
1. **The module moved to another crate.** Add that crate and change the prefix.
2. **The module stayed public, but its root convenience re-export was removed.** Keep the dependency and qualify the symbol through its module.
The second case produces the misleading-looking `no X in the root` error. It does not necessarily mean the type moved. These replacements are deliberately literal so they can be applied with ordinary search-and-replace:
| before | after |
| ---------------------------------- | ---------------------------------------------------- |
| `everruns_core::ProviderStore` | `everruns_core::provider_resolution::ProviderStore` |
| `everruns_core::SessionStore` | `everruns_core::execution_loading::SessionStore` |
| `everruns_core::MessageRetriever` | `everruns_core::message_retriever::MessageRetriever` |
| `everruns_core::SessionFileSystem` | `everruns_core::session_files::SessionFileSystem` |
| 0.17 core path | 0.18 direct path |
| --------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| `everruns_core::driver_registry::*` | `everruns_provider::driver_registry::*` |
| `everruns_core::model::*` | `everruns_provider::model::*` |
| `everruns_core::model_profiles::*` | `everruns_provider::model_profiles::*` |
| `everruns_core::model_spec::ModelSpec` | `everruns_provider::model_spec::ModelSpec` |
| `everruns_core::provider::*` | `everruns_provider::provider::*` |
| `everruns_core::runtime_provider::*` | `everruns_provider::runtime_provider::*` |
| `everruns_core::typed_id::*` | `everruns_provider::typed_id::*` |
| `everruns_core::error::*` | `everruns_provider::error::*` |
| `everruns_core::tool_types::*` | `everruns_provider::tool_types::*` |
| `everruns_core::capability_types::{CapabilityId, CapabilityRef, CapabilityError}` | `everruns_capability::{CapabilityId, CapabilityRef, CapabilityError}` |
| `everruns_core::AgentCapabilityConfig` | `everruns_capability::CapabilityRef` |
| core plugin capability ID/validation helpers | the same symbol in `everruns_capability` |
| `everruns_core::ExecutionPhase` or `message::ExecutionPhase` | `everruns_provider::execution_phase::ExecutionPhase` |
| `everruns_core::ToolResultImage` or `tools::ToolResultImage` | `everruns_provider::tool_types::ToolResultImage` |
| other root-level provider symbols | the same root symbol in `everruns_provider` |
The credential-bearing `everruns_core::ResolvedModel` is removed. Store and transport boundaries now resolve two separate values:
* `ModelSpec`, safe to serialize and pass through the kernel; and
* a host-owned runtime `Provider` (or internal `ProviderConfig`) containing endpoint and authentication state.
`ProviderStore::get_model_spec` and `get_default_model_spec` return only the first value. Hosts obtain provider configuration separately and join it only while constructing a non-serializable driver/provider execution value.
`ProviderStore::get_provider_config` no longer has a default implementation. Every custom host must state where credentials live: return its resolved `ProviderConfig`, or explicitly return `None` when the provider was registered directly in the host registry or is selected but not configured. A missing credential no longer prevents turn-context/command assembly; the constructed driver rejects the first model/list/compact operation locally, before network I/O. This keeps recovery commands reachable without turning an empty token into an outbound authorization header.
## Simulator and compaction naming
`LlmSimRuntimeExt::llm_sim` now only registers/replaces the simulator provider. It never changes the selected model. Existing compact test setups that relied on implicit selection should use the explicit name:
```diff
builder.llm_sim(config)
builder.llm_sim_as_default(config)
```
When a builder already calls `default_model(...)`, keep `.llm_sim(config)`; the selected model is preserved regardless of method order.
`everruns_builtins::CompactionConfig` is the sole application-facing policy builder. The expanded implementation value previously available as `everruns_builtins::compaction::CompactionConfig` is now `everruns_builtins::compaction::RuntimeCompactionConfig` (also re-exported at the crate root). This makes an unqualified `CompactionConfig` unambiguous.
The public core test/backend conveniences are gone as well:
| removed core value | replacement |
| ----------------------------------- | ------------------------------------------------------------------- |
| `EchoTool`, `FailingTool` | define the small test `Tool` locally, or use test-support executors |
| `InMemoryCompactionCheckpointStore` | `everruns_host::InMemoryCompactionCheckpointStore` |
## What deliberately did not move
Worth knowing so you do not go looking:
* **`SessionTask`, `TaskMessage`** and the task registry stay in `everruns_core`. They are turn-execution vocabulary, `wake_queue` decides mid-turn wakes from a task’s wake policy, and they appear in the canonical `task.created` / `task.updated` / `task.message.*` event payloads.
* **`SessionSchedule`, `SessionScheduleStore`** stay. A portable built-in (`usage_limit_auto_continue`) schedules an auto-resume after a provider usage limit, and it sits below platform in the dependency graph.
* Schedule quota and minimum-interval environment variables are no longer read by core. Local/server adapters resolve deployment policy and call the parameterized core validation helpers.
* **`SessionResourceRegistry`** stays. `resource_ownership` and the portable skills capabilities consume it.
* **`SessionFileSystem`, `SessionStorageStore`** and the other neutral store contracts stay. Core owns the contract; hosts own the backend.
The rule these follow: whether something belongs in the kernel is decided by whether a portable execution path consumes it during a turn, not by whether it is persisted. All four above are persisted, and all four are essential for execution.
## Getting unstuck
If a symbol is not in these tables, import it from the crate that defines it; `everruns-core` no longer acts as a compatibility facade for provider-owned APIs. Framework applications can continue to prefer the higher-level `everruns` facade. The crate-level docs on `everruns-core` record where each remaining family lives and why.
---
# Orchestrate multi-agent pipelines
> Chain multiple Everruns agents together by passing output from one session into another.
Source: ` block (\~100 tokens per skill):
```xml
csv-analyzer
Analyze CSV files and generate summary reports.
```
When the user’s task matches, the agent calls `activate_skill`:
```json
{ "name": "activate_skill", "arguments": { "name": "csv-analyzer" } }
```
The tool returns the full SKILL.md instructions wrapped in `` tags. The agent now has the detailed instructions in context and can run the bundled scripts.
## Test the skill
1. Start a session with an agent that has the `skills` capability enabled.
2. Write the skill files to `/.agents/skills//` in the session.
3. Send a message that matches the skill’s “When to Use” criteria.
4. Watch the event stream for an `activate_skill` tool call.
## See also
* [Agent Skills feature](https://docs.everruns.com/features/skills/)
* [Skills Registry](https://docs.everruns.com/features/skills-registry/), share skills across the organization.
* [Publish a skill to the registry](https://docs.everruns.com/how-to/publish-a-skill-to-the-registry/)
---
# Publish a skill to the registry
> Upload a SKILL.md or ZIP archive to the organization-wide Skills Registry so any agent can use it as a capability.
Source: __*` (surfaced through `tool_search`); A2A agents are reachable via `spawn_agent`.
4. `list_attached_resources()`, see what’s attached this session.
## Tools
| Tool | Description |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `discover_resources({ text, filter?, registry_id? })` | Search a configured registry (`POST /search`). Returns ranked `{ urn, displayName, type, score, source, description, attachable }`. `registry_id` is optional when one registry is configured. |
| `attach_resource({ urn })` | Resolve a discovered entry, verify `trustManifest` + `require_trust`, SSRF-validate the URL, and attach it. Idempotent per URN. |
| `list_attached_resources()` | List attachments for the session (visibility / audit). |
## Attachment Lifecycle
* Attachments are **session-scoped** and torn down when the session ends.
* MCP entries become a session-scoped `mcpServers` record; their tools are then subject to `tool_search` deferral.
* A2A entries merge into the session’s A2A delegation config and are driven through the existing `spawn_agent` / `wait_task` / `message_task` tools.
* Re-attaching the same URN is a no-op (reports `already_attached`).
## Security
* **Registry allowlist**: only configured registries are queryable; the model picks a `registry_id`, never a URL.
* **Trust gate**: an entry’s `trustManifest` identity domain must match its URN publisher, and any `require_trust` attestations (e.g. `["soc2"]`) must be present, before it can be attached.
* **SSRF protection**: every resolved artifact and endpoint URL is validated (DNS-pinned; loopback, private, link-local, and cloud-metadata addresses are blocked). `allow_local_urls` relaxes this for local testing only.
* **Attachment cap**: `max_attachments` bounds how many capabilities a single session can attach, limiting prompt-injection-driven attach storms.
* **Untrusted data**: all registry-returned text is treated as untrusted external input.
See the co-located [`SPEC.md`](https://github.com/everruns/everruns/blob/main/crates/ard/SPEC.md) for architecture and the full security review.
---
# Brave Search
> Web search through Brave: ranked results, freshness filters, pagination, source attribution, and safe search. Requires a Brave API key.
Source: .services.ai.azure.com`).
4. Provide credentials for one of the two methods, entered as discrete fields:
* **API key**: your Azure AI Foundry resource key.
* **Entra ID (OAuth)**: a client-credentials service principal: `tenant_id`, `client_id`, and `client_secret` (with optional `scope` and `authority`, which default to the Azure Cognitive Services scope and public Microsoft Entra authority).
5. Save.
Authentication is fail-closed: a stored credential is always required, and OAuth tokens are refreshed transparently for both chat execution and model sync.
## Models
Foundry’s `/models` listing is bare (ids only), so capabilities come from Everruns’ built-in Microsoft MAI model profiles, matched by id. Because Azure deployment names are operator-chosen, a deployment whose name does not match a known profile falls back to a minimal profile.
## Links
* [Azure AI Foundry](https://ai.azure.com/)
* [`everruns-mai` on crates.io](https://crates.io/crates/everruns-mai)
* [Migrate between providers](https://docs.everruns.com/how-to/migrate-providers/)
---
# Meta Model API
> Run Everruns agents on Muse Spark 1.3 through Meta Model API, including the discounted Contributor tier.
Source: `.
* The Resend account must have `everruns.com` verified and enabled for sending.
## UI API Proxy Architecture
The UI makes all REST API requests (including SSE) to `/api/*` paths. The backend serves those routes under `/api` directly. Root-level backend routes like `/oauth/*`, `/mcp`, and `/.well-known/*` bypass the UI and are proxied straight to the backend.
**Local Development:**
* Caddy on `:9300` routes `/api/*`, `/oauth/*`, `/mcp`, and `/.well-known/*` to backend at `:9301`
* Example: `/api/v1/agents` → `http://localhost:9301/api/v1/agents`
* Example: `/oauth/authorize?...` → `http://localhost:9301/oauth/authorize?...`
* Example: `/mcp` → `http://localhost:9301/mcp`
* Example: `/.well-known/oauth-authorization-server` → `http://localhost:9301/.well-known/oauth-authorization-server`
* SSE streaming works via `flush_interval -1` in Caddy config
* No CORS needed (same-origin through Caddy)
**Production:**
* Configure your reverse proxy (nginx, Caddy, etc.) to route `/api/*`, `/oauth/*`, `/mcp`, and `/.well-known/*` to the API server
* Disable response buffering for SSE endpoints
* Example Caddy config: see `local/Caddyfile`
## SSE Streaming Configuration
| Variable | Default | Description |
| ----------------------------- | ------- | ------------------------------------------------------------------ |
| `SSE_REALTIME_CYCLE_SECS` | `300` | Connection cycle interval for session event streams (seconds) |
| `SSE_MONITORING_CYCLE_SECS` | `600` | Connection cycle interval for durable monitoring streams (seconds) |
| `SSE_HEARTBEAT_INTERVAL_SECS` | `30` | Interval between heartbeat comments on all SSE streams (seconds) |
| `SSE_GLOBAL_MAX` | `10000` | Maximum total SSE connections across all users |
| `SSE_PER_SESSION_MAX` | `12` | Maximum SSE connections per session |
| `SSE_PER_ORG_MAX` | `1000` | Maximum SSE connections per organization |
**Notes:**
* Heartbeat comments (`: heartbeat\n\n`) are sent on all SSE streams to detect stale connections
* The heartbeat interval must be less than the SDK read timeout (default: 60s) with safety margin
* Connection cycling prevents stale connections through proxies and load balancers
* When running behind HTTP/1.1 proxies, increase `SSE_REALTIME_CYCLE_SECS` to reduce reconnection frequency
## Worker gRPC Configuration
### SERVER\_GRPC\_ADDRESS
Address of the server gRPC endpoint for worker communication.
| Property | Value |
| ------------ | ---------------- |
| **Required** | No (worker only) |
| **Default** | `127.0.0.1:9001` |
**Example:**
```bash
SERVER_GRPC_ADDRESS=127.0.0.1:9001
```
**Notes:**
* Workers communicate with the server via gRPC for all database operations
* `WORKER_GRPC_ADDRESS` is supported as a legacy alias
* The server exposes both HTTP (default `9000`) and gRPC (default `9001`) interfaces
* Workers are stateless and do not connect directly to the database
### WORKER\_GRPC\_AUTH\_TOKEN
Bearer token for authenticating worker gRPC connections to the control-plane.
| Property | Value |
| ------------ | ------------------------------- |
| **Required** | Yes (production); No (dev mode) |
| **Default** | Unset (auth disabled) |
**Example:**
```bash
WORKER_GRPC_AUTH_TOKEN=your-secret-token
```
**Notes:**
* Must be set on both the server and all workers (same value)
* When unset, gRPC auth is disabled (acceptable for local development only)
* Server panics on startup if unset when not in dev mode
### SERVER\_GRPC\_BIND\_ADDR
Bind address for the server-side gRPC listener.
| Property | Value |
| ------------ | ---------------- |
| **Required** | No (server only) |
| **Default** | `0.0.0.0:9001` |
**Example:**
```bash
SERVER_GRPC_BIND_ADDR=0.0.0.0:9001
```
**Notes:**
* `WORKER_GRPC_ADDR` is supported as a legacy alias
### WORKER\_GRPC\_CONNECT\_TIMEOUT
Timeout in seconds for worker initial connection to control-plane gRPC.
| Property | Value |
| ------------ | ---------------- |
| **Required** | No (worker only) |
| **Default** | `30` |
**Example:**
```bash
WORKER_GRPC_CONNECT_TIMEOUT=60
```
### WORKER\_GRPC\_TLS\_CERT
Path to PEM-encoded certificate file. On the server, this is the gRPC server certificate. On the worker, this is the client certificate presented during mTLS handshake.
| Property | Value |
| ------------ | ---------------------- |
| **Required** | No |
| **Default** | Not set (TLS disabled) |
**Example:**
```bash
WORKER_GRPC_TLS_CERT=/etc/everruns/grpc-cert.pem
```
**Notes:**
* Must be set together with `WORKER_GRPC_TLS_KEY`
* Server: enables TLS on the gRPC listener when both cert and key are set
* Worker: presents client certificate to the server when both cert and key are set (requires `WORKER_GRPC_TLS_CA_CERT`)
### WORKER\_GRPC\_TLS\_KEY
Path to PEM-encoded private key file corresponding to `WORKER_GRPC_TLS_CERT`.
| Property | Value |
| ------------ | ------- |
| **Required** | No |
| **Default** | Not set |
**Example:**
```bash
WORKER_GRPC_TLS_KEY=/etc/everruns/grpc-key.pem
```
### WORKER\_GRPC\_TLS\_CA\_CERT
Path to PEM-encoded CA certificate bundle for verifying the remote peer.
| Property | Value |
| ------------ | ------- |
| **Required** | No |
| **Default** | Not set |
**Example:**
```bash
WORKER_GRPC_TLS_CA_CERT=/etc/everruns/grpc-ca.pem
```
**Notes:**
* Server: when set, requires workers to present valid client certificates signed by this CA (mutual TLS)
* Worker: when set, verifies the server’s certificate against this CA and switches to `https://` transport
* For full mTLS, set on both server and worker alongside their respective cert/key pairs
### WORKER\_GRPC\_TLS\_DOMAIN
Override the expected server domain name for TLS certificate verification (worker only).
| Property | Value |
| ------------ | ------------------------------------------- |
| **Required** | No |
| **Default** | Derived from `SERVER_GRPC_ADDRESS` hostname |
**Example:**
```bash
WORKER_GRPC_TLS_DOMAIN=control-plane.internal
```
**Notes:**
* Useful when the server certificate CN/SAN differs from the connection hostname (e.g., connecting via IP but cert has a DNS name)
## OpenTelemetry Configuration
Everruns supports distributed tracing via OpenTelemetry with OTLP export. Agent traces follow the OpenTelemetry [Gen-AI agent and inference conventions](https://github.com/open-telemetry/semantic-conventions-genai/tree/main/docs/gen-ai) and, on the same spans, the [OpenInference conventions](https://arize-ai.github.io/openinference/spec/semantic_conventions.html) read by Arize Phoenix, so one OTLP endpoint serves both families of backends.
### OTEL\_EXPORTER\_OTLP\_ENDPOINT
OTLP endpoint for trace export (e.g., Grafana Tempo, Arize Phoenix, Datadog, or any OTLP-compatible backend).
| Property | Value |
| ------------ | -------------------------- |
| **Required** | No |
| **Default** | Not set (tracing disabled) |
**Example:**
```bash
# For a local OTLP collector or Phoenix
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# For production Tempo
OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.monitoring:4318
```
**Notes:**
* When set, traces are exported via OTLP over HTTP/protobuf (use the backend’s HTTP port, typically 4318, not the gRPC port 4317)
* Point this at the base endpoint; the `/v1/traces` path is appended for you, and a full signal URL is used as given
* Connect to any OTLP-compatible backend for trace visualization
* See [OpenTelemetry](https://docs.everruns.com/observability/opentelemetry/) for the span model and attributes
* Without this variable, only console logging is enabled
### OTEL\_SERVICE\_NAME
Service name for traces.
| Property | Value |
| ------------ | --------------------------------------------------- |
| **Required** | No |
| **Default** | `everruns-server` (API), `everruns-worker` (Worker) |
**Example:**
```bash
OTEL_SERVICE_NAME=everruns-prod-api
```
### OTEL\_SERVICE\_VERSION
Service version for traces.
| Property | Value |
| ------------ | --------------------- |
| **Required** | No |
| **Default** | Cargo package version |
### OTEL\_ENVIRONMENT
Deployment environment label.
| Property | Value |
| ------------ | ------- |
| **Required** | No |
| **Default** | Not set |
**Example:**
```bash
OTEL_ENVIRONMENT=production
```
### OTEL\_RECORD\_CONTENT
Enable recording of LLM input/output content in traces. **Warning:** May contain sensitive data.
| Property | Value |
| ------------ | ------- |
| **Required** | No |
| **Default** | `false` |
**Example:**
```bash
# Standard OTel env var (preferred)
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
# Legacy alias (also works)
OTEL_RECORD_CONTENT=true
```
**Notes:**
* When enabled, the chat span records `gen_ai.system_instructions`, `gen_ai.input.messages`, `gen_ai.output.messages`, and `gen_ai.tool.definitions` (plus the OpenInference `input.value`, `output.value`, and flattened `llm.input_messages.*`); tool spans record `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result`; the turn root records the input message and final answer; the thinking span records the reasoning text
* Disabled by default for privacy and data size concerns
* Only enable in development or when debugging specific issues
### EVERRUNS\_TRACE\_CONVENTIONS
Which attribute vocabularies agent spans carry.
| Property | Value |
| ------------ | ---------------------- |
| **Required** | No |
| **Default** | `gen_ai,openinference` |
**Example:**
```bash
# Only the OpenTelemetry Gen-AI attributes (Tempo, Jaeger, Datadog, Langfuse)
EVERRUNS_TRACE_CONVENTIONS=gen_ai
# Only the OpenInference attributes (Arize Phoenix)
EVERRUNS_TRACE_CONVENTIONS=openinference
```
**Notes:**
* Span names, kinds, and hierarchy are the same under either vocabulary; only attributes differ
* Unknown values are ignored, and an empty selection falls back to both
## Local Development with OpenTelemetry
To visualize traces locally, point `OTEL_EXPORTER_OTLP_ENDPOINT` at any OTLP-compatible collector:
```bash
# Set OTLP endpoint for API and Worker
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# Start services
just start-all
```
To see traces in Arize Phoenix, run Phoenix locally and point the same variable at its OTLP/HTTP port (`http://localhost:6006`); spans render as AGENT, LLM, and TOOL spans with token counts and, when content capture is on, the messages.
### Gen-AI Trace Structure
Traces follow the agentic execution lifecycle with 13 event types; every span starts and ends at the timestamp of the event it records:
```plaintext
invoke_agent {agent name} (root span, INTERNAL)
├── reason (LLM reasoning phase)
│ └── chat {model} (LLM API call, CLIENT)
│ └── thinking (extended thinking, if enabled)
├── act (tool execution phase)
│ ├── execute_tool {name}
│ └── execute_tool {name}
├── reason (iteration 2)
│ └── chat {model}
└── ...
```
### Gen-AI Trace Attributes
Spans carry the OpenTelemetry Gen-AI attributes and the OpenInference attributes side by side (see `EVERRUNS_TRACE_CONVENTIONS`). The most useful ones:
| Attribute | Span Types | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `gen_ai.operation.name` | invoke\_agent, chat, execute\_tool | `invoke_agent`, `chat`, or `execute_tool` |
| `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.agent.description` | invoke\_agent | Agent the turn runs as |
| `gen_ai.conversation.id` / `session.id` | All | Session identifier |
| `gen_ai.provider.name` / `llm.provider` | chat | Provider (`openai`, `anthropic`, `gcp.gemini`, `aws.bedrock`, …) |
| `gen_ai.request.model`, `gen_ai.response.model` / `llm.model_name` | chat | Model name |
| `gen_ai.response.id` | chat | Provider response identifier |
| `gen_ai.response.finish_reasons` | chat | Why generation stopped (string array) |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` / `llm.token_count.*` | chat, invoke\_agent | Token usage (turn root carries the cumulative total) |
| `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_write.input_tokens` | chat, invoke\_agent | Prompt-cache tokens |
| `gen_ai.request.temperature`, `gen_ai.request.max_tokens`, `gen_ai.request.reasoning.level`, `gen_ai.request.stream` / `llm.invocation_parameters` | chat | Request parameters |
| `gen_ai.response.time_to_first_chunk` | chat | Streaming latency in seconds |
| `gen_ai.conversation.compacted` | chat | Context was compacted before the call |
| `llm.cost.total` / `everruns.usage.cost_usd` | chat, invoke\_agent | Cost in USD when known |
| `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.type`, `gen_ai.tool.description` / `tool.name`, `tool.description` | execute\_tool | Tool identity |
| `openinference.span.kind` | All | `AGENT`, `LLM`, `TOOL`, or `CHAIN` |
| `error.type` | All | Low-cardinality error class on failure (error code, HTTP status, `timeout`, or `_OTHER`); the span status carries the message |
| `everruns.phase` | reason, act, thinking | Phase span marker |
| `everruns.turn.id`, `everruns.exec.id` | All | Everruns correlation ids |
| `everruns.turn.iterations`, `everruns.turn.tool_call_count`, `everruns.turn.llm_call_count` | invoke\_agent | Turn counters |
| `everruns.tool.status` | execute\_tool | `success`, `error`, `timeout`, or `cancelled` |
## Braintrust Integration
Everruns supports sending turn, reasoning, tool, and session lifecycle events to [Braintrust](https://www.braintrust.dev/) for observability, evaluation, and logging.
For setup instructions and configuration details, see the [Braintrust Integration Guide](https://docs.everruns.com/observability/braintrust/).
| Variable | Required | Default | Description |
| -------------------------------- | -------- | ------------------------------- | --------------------------------------------------------------- |
| `BRAINTRUST_ENABLED` | No | enabled when API key is present | Explicit Braintrust on/off switch |
| `BRAINTRUST_API_KEY` | Yes | - | API key from Braintrust settings |
| `BRAINTRUST_PROJECT_NAME` | No | `My Project` | Project name for organizing traces |
| `BRAINTRUST_PROJECT_ID` | No | - | Direct project UUID (skips name lookup) |
| `BRAINTRUST_API_URL` | No | `https://api.braintrust.dev` | API base URL |
| `BRAINTRUST_QUEUE_CAPACITY` | No | `1024` | Buffered event capacity before new exports are dropped |
| `BRAINTRUST_MAX_BATCH_SIZE` | No | `50` | Max events per Braintrust insert call |
| `BRAINTRUST_FLUSH_INTERVAL_MS` | No | `500` | Max delay before flushing a partial batch |
| `BRAINTRUST_REQUEST_TIMEOUT_MS` | No | `10000` | Per-request timeout for Braintrust insert calls |
| `BRAINTRUST_MAX_RETRIES` | No | `3` | Retries for `429`, `5xx`, and timeout/connect failures |
| `BRAINTRUST_RETRY_BASE_DELAY_MS` | No | `250` | Initial retry backoff |
| `BRAINTRUST_RETRY_MAX_DELAY_MS` | No | `5000` | Retry backoff cap |
| `BRAINTRUST_RECORD_CONTENT` | No | `false` | Export raw turn and LLM text content |
| `BRAINTRUST_RECORD_THINKING` | No | `none` | Extended thinking export mode: `none`, `summary`, `full` |
| `BRAINTRUST_TOOL_ARGS_MODE` | No | `redacted` | Tool argument export mode: `full`, `redacted`, `none` |
| `BRAINTRUST_TOOL_RESULTS_MODE` | No | `summary` | Tool result export mode: `full`, `summary`, `redacted`, `none` |
| `BRAINTRUST_DEBUG_PAYLOADS` | No | `false` | Print full outbound Braintrust payload JSON to local debug logs |
---
# Authentication Configuration Runbook
> Configure authentication modes, personal access tokens, OAuth providers, JWT secrets, token lifetimes, and sign-up controls.
Source: ';
```
## Implementation Status
| Phase | Status | Description |
| --------- | --------------- | -------------------------------------------------------- |
| Phase 1-4 | ✅ Complete | Core abstractions, persistence, reliability, worker pool |
| Phase 5 | 🔄 Planned | Observability & Metrics (OpenTelemetry integration) |
| Phase 6 | 🔄 Planned | Scale Testing (1000+ concurrent workers) |
| Phase 7 | ✅ Core Complete | gRPC-based worker integration, crash recovery |
The durable execution engine is production-ready for single-instance deployments.
---
# Encryption Key Rotation
> Rotate the secrets encryption key (KEK): key deployment, data re-encryption, and old key removal.
Source: Process N records at a time (default: 100)
-t, --table Only process specified table (default: all)
-h, --help Show this help message
```
### Phase 4: Verify Migration
Confirm all data has been migrated by running another dry run:
```bash
docker run --rm \
-e DATABASE_URL="$DATABASE_URL" \
-e SECRETS_ENCRYPTION_KEY="kek-v2:..." \
-e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \
everruns-admin reencrypt --dry-run
```
Expected output:
```plaintext
2024-01-15T11:00:00Z INFO DRY RUN: Would re-encrypt 0 of 100 records
```
You can also verify directly in the database:
```sql
-- Check for any remaining records with old key
SELECT COUNT(*)
FROM llm_providers
WHERE api_key_encrypted::text LIKE '%"key_id":"kek-v1"%';
-- Should return 0
```
### Phase 5: Remove Old Key
Once verified, remove the old key from configuration:
```bash
# Remove the previous key
SECRETS_ENCRYPTION_KEY=kek-v2:xR7qW2mN9pL4kJ8vB3tY6fE1hG5sD0cA9uI7oP2nM6w=
# SECRETS_ENCRYPTION_KEY_PREVIOUS= (remove or leave empty)
```
Deploy the updated configuration.
**Important**: Keep the old key archived securely for disaster recovery. You may need it if backup restoration is required.
## Rollback Procedure
If issues occur during rotation:
### During Phase 2-3 (Both Keys Active)
No rollback needed - both keys work. Simply stop the re-encryption CLI if causing issues.
### After Phase 5 (Old Key Removed)
If old key was removed but some data wasn’t migrated:
1. Re-add the old key as `SECRETS_ENCRYPTION_KEY_PREVIOUS`
2. Deploy
3. Run the re-encryption CLI again
4. Verify again before removing
## Monitoring
During rotation, monitor:
* **CLI Progress Output**: The tool logs progress every 1000 records
* **API Error Rates**: Watch for decryption failures in application logs
* **Database Load**: Ensure re-encryption isn’t causing performance issues
## Emergency: Compromised Key
If a key is suspected compromised:
1. **Immediately** generate new key and deploy with both keys
2. Run re-encryption CLI with **highest priority**:
```bash
docker run --rm \
-e DATABASE_URL="$DATABASE_URL" \
-e SECRETS_ENCRYPTION_KEY="kek-v2:..." \
-e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \
everruns-admin reencrypt
```
3. Remove compromised key as soon as all data is migrated
4. Rotate any credentials that may have been exposed
## Key Storage Best Practices
* Store keys in a secrets manager (AWS Secrets Manager, HashiCorp Vault, etc.)
* Enable audit logging for key access
* Rotate keys on a regular schedule (e.g., annually)
* Keep previous key archived for disaster recovery (separate secure storage)
* Never commit keys to source control
---
# Tutorial: Build your first agent
> Create an Everruns agent in Python, send a message, and stream the response.
Source: