Create a new eval.
const url = 'https://app.everruns.com/api/v1/evals';const options = { method: 'POST', headers: {'Content-Type': 'application/json'}, body: '{"description":"Regression suite for the support agent","model_override":"gpt-5.1","name":"Support agent regression","tags":["regression","nightly"],"target":{"agent_id":"example","harness_id":"example","harness_name":"example","max_iterations":1,"model_id":"example","system_prompt":"example","type":"session"}}'};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request POST \ --url https://app.everruns.com/api/v1/evals \ --header 'Content-Type: application/json' \ --data '{ "description": "Regression suite for the support agent", "model_override": "gpt-5.1", "name": "Support agent regression", "tags": [ "regression", "nightly" ], "target": { "agent_id": "example", "harness_id": "example", "harness_name": "example", "max_iterations": 1, "model_id": "example", "system_prompt": "example", "type": "session" } }'Request Bodyrequired
Section titled “Request Bodyrequired”Request to create a new eval
object
Human-readable description. Safe to render in user-facing messages.
Example
Regression suite for the support agentDefault model override applied to runs of this eval.
Example
gpt-5.1Human-readable name. Safe to render in user-facing messages.
Example
Support agent regressionFree-form tags attached to this resource.
Example
[ "regression", "nightly"]Session creation parameters (mirrors CreateSessionRequest).
object
Agent to work in this session.
Harness for the session. If omitted, org default harness is used.
Addressable harness name (alternative to harness_id).
Max LLM iterations per turn.
LLM model override.
System prompt override (prepended to agent prompt).
Reference to a deployed app.
object
Label-only target for externally-executed runs (e.g. imported from Mira).
Carries provider/model labels and opaque params instead of session setup:
external runs are ingested already-complete, so everruns never builds a
session from this. Mirrors a provider-agnostic (provider, model) pair.
object
Responses
Section titled “ Responses ”Created
An eval: a named collection of test cases for an agent.
object
When the eval was archived, if archived.
Number of cases.
When the eval was created.
When the eval was deleted, if deleted.
Optional description.
External identifier (eval_<32-hex>). Shown as “id” in API.
Last run summary (if any).
object
When the run was created.
Eval run identifier.
Run lifecycle status.
Aggregate metrics, if the run has completed.
object
Mean case latency in milliseconds.
Mean score across cases, 0.0 to 1.0.
Mean agent turns per case.
Number of cases that errored.
Number of cases that failed.
Fraction of cases that passed, 0.0 to 1.0.
Number of cases that passed.
Total number of cases in the run.
Total input tokens across cases.
Total output tokens across cases.
Optional default model override for runs.
Display name.
Lifecycle status.
Organization tags.
Session creation parameters (mirrors CreateSessionRequest).
object
Agent to work in this session.
Harness for the session. If omitted, org default harness is used.
Addressable harness name (alternative to harness_id).
Max LLM iterations per turn.
LLM model override.
System prompt override (prepended to agent prompt).
Reference to a deployed app.
object
Label-only target for externally-executed runs (e.g. imported from Mira).
Carries provider/model labels and opaque params instead of session setup:
external runs are ingested already-complete, so everruns never builds a
session from this. Mirrors a provider-agnostic (provider, model) pair.
object
When the eval was last updated.
Example
{ "archived_at": "2026-01-15T10:30:00Z", "case_count": 12, "created_at": "2026-01-15T10:30:00Z", "deleted_at": "2026-01-15T10:30:00Z", "description": "Regression suite for the support agent", "id": "eval_01933b5a000070008000000000000001", "last_run": { "created_at": "2026-01-15T10:30:00Z", "status": "pending", "summary": { "avg_latency_ms": 8450, "avg_score": 0.85, "avg_turns": 3.2, "errored": 1, "failed": 2, "pass_rate": 0.75, "passed": 9, "total": 12, "total_input_tokens": 14400, "total_output_tokens": 10200 } }, "model_override": "gpt-5.1", "name": "Support agent regression", "status": "active", "tags": [ "regression", "nightly" ], "target": { "type": "session" }, "updated_at": "2026-01-15T10:30:00Z"}