Create an observer (online scoring).

POST

/v1/observers

const url = 'https://app.everruns.com/api/v1/observers';
const options = {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: '{"description":"Score replies from the support agent","match":{"agent_ids":["example"],"harness_ids":["example"],"session_tags":["example"]},"name":"Support quality","sampling_rate":0.25,"scorers":[{"model_id":"example","pass_threshold":1,"rubric":"example","method":"llm_judge","key":"example","scope":"turn"}]}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

curl --request POST \
  --url https://app.everruns.com/api/v1/observers \
  --header 'Content-Type: application/json' \
  --data '{ "description": "Score replies from the support agent", "match": { "agent_ids": [ "example" ], "harness_ids": [ "example" ], "session_tags": [ "example" ] }, "name": "Support quality", "sampling_rate": 0.25, "scorers": [ { "model_id": "example", "pass_threshold": 1, "rubric": "example", "method": "llm_judge", "key": "example", "scope": "turn" } ] }'

Request Body^required

application/json

Request to create a new observer.

object

description

Human-readable description. Safe to render in user-facing messages.

string | null

Example

Score replies from the support agent

match

One of:

object
null

Which production sessions to score. Empty matches all org traffic.

object

agent_ids

Match sessions running any of these agents.

Array<string> | null

harness_ids

Match sessions on any of these harnesses.

Array<string> | null

session_tags

Match sessions carrying any of these tags.

Array<string> | null

name

required

Human-readable name. Safe to render in user-facing messages.

string

Example

Support quality

sampling_rate

Fraction of matching turns to score (0.0–1.0). Defaults to 0.1.

number | null format: double

Example

0.25

scorers

required

Scoring rules. Must contain at least one.

Array

One scorer inside an observer. key names the score series in listings and future dashboards; scope selects the trace slice; method is how it grades.

object

One of:

object
object

Deterministic rule. file_contains is rejected for observers (session filesystems are not part of the observable trace contract).

object

method

required

string

Allowed values: rule

rule

required

One of:

Final assistant message contains substring.

object

text

required

string

type

required

string

Allowed values: contains

weight

number format: double

Agent called named tool at least min times.

object

min

integer format: int32

tool

required

string

type

required

string

Allowed values: tool_called

weight

number format: double

Citation faithfulness: the answer’s citations must cover the claim and be verified as supported. Scored from the TextAnnotations on the final message — pair with the citation_verification capability so verdicts are present.

object

min_citations

Minimum number of citations the answer must carry.

integer format: int32

Example

pass_threshold

Minimum fraction of citations verified entailed to pass.

number format: double

Example

0.8

type

required

string

Allowed values: citation_faithful

weight

Relative weight of this scorer in the case’s weighted average.

number format: double

Example

Citation faithfulness judged by an LLM: each cited claim/source pair is graded by a model, so the eval works even without the citation_verification capability.

object

model_id

One of:

string
null

Judge model; the org’s default is used when absent.

string

/^model_[0-9a-f]{32}$/

Example

model_01933b5a000070008000000000000001

pass_threshold

Minimum judged score [0,1] to pass.

number format: double

Example

0.8

rubric

Rubric override; a citation-faithfulness rubric is used when absent.

string | null

Example

Score the fraction of cited claims supported by their source.

type

required

string

Allowed values: citation_judged

weight

Relative weight of this scorer in the case’s weighted average.

number format: double

Example

LLM-as-judge.

object

model_id

Org model to judge with. When None, the org’s default model is used. Judge calls go through the org’s own providers and are billed to it.

string | null

pass_threshold

Score value at/above which pass is true.

number format: double

rubric

required

Grading rubric shown to the judge model. Should describe what a high vs. low score means.

string

method

required

string

Allowed values: llm_judge

key

required

Stable name within the observer (score series name).

string

scope

Trace slice this scorer grades.

string

Allowed values: turn

Responses

201

Created

application/json

An observer: online scoring config over production sessions.

object

archived_at

string | null format: date-time

created_at

required

string format: date-time

description

Optional description.

string | null

required

External identifier (observer_<32-hex>). Shown as “id” in API.

string

match

Which sessions to score.

object

agent_ids

Match sessions running any of these agents.

Array<string> | null

harness_ids

Match sessions on any of these harnesses.

Array<string> | null

session_tags

Match sessions carrying any of these tags.

Array<string> | null

name

required

Display name.

string

sampling_rate

required

Fraction of matching turns to score (0.0–1.0), applied after match.

number format: double

scorers

required

Scoring rules.

Array

One scorer inside an observer. key names the score series in listings and future dashboards; scope selects the trace slice; method is how it grades.

object

One of:

object
object

Deterministic rule. file_contains is rejected for observers (session filesystems are not part of the observable trace contract).

object

method

required

string

Allowed values: rule

rule

required

One of:

Final assistant message contains substring.

object

text

required

string

type

required

string

Allowed values: contains

weight

number format: double

Agent called named tool at least min times.

object

min

integer format: int32

tool

required

string

type

required

string

Allowed values: tool_called

weight

number format: double

object

min_citations

Minimum number of citations the answer must carry.

integer format: int32

pass_threshold

Minimum fraction of citations verified entailed to pass.

number format: double

type

required

string

Allowed values: citation_faithful

weight

Relative weight of this scorer in the case’s weighted average.

number format: double

Citation faithfulness judged by an LLM: each cited claim/source pair is graded by a model, so the eval works even without the citation_verification capability.

object

model_id

One of:

string
null

Judge model; the org’s default is used when absent.

string

/^model_[0-9a-f]{32}$/

pass_threshold

Minimum judged score [0,1] to pass.

number format: double

rubric

Rubric override; a citation-faithfulness rubric is used when absent.

string | null

type

required

string

Allowed values: citation_judged

weight

Relative weight of this scorer in the case’s weighted average.

number format: double

LLM-as-judge.

object

model_id

Org model to judge with. When None, the org’s default model is used. Judge calls go through the org’s own providers and are billed to it.

string | null

pass_threshold

Score value at/above which pass is true.

number format: double

rubric

required

Grading rubric shown to the judge model. Should describe what a high vs. low score means.

string

method

required

string

Allowed values: llm_judge

key

required

Stable name within the observer (score series name).

string

scope

Trace slice this scorer grades.

string

Allowed values: turn

status

required

Lifecycle status.

string

Allowed values: active paused archived deleted

updated_at

required

string format: date-time

Example

{
  "description": "Grades support replies for grounded answers.",
  "id": "observer_01933b5a000070008000000000000001",
  "name": "Support answer quality",
  "sampling_rate": 0.1,
  "scorers": [
    {
      "method": "llm_judge",
      "scope": "turn"
    }
  ],
  "status": "active"
}

Create an observer (online scoring).

Request Bodyrequired

Example

Example

Example

Example

Example

Example

Example

Example

Example

Example

Responses

201

Example

Request Body^required