Evaluation

Evaluation is how you answer one question: is this version good enough to ship? You assemble a plan of checks, run it as an experiment over a dataset, and read the results. Nothing gets deployed on a hunch.

Checks

A check is one atomic measurement over a function’s output — for example a schema-conformance check, a judge scoring relevance, or a statistical measure of consistency across repeats. Each check in the catalogue is tagged with what it needs (expected values, multiple repeats, etc.) and whether its engine is runnable yet.

Checks come in a few flavours you’ll see across the UI:

• Deterministic — exact/structural measures (schema valid, field present).
• Judge — an LLM scores the output against a rubric (relevance, grounding, faithfulness, holistic quality).
• Statistical — measures computed across repeats/rows (consistency, variance).
• Human — a person labels or corrects outputs.

Plans

A plan is a named, editable collection of checks, scoped to one function. You can author a plan by hand from the catalogue, or have the evaluation planner propose one from the version’s contract. Plans are reusable — the same plan runs against many versions so comparisons stay apples-to-apples.

Datasets

A dataset is a set of sample inputs X, optionally with expected outputs Y, scoped to a space. Datasets can be uploaded, built by hand, or grown from production traces you flag as interesting (see deployment).

Experiments

An experiment is the unit that actually runs:

Experiment = Plan × Version × Dataset × models × repeats

It executes every check in the plan against every row, for each model and each repeat. That produces a result per run and a comparison grain that lets you rank versions and models against each other. Repeats matter because model output varies — running each input several times exposes consistency, not just a lucky single sample.

Reading results

Results tell you two things: whether the version clears your quality bar, and where it fails when it doesn’t. Comparisons put versions and models side by side so you can pick the cheapest option that still passes, or catch a regression against a previous baseline before it ships.

Gating a deploy on quality

Evaluation is the second of the two gates. The design gate keeps a version from being created if it breaks its own contract; evaluation is where you decide it’s good enough to approve and deploy. Only an approved version can go live — see the deployment guide.

Next

• Deployment — promote an evaluated version and watch it in production.
• API reference — the plan, experiment, and result endpoints.