Eval commands

eval run

Runs the project’s eval suite and prints per-case results. By default, the CLI uploads the local ontology as ephemeral run input; it does not publish or replace the project’s ontology. Pass --branch to score an existing Cassis branch instead, in which case local files are not uploaded.

# Score the local files and wait for results
cassis eval run --project ...

# Run only specific cases (repeatable)
cassis eval run --project ... --case 019f0000-... --case 019f0001-...

# Run against an existing Cassis branch instead of the local ontology
cassis eval run --project ... --branch feature-x

# Start the run and return immediately, poll in the web app
cassis eval run --project ... --no-wait

Subset

--case <id> (repeatable) runs only the named cases. Use it to prove one freshly added case in seconds instead of rerunning everything. Ids come from list-cases.

Run input

Omit --branch to upload and score the local ontology without publishing it. --branch <name> selects an existing Cassis branch and ignores the local tree.

Run label

Local runs are labeled with your git branch name on the Evals page, taken from the CI environment or checkout. --label overrides it, and is rejected with --branch, whose runs use the Cassis branch name.

Waiting

--wait or --no-wait, --poll-interval (5 seconds), --timeout (30 minutes). The run keeps going server-side if the CLI stops waiting. Ctrl-C cancels the run and exits 130.

Deep link

It prints a link to the run’s page in the Evals UI. --app-url or CASSIS_APP_URL overrides the base URL when the web app is not served from the API host.

eval add-case

Adds an approved question and SQL case to the project’s suite. Before running it, have a domain owner or reviewer validate the SQL and the business assumptions it encodes. Executable SQL alone does not establish correct business meaning.

cassis eval add-case --project ... \
  -q "How much was refunded last month?" \
  --gold-sql "SELECT SUM(refunded_cents) / 100.0 FROM public.orders WHERE ..."

# Multi-line SQL from a file, which avoids shell quoting issues
cassis eval add-case --project ... \
  -q "Revenue by region last quarter" \
  --gold-sql-file gold.sql

The gold SQL is run first

On an executable data source, Cassis runs your SQL before storing the case, so a case that cannot execute never enters the suite. This catches execution errors, not incorrect business interpretation.

What it stores

The question plus the gold SQL. On a warehouse-connected project, both generated and gold queries run and their rows are compared deterministically with fuzzy numeric matching. On a schema-only project, an LLM judge compares their SQL logic.

Rejections

Exit 1 on a duplicate question or on gold SQL that does not run. Both print the reason; nothing is added.

No checkout needed

It talks to the project, not to your files, so it takes no path and no --base-path.

eval list-cases

Lists the project’s cases, one id and question per line. With --json each case also carries its gold SQL. The ids are what delete-case takes.

cassis eval list-cases --project ...

# Include the gold SQL
cassis eval list-cases --project ... --json

eval delete-case

Deletes one case by the id list-cases prints. Use it to prune a case that is stale or wrong, for instance gold SQL encoding a definition the ontology has since changed.

cassis eval delete-case 019f0000-0000-7000-8000-000000000000 --project ...

Exit 1 when the project has no case with that id; nothing is deleted.