# Eval commands Score an uploaded local ontology or an existing Cassis branch against the project's gold cases, and maintain those cases. ## eval run Runs the project’s eval suite and prints per-case results. By default, the CLI uploads the local ontology as ephemeral run input; it does not publish or replace the project’s ontology. Pass `--branch` to score an existing Cassis branch instead, in which case local files are not uploaded. ```bash # Score the local files and wait for results cassis eval run --project ... # Run only specific cases (repeatable) cassis eval run --project ... --case 019f0000-... --case 019f0001-... # Run against an existing Cassis branch instead of the local ontology cassis eval run --project ... --branch feature-x # Start the run and return immediately, poll in the web app cassis eval run --project ... --no-wait ``` - **Subset** `--case ` (repeatable) runs only the named cases. Use it to prove one freshly added case in seconds instead of rerunning everything. Ids come from `list-cases`. - **Run input** Omit `--branch` to upload and score the local ontology without publishing it. `--branch ` selects an existing Cassis branch and ignores the local tree. - **Run label** Local runs are labeled with your git branch name on the [Evals page](/curate/evals/), taken from the CI environment or checkout. `--label` overrides it, and is rejected with `--branch`, whose runs use the Cassis branch name. - **Waiting** `--wait` or `--no-wait`, `--poll-interval` (5 seconds), `--timeout` (30 minutes). The run keeps going server-side if the CLI stops waiting. Ctrl-C cancels the run and exits 130. - **Deep link** It prints a link to the run’s page in the Evals UI. `--app-url` or `CASSIS_APP_URL` overrides the base URL when the web app is not served from the API host. ## eval add-case Adds an approved question and SQL case to the project’s suite. Before running it, have a domain owner or reviewer validate the SQL and the business assumptions it encodes. Executable SQL alone does not establish correct business meaning. ```bash cassis eval add-case --project ... \ -q "How much was refunded last month?" \ --gold-sql "SELECT SUM(refunded_cents) / 100.0 FROM public.orders WHERE ..." # Multi-line SQL from a file, which avoids shell quoting issues cassis eval add-case --project ... \ -q "Revenue by region last quarter" \ --gold-sql-file gold.sql ``` - **The gold SQL is run first** On an executable data source, Cassis runs your SQL before storing the case, so a case that cannot execute never enters the suite. This catches execution errors, not incorrect business interpretation. - **What it stores** The question plus the gold SQL. On a warehouse-connected project, both generated and gold queries run and their rows are compared deterministically with fuzzy numeric matching. On a schema-only project, an LLM judge compares their SQL logic. - **Rejections** Exit 1 on a duplicate question or on gold SQL that does not run. Both print the reason; nothing is added. - **No checkout needed** It talks to the project, not to your files, so it takes no path and no `--base-path`. ## eval list-cases Lists the project’s cases, one id and question per line. With `--json` each case also carries its gold SQL. The ids are what `delete-case` takes. ```bash cassis eval list-cases --project ... # Include the gold SQL cassis eval list-cases --project ... --json ``` ## eval delete-case Deletes one case by the id `list-cases` prints. Use it to prune a case that is stale or wrong, for instance gold SQL encoding a definition the ontology has since changed. ```bash cassis eval delete-case 019f0000-0000-7000-8000-000000000000 --project ... ``` Exit 1 when the project has no case with that id; nothing is deleted.