Agent Evals

Agent Evals is a YAML ground-truth harness that runs canned questions through the AI Ask surface and reports where the LLM diverged from expectations. Every case is a YAML file committed alongside your semantic model; CI runs the suite and fails the build on any regression.

Deterministic, no LLM-as-judge

LLM-as-judge is expensive, non-reproducible, and lets subtle wrong answers slip through. Saiku’s runner does the same job the hard way: direct value comparison with numeric tolerance and explicit key normalisation — the same discipline the semantic layer itself gets.

Ships in saiku v4.7 as saiku#1424.

Why evals

Catch regressions on provider swaps Switching from Claude to GPT? Bumping the model version? Evals surface every question where the answer drifted before your users notice.

Freeze correctness at review time Every PR touching the semantic model runs the eval suite. A rename that breaks the AI's ability to answer the finance team's canonical question fails CI before it lands.

Explicit, reviewable expectations Ground truth lives in YAML in your repo. Reviewers see what the finance team asked for, what the correct answer is, and which cases the LLM currently passes or fails.

Same discipline the schema gets Semantic-model changes get code review; ask-flow correctness gets code review too. Regression protection travels with the schema.

File format

Suites live as YAML under saiku-home/evals/. One suite per file. Each suite targets one cube; every case in the suite runs against it.

saiku-home/evals/foodmart-sales.eval.yaml

name: foodmart-sales-evals
description: >
  Ground-truth cases for the FoodMart Sales cube. Baseline for AI ask
  regressions when the LLM provider, model, or prompt changes.

cube:
  connectionName: unknown_foodmart
  catalog: FoodMart
  schema: FoodMart
  cubeName: Sales

cases:

  # QUERY case — expected rows compared with tolerance.
  - name: sales-by-country
    question: show me store sales by country
    expectedIntent: QUERY
    expectedRows:
      - {country: "USA",    storeSales: 565238.13}
      - {country: "Canada", storeSales:  79063.11}
      - {country: "Mexico", storeSales:  51298.13}
    tolerance:
      relative: 0.001    # 0.1% relative — masks sub-cent warehouse drift
    orderMatters: true

  # INSIGHT case — insight markdown must contain each string.
  - name: trend-analysis
    question: are sales trending up week-on-week?
    expectedIntent: INSIGHT
    expectedInsightContains:
      - Store Sales
      - week-on-week

  # REFUSED case — model must decline off-topic with a matching reason.
  - name: refuse-off-topic
    question: what's the weather in Paris?
    expectedIntent: REFUSED
    expectedRefusalContains: cube

  # Multi-turn case — history seeds the ask.
  - name: follow-up-turn
    question: now break it down by state
    history:
      - {role: user,      content: show store sales by country}
      - {role: assistant, content: 3 rows returned.}
    expectedIntent: QUERY
    expectedRows:
      - {state: "CA", storeSales: 214893.10}

Suite fields

Field	Required	Type	Notes
name	yes	string	Displayed in the report and CI summary
description	no	string	Free text shown in the report header
cube	yes	object	connectionName / catalog / schema / cubeName
cases	yes	array	Non-empty list of ground-truth cases

Case fields

Field	Required	Type	Notes
name	yes	string	Unique within the suite; used in mismatch paths
question	yes	string	Natural-language ask fed to /ai/ask
history	no	{role,content}[]	Prior turns to seed the ask (empty for single-shot cases)
expectedIntent	no	string	QUERY | INSIGHT | VIEW_CHANGE | REFUSED (case-insensitive)
expectedRefusalContains	no	string	Substring the refusal reason must contain
expectedRows	no	{key:value}[]	Static expected rows for QUERY. See Row comparison.
referenceQuery	no	object	Drift-proof ground truth — a typed AiQueryRequest executed live against the same cube; its rows become the expected set. Survives data changes that would break hard-coded expectedRows. Prefer this for QUERY cases.
orderMatters	no	boolean	Defaults true. When false, both sides sort by keys before diff.
expectedInsightContains	no	string[]	Substrings the insight markdown must contain
tolerance	no	{absolute,relative}	Numeric tolerance for row comparisons; both default 0.0 (exact)

Row comparison

Key normalisation

Column keys compare case-insensitively with whitespace, underscores, and hyphens stripped:

• Store Sales
• storeSales
• store_sales
• STORE-SALES

All normalise to the same key. Prevents a rename in the schema-projector’s output shape from failing every eval that references the column.

Mismatch paths preserve what you wrote

If a comparison fails, the mismatch path is reported using the original key from the YAML — rows[2].storeSales, not the normalised rows[2].storesales. Error messages point at the right line so you fix the right thing.

Numeric parsing

Expected values that parse as numbers are compared numerically with tolerance. The parser is forgiving about how eval authors write numbers:

YAML value	Parsed as	Notes
500	500	Integers
500.0	500.0	Decimals
"$565,238.13"	565238.13	Currency prefix + thousands separator stripped
"(500)"	-500	Parenthesised negatives
"12%"	12	Trailing % stripped

Non-numeric expected values compare as strings (whitespace-trimmed on both sides).

Tolerance

tolerance:
  absolute: 0.01     # cell passes if |actual - expected| <= 0.01
  relative: 0.001    # cell passes if |actual - expected| / |expected| <= 0.001

A cell passes if either tolerance is satisfied — set both when you want “5 cents absolute OR 0.1% relative, whichever is looser.”

Both default to zero (exact match).

Missing keys are asymmetric

• An expected key not in the actual row is a mismatch — the eval author wrote an expectation the schema doesn’t produce.
• An actual key not in the expectation is not a mismatch — expectations are additive, so growing the schema with a new column doesn’t break every eval that predates it.

Reports

The runner emits an aggregated report per suite with one outcome per case:

Suite: foodmart-sales-evals
Description: Ground-truth cases for the FoodMart Sales cube
9/10 passed, 1 failed, 0 degraded, 0 skipped (elapsed 12483ms)

PASS: sales-by-country (854ms, intent=QUERY, model=claude-sonnet-4-6)
PASS: trend-analysis (742ms, intent=INSIGHT, model=claude-sonnet-4-6)
PASS: refuse-off-topic (312ms, intent=REFUSED, model=claude-sonnet-4-6)
FAIL: top-3-product-families (1024ms, intent=QUERY, model=claude-sonnet-4-6)
  rows[0].productFamily: expected "Food", got "Drink"
  rows[1].productFamily: expected "Non-Consumable", got "Food"
  rows[2].productFamily: expected "Drink", got "Non-Consumable"

Or as pretty-printed JSON for CI archives via EvalReportWriter.toJson().

Parse-error codes

Structured YAML-parse failures carry a stable code so CI can distinguish “author wrote a broken YAML” from “the ask surface produced a wrong answer”:

Code	When
MALFORMED_YAML	The YAML parser rejected the file.
MISSING_FIELD	A required field is absent (name, cube, cases, question).
BLANK_FIELD	A required string field is present but empty.
TYPE_MISMATCH	A field has the wrong type (e.g. cases is a mapping, not an array).
INVALID_TOLERANCE	tolerance.absolute or tolerance.relative is negative.

Running a suite

Evals run against a live server — they exercise the same ask + execute path your users hit. Point the bundled saiku eval CLI at a running launcher; it POSTs to the admin run endpoint and reports pass-rate, returning a non-zero exit on any regression so it drops straight into CI.

CLI

# against a locally-running server (default admin/admin)
java -jar saiku-<version>.jar eval

# against a remote server
java -jar saiku-<version>.jar eval \
  --server https://analytics.example.com \
  --username admin --password '<secret>'

Exit codes: 0 = every suite passed · 1 = a suite regressed · 2 = transport/config error. Add --no-fail-on-regression for a report-only run that never exits non-zero.

REST

# admin-gated; runs every suite in saiku-home/evals/ synchronously
curl -sS -X POST -u admin:admin \
  http://localhost:8080/saiku/admin/ai-evals/run | jq

The admin panel reads the persisted results — GET /saiku/admin/ai-evals (per-suite cards), .../{suite}/runs, and .../{suite}/trend.

GitHub Actions

.github/workflows/agent-evals.yml

name: agent-evals
on: [pull_request]

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
      - name: Run eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          # Boot the launcher, wait for health, run the suites.
          java -jar saiku-*.jar serve --port 8080 &
          npx wait-on http://localhost:8080/saiku/api/info
          java -jar saiku-*.jar eval

Live accuracy monitoring

Evals aren’t only a CI gate. Every run is scored and persisted (H2, in your saiku-home), and the Admin → Agent evals dashboard plots pass-rate over time per suite — so you can watch for accuracy drift on a production deployment, not just catch it in a pull request. Schedule the saiku eval CLI on a cron to feed the trend continuously.

Bundled example

A working suite ships with the launcher at saiku-launcher/src/main/resources/seed/evals/foodmart-sales.eval.yaml.

Four baseline cases across two intents, all drift-proof:

• QUERY — store-sales-by-country + unit-sales-by-product-family, each with a referenceQuery as ground truth (executed live, so a data refresh can’t break them)
• REFUSED — refuse-general-knowledge + refuse-coding-help with a reason-substring check

Non-goals for v1

• Recording adapter — an adapter that writes each live response to a fixture file so a subsequent fixture adapter can replay deterministically without spending LLM budget. Design ready; implementation deferred.
• Structural diff on the emitted AiQueryRequest — multiple structurally-different requests can produce the same rows, so row-diff is a better ground truth.

Where to go next

AI Ask API The endpoint every eval case runs through. Configuration, response envelope, streaming variant.

Agent Skills Admin-authored workflows that appear in ask responses. Skills can be eval-covered like any other question.

Agent Spaces Personas that scope the ask surface. A per-space eval suite catches persona-specific regressions.

AI Query API Typed layer that /ai/ask translates into. Programmatic evals could target this directly for shape-only regressions.