CI for agents: gating merges on eval scores without blocking every PR

The eval gate that gets skipped is the one that runs on every commit. A team wires an LLM-as-judge suite to a required GitHub Actions check, the suite takes 8 to 12 minutes per push, and by week three someone marks it non-required or comments out the pull_request trigger during a late deploy. Nobody logs a decision. The check just stops being a wall.

That is not a discipline problem you fix with a Slack reminder. It is a gate-design problem, and the fix is not faster hardware. The guard that slows every commit gets disabled. The guard that only fires on merge survives, because it does not stand between a developer and a green PR. So you split the work by how much latency each check earns the right to spend.

If you got here from spec-driven agent development, you already have acceptance evals derived from your spec. This is how you wire them into CI without the gate becoming the thing your team routes around.

Three tiers, one latency budget each

Each tier has a contract: what it catches, what it misses, and what it is allowed to block.

Tier	When it runs	What it checks	Latency target	Blocks merge?
T1: Deterministic	Every commit on the PR	Schema validation, tool-call format, trajectory:tool-used / tool-sequence, cost-guard assertions, golden I/O pairs	< 90 seconds	Yes
T2: LLM-judge	On merge to the base branch	Full behavioral spec: refusal cases, acceptance-eval correctness, response-time budget, rubric scores	< 10 minutes	Yes
T3: Regression	Nightly, or on tag	Full suite including edge cases, cross-model comparison, cost-per-task audit	< 60 minutes	No, alert only

T3 is alert-only for one reason: a regression found by a 60-minute nightly run at 2 AM should not hold the release until standup. Blocking on it converts a signal into an outage. Page someone, open an issue, keep shipping.

The deterministic tier does most of the real gatekeeping. Promptfoo's agent assertions need no judge: trajectory:tool-used, trajectory:tool-args-match, and trajectory:tool-sequence confirm the agent called the right tools, with the right arguments, in the right order. They run in milliseconds and catch the failure class that actually breaks agents in production: a wrong tool call, a bad argument, a skipped step.

The worked config

Two jobs in one workflow. T1 runs on pull_request with deterministic asserts only; T2 runs on push to main with the LLM-judge suite. Both use the Promptfoo GitHub Action, which posts a pass/fail summary comment on the PR automatically. No account is required; the action runs npx promptfoo on your runner.

name: agent-evals
on:
  pull_request:
    paths: ['agent/**', 'evals/**']
  push:
    branches: [main]

jobs:
  t1-deterministic:                 # every commit, no judge, target < 90s
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/cache@v4
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-${{ hashFiles('evals/**') }}
          restore-keys: ${{ runner.os }}-promptfoo-
      - uses: promptfoo/promptfoo-action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: evals/t1.deterministic.yaml   # schema + trajectory asserts only
          cache-path: ~/.cache/promptfoo
          fail-on-threshold: 100                 # annotation 1
          max-concurrency: 8

  t2-llm-judge:                     # on merge, full spec, target < 10m
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/cache@v4
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-${{ hashFiles('evals/**') }}
          restore-keys: ${{ runner.os }}-promptfoo-
      - uses: promptfoo/promptfoo-action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          config: evals/t2.behavioral.yaml       # llm-rubric asserts, judge pinned in config
          cache-path: ~/.cache/promptfoo          # annotation 2
          fail-on-threshold: 90
          repeat: 3                               # annotation 3
          repeat-min-pass: 2
          max-concurrency: 4

Three annotations, keyed to the comments above:

1. fail-on-threshold is the required suite pass percentage from 0 to 100, per the action's inputs. T1 sits at 100: every deterministic case must pass, because a malformed tool call is never acceptable. T2 sits at 90, which leaves room for a single flaky judge score without red-lining the merge.
2. Judge token cost is bounded by the cache, not by trimming the suite. actions/cache@v4 plus cache-path stores prior LLM requests and responses; unchanged eval cases reuse the cached judge call and are not re-billed. max-concurrency (default 4) caps parallel API calls so a large suite does not spike your rate limit.
3. The reviewer surface for a red gate is the action's auto-posted comment, which carries pass/fail counts and a web-viewer link. If you want the raw run retained in the Actions tab, add an explicit output step:

      - run: npx promptfoo@latest eval -c evals/t2.behavioral.yaml --output results.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: t2-eval-results
          path: results.json

Braintrust is the managed equivalent. Swap the action for bt eval with a BRAINTRUST_API_KEY secret; bt eval --first 20 tests/ --no-input --json runs a 20-case subset as the PR smoke check, and bt eval tests/ runs the full suite on merge. Same tiering, hosted trace UI, per-seat cost. Which harness to standardize on is a separate call covered in the eval-harnesses comparison.

The gate that cries wolf

An LLM judge that scores identical agent output at 7 on one run and 5 on the next is not a quality signal. It is noise, and noise trains developers to override the gate. Three sources, three fixes.

Judge temperature above 0. Set it to 0 for every scoring pass. In Promptfoo the judge lives in defaultTest.options.provider, and temperature is a config field on it:

defaultTest:
  options:
    provider:
      id: anthropic:messages:claude-haiku-4-5
      config:
        temperature: 0
tests:
  - vars: { order_id: '123' }
    assert:
      - type: trajectory:goal-success
        value: Determine shipping status for order {{order_id}} and tell the user
        threshold: 0.8
      - type: llm-rubric
        value: Response cites the tracking number and gives no delivery date it did not look up

Temperature near zero reduces run-to-run decision variance without shifting the mean score much, per a 2026 study on judge temperature. It is the cheapest fix on this list. Do it first.

Underspecified rubric. A judge told to score "quality" scores on vibes. The rubric has to be as explicit as the acceptance spec it came from: name the exact condition, as in the llm-rubric above ("cites the tracking number and gives no delivery date it did not look up"), not "is the response good." A vague rubric is the largest source of judge variance you actually control.

Single-sample scoring. One judge call per case amplifies self-inconsistency, which is why repeat: 3 with repeat-min-pass: 2 above takes a majority of three. That smooths a model scoring itself differently on reruns (2025 work on judge self-inconsistency). One caveat the vendor docs skip: three calls to the same judge model are correlated, not three independent opinions, so majority voting cuts jitter but not a systematic bias. Research on judge panels found the votes share errors, so nine judges can behave like two. If a rubric is systematically wrong, run it past a second judge model, not a second seed.

What breaks first

Three failure modes, each with a measurable trigger.

Evals encoding yesterday's spec. The suite was derived from the spec on day one. The spec changes on day 30, the evals do not, and by month two the T2 gate is enforcing requirements that no longer exist while missing the ones that do. The trigger to watch: a rising rate of gate overrides where the reviewer's comment is "eval is stale," not "code is wrong." The fix is ownership, not tooling. Whoever owns the spec owns the evals, and the spec-and-evals change lands in the same PR or the PR does not merge.

Cost creep on the judge. Running Opus on every merge adds up faster than teams model. Here is the arithmetic at current pricing, so you can drop in your own numbers.

Example (2026-07-31 pricing): a 50-case T2 suite, repeat: 3 for majority voting, roughly 2,000 input and 200 output tokens per judge call. That is 150 judge calls and about 0.30M input plus 0.03M output tokens per merge. At 10 merges/day over ~21 working days (210 merges/month):

• Opus 4.8 at $5/$25 per MTok (Anthropic pricing): ~$2.25/merge, about $470/month.
• Haiku 4.5 at $1/$5 per MTok (same page): ~$0.45/merge, about $95/month.
• GPT-4o mini at $0.15/$0.60 per 1M (per current pricing): about $13/month.

The gate that costs several hundred dollars a month gets budget-cut before the quarter closes; the same suite on a small judge like Haiku 4.5, an order of magnitude cheaper in the math above, survives. Size the judge to the task. A rubric that names an explicit condition is exactly where a small judge holds up, because there is little left for a bigger model to reason about. Do not take that on faith: run your last 50 cases through both the small and large judge, check score agreement, and only then swap. Override the judge with --grader or the provider field shown above.

Eval results nobody reads. A red gate a developer clears with a manual "LGTM" bypass is not a gate. The score has to be visible on the PR, which is why the Promptfoo action posts its summary as a comment and Braintrust links the trace. The number that matters is the delta from the last green run, not a pass/fail boolean: a suite that drops from 96% to 91% while still clearing a 90% threshold is a regression in progress. Wire the comment; do not bury the result in an Actions log.

Before you ship this

Six checks a platform lead can drop into a team wiki as a pre-launch gate.

1. Acceptance evals exist and are version-controlled next to the agent code.
2. T1 checks run in under 90 seconds on your CI runner. Measure it on a real PR; do not assume.
3. Judge model temperature is fixed at 0 in the eval config.
4. The rubric is grounded in the acceptance spec, naming explicit conditions, not a general quality prompt.
5. Eval results post to the PR as a comment, not only to the Actions log.
6. "Spec and evals updated together" is a required line item in the PR template.

The merge gate catches behavioral regressions before they ship. It is the pre-deploy half of the story; the runtime half, containing what a compromised or misbehaving agent can touch once it is live, is agent sandboxing.