CI for Agents: Gating Merges on Eval Scores Without Blocking Every PR

An eval suite that does not gate a merge is a decoration. It sits in the repo, it runs when someone remembers, and it catches nothing the day a prompt tweak drops the real pass rate and the unit tests still go green. Wire it in badly and you hit the opposite wall: every push runs the full suite, and feature branches queue behind a job that takes the better part of an hour. Both failure modes are common. The pattern that dodges both is a sampled gate on a versioned threshold, and it is about thirty lines of YAML.

What the gate actually checks

Start with what it is not. It is not test coverage, and it is not a linter. A green unit-test run tells you the code wrapping the agent works. It says nothing about whether the agent still refuses the requests it is supposed to refuse. The number that matters is eval pass rate against your acceptance suite: the cases derived from the spec, where each case asserts one behavior the agent owes its users. Our spec-driven development guide covers how to build that suite; this piece assumes you have one and turns it into the merge gate. The mechanics are small. The job reads one number off the eval run and compares it to a threshold committed in the repo. Above the line, the check is green and the PR can merge. Below it, the check goes red, and because it is a required check, the merge button is disabled.

The CI architecture

Three moving parts: an eval job on the pull request, a heavier job on a schedule, and a threshold you can read in a diff.

The PR job runs a sampled subset, not the whole suite. Point it at a smaller config that stratifies across your failure-prone categories (escalation logic, refusals, anything that has burned you before) rather than a flat random draw, so the sample keeps signal on the cases that actually regress. The full suite runs nightly and on release branches, where a longer wall time buys you nothing on a feature PR but catches slow drift before a release. Store the threshold as a file in the repo, not a CI environment variable. A number in threshold.txt is a reviewable diff with a name attached; a number buried in CI settings changes silently and nobody can say when or why.

Here is the PR gate. Promptfoo is the runner in this example, but the shape is the runner-agnostic part:

name: agent-eval-gate
on:
  pull_request:
    paths: ['agent/**', 'evals/**']
jobs:
  eval-gate:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write   # post the pass rate back onto the PR
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - name: Run the PR sample suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: npx promptfoo@latest eval -c evals/pr-sample.yaml -o results.json
      - name: Enforce the versioned threshold
        run: |
          THRESHOLD=$(cat evals/threshold.txt)
          PASS=$(jq '.results.stats.successes / (.results.stats.successes + .results.stats.failures) * 100' results.json)
          echo "pass rate ${PASS}% vs threshold ${THRESHOLD}%"
          if (( $(echo "$PASS < $THRESHOLD" | bc -l) )); then
            echo "eval gate failed: below ${THRESHOLD}%"; exit 1
          fi

The threshold-parsing step follows the quality-gate pattern in Promptfoo's CI/CD docs, which reads pass rate out of the JSON with jq and exits non-zero below the line (their worked example uses a 95% bar). The last thing to do is not in the YAML: add this job to the branch's required status checks. Per GitHub's protected-branch docs, once a check is required, every required check must pass before anyone can merge into the protected branch. That single setting is what turns a red job into a real gate.

The runner swaps out cleanly. Promptfoo's GitHub Action runs the eval from a YAML config and posts a before/after comment on PRs that touch your prompts. Braintrust ships braintrustdata/eval-action@v2, which does the same from JS or TS eval files. LangSmith runs evals from its SDK inside the same job. Pick on where your traces already live, not on the CI wiring; the comparison is our eval harnesses piece. And the whole job can call your agent through the Claude Agent SDK, which is what Anthropic's own GitHub Actions integration is built on.

On Buildkite the shape is identical: one pipeline step runs the eval CLI and writes results.json, a second parses it and exits non-zero below the threshold, and your VCS provider's required-check integration blocks the merge. The eval logic is portable; only the branch-protection glue is provider-specific.

Example: a 50-case acceptance suite, PR job sampling 20% of it, a threshold of 92% held in threshold.txt. A passing PR clears the sample in a couple of minutes; a regression that drops the sample below 92% turns the check red and locks the merge until it is fixed or the threshold change is justified in its own PR. Numbers here are illustrative, not measured.

What breaks first

The gate does not create these problems. It makes them visible, usually in the first month, which is exactly when a new required check is least welcome.

Failure mode	Earliest symptom	The fix
Noisy evals	The same eval flips verdict on unchanged code between runs	An eval whose result is not stable is a bad assertion, not a flaky test. Require every eval to clear a determinism check (run it repeatedly on a pinned model at temperature 0; if the verdict flips, it never enters the gated suite)
Threshold drift	The team ratchets the bar up after a good week, then panics on the next dip	The threshold lives in threshold.txt. Changing it is its own PR with a one-line reason, reviewed like any code change, so the bar cannot creep silently
Slow full runs	The nightly grows past its window and starts stalling release branches	Parallelize the eval workers (3 to 5 concurrent) and keep the PR gate on the sample. The full suite is a signal, not a merge blocker on feature branches

The determinism check is the one teams skip and regret. An eval that cannot reproduce its own verdict will fail a PR that changed nothing, and after the second false alarm your engineers stop trusting the gate and start asking for override rights. Kill non-determinism at authoring time, before the eval can ever block a merge.

Cost rarely turns out to be the constraint, but size it so nobody has to guess:

Example: a 50-case suite spending roughly 2,000 tokens per case round trip is about 100,000 tokens per full pass. Multiply by your model's posted per-token price to size the nightly. Run that arithmetic against your provider's own pricing page, not this example.

What your security team will ask

This is the block you forward internally. The reviewer who signs off on the rollout has three questions, and each one has an artifact behind it, not a promise.

Question	Answer	The artifact
Does the eval runner touch production data?	No. The CI job runs against a sandboxed environment with synthetic fixtures	The isolation ladder and egress policy in our agent sandboxing guide
Are results stored and auditable?	Yes. The runner writes results.json; retain it as a CI artifact and link it from the PR	The JSON output plus the PR link; set artifact retention to your audit window
Who sets the threshold, and can it be lowered quietly?	The platform team, in threshold.txt. A drop below the agreed floor requires a reviewed PR	The threshold's git history, which shows every change and its stated reason

The pattern that satisfies all three is the same one: everything the gate depends on lives in version control, so every answer is a link to a diff instead of a claim in a meeting. Security teams reject vibes and accept artifacts. Give them artifacts.

The rollout, week by week

• Week 1: measure your current suite's pass rate on main. That number is your baseline and your starting threshold.
• Week 2: add the sampled eval job as a non-required check. Let it run on every PR for a week and watch for noise before it can block anyone.
• Week 3: promote it to a required status check, with the threshold set to baseline minus 5 points, so the first month's noise does not wall off every merge.
• Month 2: move the threshold back toward baseline as the noise drains, review it quarterly, and add the determinism check to your eval-authoring guide so no new eval can enter the suite flaky.