Table of Contents
Eval Harnesses Compared: Braintrust vs LangSmith vs Promptfoo vs Arize Phoenix
Most teams pick an eval harness by inertia. LangChain shops default to LangSmith, CLI-first shops reach for Promptfoo, and nobody checks whether the harness actually catches the regressions that reach production. The pick is not cheap to reverse. It locks in your dataset format, your CI integration pattern, and your human-annotation workflow, and unwinding all three later is a quarter of platform work you did not budget.
Three questions decide it, not a feature checklist: where your traces already live, whether you must self-host, and whether evals are a gate in CI or a layer in your observability stack. Answer those and one harness falls out. The table comes first because it is the part you forward to your staff engineer.
Why eval harnesses fail silently
The harness is rarely what breaks. Three failure modes are, and they are independent of which vendor you chose.
Evals that never run in CI come first. A suite that lives in a notebook and runs when someone remembers is not a gate; it is a museum. Next is a golden dataset that drifts from the production distribution, so scores stay green while real traffic moves somewhere your test cases never go. Third is a threshold set so loose it passes everything, which feels like coverage and is theater.
Take a support-ticket agent graded on three scores: faithfulness to the retrieved docs, task completion, and tone. A harness cannot write your thresholds or curate your golden set. What it can do is make each failure mode harder to fall into, through a first-class CI action, dataset versioning tied to production traces, and alerts when a score regresses. That is the axis that matters, and it is where these four genuinely differ.
The four, compared
| Axis | Braintrust | LangSmith | Promptfoo | Arize Phoenix |
|---|---|---|---|---|
| Integration path | Hosted SaaS + SDK; on-prem on Enterprise | Hosted SaaS + SDK; self-hosted/hybrid on Enterprise | Local CLI + YAML, no account | Self-host (Docker/K8s/pip) plus hosted cloud |
| Eval types | LLM-as-judge autoevals, code scorers, human review | LLM-as-judge, code, human annotation | Deterministic and LLM-as-judge assertions | LLM-based, code, human labels, Ragas/Deepeval |
| Dataset management | Hosted versioning, curate from prod logs | Datasets from captured traces, no ETL | Test cases as YAML in the repo, git-versioned | Datasets built from captured spans |
| CI gating | GitHub Action, PR comment, quality gate | Pytest/JS SDK in CI | Native: JUnit XML and JSON, build-fail flag | Runs in CI via SDK; not gate-first |
| Cost / self-host | Free tier; paid Pro; Enterprise on-prem | Free tier; paid per seat; Enterprise self-host | MIT, free | Elastic License 2.0, free self-host |
Dollar figures are dated to their source below. Now the flip conditions. Each tool owns one.
When Braintrust wins
Reach for Braintrust when you need LLM-as-judge scoring plus human annotation at scale and cannot self-host. Its autoevals library ships pre-built judge scorers such as factuality, you add custom code or judge scorers on top, and online scoring grades production traces as they log (per Braintrust's autoevals repo). The GitHub Action posts a per-PR comment showing which cases regressed and by how much, and a quality gate blocks the merge below your threshold (per the Braintrust eval action). Pricing: a free Starter tier with 10K scores per month, Pro at $249 per month with 50K scores then $1.50 per 1K, and on-prem deployment only on Enterprise (per the Braintrust pricing page, July 2026).
When LangSmith wins
LangSmith wins when your stack is already LangChain or LangGraph. It captures traces with zero added instrumentation and turns them into eval datasets with no ETL step, which is the whole pitch: production behavior feeds the golden set directly. Pricing runs $0 for the single-seat Developer tier with 5k base traces per month, $39 per seat per month for Plus with 10k base traces, and self-hosted or hybrid deployment only on Enterprise, with usage metered at $1.50 per LCU compute and $1.00 per LSU storage (per LangChain's pricing page, July 2026). Off a LangChain stack, that zero-instrumentation edge disappears and you are paying for capture you could get cheaper elsewhere.
When Promptfoo wins
Promptfoo is the pick for teams that treat evals as a test suite and want a gate on every PR. It runs from a single promptfooconfig.yaml with no account, and emits JUnit XML and JSON so any CI reporter renders the result; the --fail-on-error flag blocks the build when assertions miss the threshold (per the Promptfoo CI/CD docs). It is MIT-licensed and free, and although it became part of OpenAI in March 2026, the project stated it stays open source and MIT-licensed (per the promptfoo repo). There is no hosted dataset versioning and no annotation UI. The cases live in your repo, which is the point.
When Arize Phoenix wins
Phoenix wins when you already run an OpenTelemetry collector and want LLM spans next to your APM traces in one view. It is OTel-native, built on OpenInference instrumentation, and accepts traces over OTLP with auto-instrumentation for LangChain, LlamaIndex, and the major model providers (per the Phoenix docs). Built-in evals cover LLM-based, code-based, and human-label scoring. One caveat your legal team will raise: Phoenix ships under the Elastic License 2.0, which is not an OSI-approved open-source license and restricts offering it as a managed service (per the Phoenix license page). Self-hosting is free via Docker, Kubernetes, or pip.
One category note: Weights & Biases and MLflow added LLM tracking, but they are experiment trackers rather than purpose-built eval harnesses, so they sit out of this comparison.
The decision matrix
| Your situation | Harness |
|---|---|
| Stack is already LangChain or LangGraph | LangSmith |
| Evals must gate every PR, no vendor account | Promptfoo |
| You run an OTel collector and want LLM spans in it | Arize Phoenix |
| Need LLM-as-judge plus human annotation at scale, hosted | Braintrust |
| Data must stay in your VPC, small platform team | Arize Phoenix (self-host) |
| Data must stay in your VPC, need an annotation UI | Braintrust Enterprise (on-prem) |
Migration cost, stated plainly
Switching is not free, and the bill lands in three places. Dataset export is the first: Braintrust and LangSmith hold your golden sets in their hosted store, so leaving means exporting to JSON or CSV and reshaping to the next tool's schema, roughly a day per dataset of nontrivial size. Trace instrumentation is the second: moving off LangSmith's zero-instrumentation capture onto Phoenix means adding OpenInference/OTLP instrumentation across your call sites, and moving onto Promptfoo means you stop capturing production traces for evals entirely and rebuild the golden set from logs. CI config is the third: going from Promptfoo to anything hosted means rewriting the YAML assertions as SDK eval scripts and wiring API keys into CI, a few days of work. The lightest switch is into Promptfoo, since your cases become YAML in the repo. The most expensive is out of a hosted trace-capture tool once production traces have become your dataset pipeline. Price that lock-in before you adopt, not after.
For the CI pattern that gates a merge on eval scores without blocking every PR, and the spec-first workflow that defines those evals as the acceptance contract, see the spec-driven agent development guide and the enterprise agent pillar.