The Agent Platform RFP Scorecard

Here is the pattern that costs a platform team a quarter. You run a two-week proof of concept on a cooperative use case, the demo lands, and you sign. Somewhere past 150 seats the invoice stops matching the plan, the security team asks for an audit trail that was never built, and finance cannot say which department burned the budget. Now you are either writing the missing controls yourself or migrating off a platform you picked eight weeks ago.

The token-economics version of this is on the public record. GitHub moved every Copilot plan to usage-based billing on June 1, 2026, and removed the silent fallback to a cheaper model once limits are hit (per the GitHub blog, April 27, 2026). Developers on a community thread that runs to hundreds of replies describe the same wall: no visible per-feature token count, charges that surface after the fact, and no way to attribute spend to the work that generated it. TechCrunch covered the backlash under a headline a developer supplied, "What a joke" (May 30, 2026).

None of that was hidden. Each blocker was answerable before the POC by a vendor question the buyer did not ask. That is the whole job of a scorecard.

How to run it

Send the scorecard to the vendor before the POC, not after. Score the answers, then use the low scores to write the POC. A vendor that scores a 1 on governance is not a reason to walk yet. It is the exact claim your two weeks should try to disprove, instead of demoing another happy path. The scores never name a winner. They tell you which questions have to be closed before a purchase order moves, and they leave a paper trail security and finance can co-sign. Five dimensions carry most of the risk.

Harness architecture and SDK compatibility

The question: can your platform team ship a new agent skill in a day against a real SDK, and does the trace output drop into the OpenTelemetry collector you already run? Teams fail this when they buy on the demo and later find the "SDK" is JSON over HTTP with no type safety, no local test loop, and a bespoke trace format that needs a custom exporter.

Two things make the difference concrete. First, a genuine SDK gives you typed tool registration and an in-process test path. The Claude Agent SDK ships Python and TypeScript libraries where a custom tool is a typed function, registered through an in-process MCP server (per Anthropic's Agent SDK docs):

// A typed custom tool, registered through the SDK's in-process MCP server
import { tool, createSdkMcpServer } from "@anthropic-ai/claude-agent-sdk";
import { z } from "zod";

const issueRefund = tool(
  "issue_refund",
  "Issue a refund for an order",
  { orderId: z.string(), amount: z.number() },
  async ({ orderId, amount }) => {
    // typed handler; return isError:true to control the message Claude reads
    return { content: [{ type: "text", text: `refunded ${amount}` }] };
  }
);

const server = createSdkMcpServer({ name: "billing", tools: [issueRefund] });

Second, the observability vocabulary already exists, so you should not accept a proprietary one. OpenTelemetry's GenAI semantic conventions (semconv 1.43.0) define the attribute set a harness should emit, including gen_ai.agent.id, gen_ai.operation.name, and gen_ai.evaluation.score.value (per the OTel GenAI registry):

# A subset of OTel GenAI attributes a compliant harness emits (semconv 1.43.0)
gen_ai.agent.id: "invoice-triage-v3"
gen_ai.conversation.id: "c-9f2a"
gen_ai.operation.name: "invoke_agent"
gen_ai.evaluation.name: "grounding-check"
gen_ai.evaluation.score.value: 0.94

A platform that emits those attributes plugs into your existing pipeline. One that does not means you write and maintain the bridge. Ask: "Show me a custom tool registered through the SDK." "Which OTel GenAI conventions do you emit, and at what semconv version?" Our OpenTelemetry-for-agents walkthrough covers the collector side.

Eval maturity and CI integration

The question: does the platform have a native eval runner, and can that eval gate block a model-version rollout before it reaches production seats? Teams fail this when they ship without a gate, a model upgrade quietly degrades their highest-stakes skill, and 200 users see the regression before anyone opens a dashboard. A model swap you did not choose can move eval scores overnight, which is why the gate has to sit in CI, not in a weekly review.

Ask for the mechanics, not the marketing: "Show me an acceptance eval for a new skill running in CI." "When a model upgrade drops eval scores below threshold, is rollback automatic or a support ticket?" If you are choosing the eval layer separately, our comparison of Braintrust, LangSmith, Promptfoo, and Arize maps the tradeoffs.

Governance and audit controls

The question: does the platform produce an immutable audit log of every agent action (tool call, input, output, user identity, timestamp), exportable to your SIEM, with per-skill egress policies you can lock to an allowlist? This is the dimension your security team scores, and most platforms fail on egress granularity. A log that exists but cannot leave the vendor's console is not an audit trail your auditors will accept.

The shape you are scoring for looks like this per tool call:

{
  "timestamp": "2026-08-11T14:03:22Z",
  "actor": "user:[email protected]",
  "agent": "invoice-triage-v3",
  "tool_call": "sql_query",
  "input": "SELECT amount FROM invoices WHERE vendor_id = 8841",
  "egress": "postgres.internal (allowlisted)",
  "output_hash": "sha256:1f3c9a...",
  "outcome": "allow"
}

Ask: "Show a sample audit-log entry for a tool call." "How do I set an egress allowlist for one specific skill, and what happens on a blocked call?" A vendor that answers both without a services engagement is rare, and worth points.

Token economics and billing transparency

The question: does the vendor bill by seat, by token, or by a credit model that hides the unit cost, and can you set per-user and per-skill budgets that hard-stop at a ceiling? Finance fails this when the monthly invoice cannot be mapped back to the teams that generated it. That is not hypothetical. GitHub added admin budget controls at the enterprise, cost-center, and user level on the same June 2026 change, because aggregate-only billing does not survive a CFO review (per the GitHub blog). If your vendor cannot show per-user spend today, and cannot say what happens when a user hits their ceiling, you are buying the pre-June-2026 problem on purpose.

Ask: "Show me per-user spend in the billing dashboard, broken out by cost center." "At a user's monthly ceiling, is it graceful degradation or a hard error?" For the budget math before you buy, see our token-budget modeling guide. Score this dimension higher than instinct tells you; it is the one teams underweight and pay for at 200 seats.

Vendor lock-in and model portability

The question: if the vendor raises prices or changes model access next year, what is the migration cost? A platform that abstracts the model layer lets you swap the underlying model with a config change. One that bakes the model into the harness turns a price change into a rebuild. The related trap is agent state and memory stored in a proprietary format you cannot export.

This dimension matters least at pilot and most at renewal, so weight it for the renewal you will actually negotiate. Ask: "Show me the config change to swap the model for a single skill." "What agent state or memory lives in your format versus an open standard, and how do I export it?"

Scoring, weighting, and what breaks first

The five dimensions are not equal weight, and the defaults are a starting point you should overwrite for your context. Governance and eval maturity are veto criteria for a regulated industry: a score below 2 on either blocks the purchase regardless of the average. Token economics is the dimension teams most often score generously to avoid re-opening the POC, and it is the one that breaks first past roughly 200 seats. Vendor lock-in reads cheap at pilot time and expensive at renewal, so score it for the second contract, not the first.

Dimension	Key vendor question	1 (fail)	3 (pass)	5 (top)
Harness architecture	"Show a custom tool registered via the SDK."	REST-only, no SDK	SDK exists, no OTel	SDK + OTel + typed errors
Eval maturity	"Block a model upgrade on a failing eval in CI."	No eval runner	Manual eval, no CI gate	Automated CI gate + rollback
Governance	"Show an audit-log entry for a tool call."	No audit log	Log exists, not exportable	Immutable, SIEM-exportable
Token economics	"Show per-user spend in the billing dashboard."	Aggregate only	Per-seat breakdown	Per-skill + cost-center export
Vendor portability	"Swap the model with a config change."	Model baked in	Config change possible	Open-standard agent state

Applied to two current SDKs, the Claude Agent SDK and the OpenAI Agents SDK, this table returns your answer, not ours. The framework decides; the vendor column is yours to fill.

Run it as a live document

The downloadable scorecard is one table: vendor, dimension, vendor response, evaluator score (1 to 5), and notes. Fill it in during vendor calls, hand it to the security and finance stakeholders who have to co-sign the purchase, and open it again at renewal when the model-access and pricing terms are back on the table. It is gated on the Pondero email list because it is meant to be a shared internal artifact, not a one-time read. If you want the reasoning behind the five dimensions, the six patterns deployed teams share is the companion piece, and the enterprise agent pillar is the hub for the rest.