How I'd Evaluate Agentic Coding Tools If I Were Building as a Platform Product Manager

A product builder's framework for the decision no one is making carefully enough.

TL;DR

• Most teams pick agentic coding tools the wrong way: by demo impression or benchmark score

• Before any evaluation starts, define what “working” looks like as specific, written assertions otherwise your scoring is just a longer demo

• The right evaluation framework has four layers: cost model, workflow fit, lock-in surface, and governance readiness

• Cost-per-accepted-change is the metric that matters, not monthly seat price

• Model version drift is an underrated operational risk teams without written assertions face weeks of re-testing every time a vendor ships a new model

• AGENTS.md is developer platform infrastructure, not a config file

• Routing work by type across tools beats single-tool loyalty for most eng orgs

• Scoring matrix included with directional scores use it as a starting point, not a verdict

---

Let me tell you how most teams are making this decision right now.

A developer tries one of these tools over a weekend. It impresses them. They write a Slack message: “we should standardize on this.” A manager approves a few Pro licenses. Six months later, the team has wired their review process, their CI hooks, and their onboarding docs around that tool and switching feels impossible.

That’s not a technology decision. That’s habit formation masquerading as strategy.

I’ve spent time thinking about what a rigorous evaluation looks like for a Platform PM who’s being asked to make this call for a mid-sized engineering org. Here’s the framework I’d use.

---

Before the Layers: Define What “Working” Looks Like

This is the step almost every team skips and it’s why most pilots produce feelings instead of data.

Before you evaluate any tool, write down what passing looks like. Not “the agent feels productive.” Specific, checkable assertions:

• “An agent session on our 50k-line repo completes a 3-file refactor and opens a mergeable PR in under 10 minutes”

• “The audit log captures the full session plan proposed, plan approved, files changed in a format our compliance team can read without engineering help”

• “The agent’s proposed plan is rejected or significantly redirected by the engineer in fewer than 25% of sessions”

• “CI pass rate on agent-authored PRs is within 10 points of the baseline for human-authored PRs”

This is the same principle behind eval design for AI agents: you grade against assertions, not vibes. Without this step, your scoring matrix measures impressions, and your pilot is just a demo that ran longer.

The assertions also do something else: they force alignment before the tool decision. When you write “the audit log must be readable by our compliance team,” you’ve already surfaced that compliance has a requirement and that’s a conversation worth having before you’re six weeks into a pilot on a tool that doesn’t support it.

Builder note: Write your assertions collaboratively with PM, a senior engineer, and whoever owns compliance or security sign-off. The disagreements that surface in that conversation are exactly the requirements you’d otherwise discover in an incident.

---

The Four Layers of a Real Evaluation

Layer 1: The Cost Model (what you’re actually buying)

The first mistake is comparing monthly seat prices. An agentic coding tool doesn’t bill like a SaaS seat. It bills like token and compute because that’s what it is.

An agent session on a real codebase involves: reading the full repository context, proposing a plan, potentially spinning up a sandbox, running tests, looping through retries, opening a pull request. The token and compute cost of that session is an order of magnitude higher than a simple code completion.

The metric that matters: cost per accepted change.

Here’s how to calculate it:

Cost per accepted change =
  (total agent compute cost over pilot period)
  ÷
  (number of changes merged to main)

Run a two-week pilot on a real refactor not a toy repo, not a greenfield project.

Track:

1. Agent sessions initiated

2. Changes proposed

3. Changes accepted by reviewers

4. CI pass rate on first submission

5. Total compute cost (tokens + any sandbox/execution costs)

That gives you a real number. The tool with the lowest sticker price is rarely the tool with the lowest cost per accepted change at team scale.

What this looks like across the current field:

Builder note: If you’re designing the billing model for a developer platform that uses these tools under the hood, cost-per-accepted-change is also the metric your customers will eventually hold you to. Build your margin model around it from day one.

---

Layer 2: Workflow Fit (where the agent lives in your developer loop)

The second mistake is evaluating the tool in isolation rather than evaluating how it fits inside the existing developer loop.

There are two fundamentally different places an agent can live:

Inside the editor: Cursor’s model. The agent is present while the developer is present. It operates on the files they’re already looking at. Low context-switch cost. The developer never fully hands off; they stay in the loop.

Outside the editor: Claude Code, Codex, Grok Build’s model. The developer hands off a task, the agent works, the developer reviews the output. Higher throughput potential. Higher risk if approval gates are weak.

Neither is better. They serve different workflows.

The questions that determine fit:

• How strong is your team’s code review culture? (Higher culture = more tolerance for approval-gate tools)

• How large are the typical tasks? (Multi-file refactors favor terminal-native; single-function edits favor editor-integrated)

• How senior is the team using the tool? (Junior engineers benefit from staying inside the editor loop; senior engineers often prefer handing off and reviewing async)

• What does your CI pipeline look like? (Tools that can close the loop by running tests themselves are more valuable when CI is fast and reliable)

Routing heuristic I’d use:

Terminal-native agent (Claude Code, Grok Build) → large refactors, regulated codebases, teams with strong review culture

Editor-integrated agent (Cursor) → everyday edits, model-agnostic orgs, teams resistant to workflow change

Bundled agent (Codex) → orgs already on ChatGPT Enterprise, async task delegation, broad rollout with minimal onboarding

Platform agent (Antigravity) → Google Cloud shops, teams building toward managed agent pipelines, orgs willing to accept preview risk

---

Layer 3: Lock-In Surface (what you’re actually committing to)

This is the layer most evaluations skip entirely and it’s the one that bites hardest.

Lock-in for agentic coding tools doesn’t live in the model (those are increasingly swappable). It lives in three places:

1. The instruction file

Every tool has one. Claude Code reads CLAUDE.md. Codex, Cursor, and others read AGENTS.md. This file encodes your team’s conventions: how to run tests, style rules, what not to touch. The deeper your conventions go into this file, the harder it is to move to a tool that reads a different format.

The builder move: Write your conventions in AGENTS.md format from day one, even if you’re currently on Claude Code. The Linux Foundation’s Agentic AI Foundation is pushing toward a single standard here, and being on the portable format now reduces migration risk later.

2. The review and approval workflow

Teams wire their PR templates, review checklists, and CI hooks around how their agent submits work. After three months, the workflow is the institutional memory not the tool. Changing tools means rewriting the workflow, not just the config.

The builder move: Document your approval workflow as a standalone artifact before you choose a tool. What does an agent-submitted PR look like? What does the reviewer check? What does the CI gate enforce? Make the workflow tool-agnostic, then choose the tool that fits it not the other way around.

3. The data surface

Some tools send repository context to the vendor’s servers during sessions. Others (Grok Build’s local-first claim, some Cursor configurations) keep it on the machine. For teams handling IP-sensitive code, health data, or regulated information, this is not a minor footnote.

The builder move: Map your data classification to the tool’s data flow before the pilot. What leaves the machine? What reaches the vendor? What does the vendor’s data retention policy say? This question is much easier to answer before you’re committed than after.

---

Layer 4: Governance Readiness (what you’ll need six months from now)

The fourth layer is the one that Platform PMs are uniquely positioned to see and that most developer-led evaluations miss entirely.

Agentic coding tools will eventually touch your most sensitive code paths. They will open PRs that go to production. They will run commands with real side effects. The governance questions that feel optional today become mandatory once scale kicks in.

The questions to answer before you commit:

• Audit trail: Can you reconstruct what the agent did and why? Not just the diff the full session, the plan, the approval chain?

• Access controls: What does the agent have permission to read and touch? Is that scoped correctly? Can you change it per-repo, per-team, per-task?

• Override and kill switch: If an agent session goes wrong. It’s deleting things it shouldn’t, it’s opening PRs to the wrong branches. Can you stop it cleanly?

• Model version pinning: If the vendor updates the underlying model, does your agent’s behavior change? Can you pin to a version for stability?

This last one deserves more space than it usually gets. Agentic tools are non-deterministic by design, the same prompt won’t produce the same output twice. But there’s a second layer of non-determinism that’s entirely outside your control: when the vendor ships a new model version, your agent’s behavior can shift in ways that aren’t announced in a change log. A plan it used to propose one way now looks different. A convention it used to follow reliably now requires a new AGENTS.md entry. Edge cases it handled conservatively now get handled more aggressively.

Teams without written assertions face the same problem every time this happens: weeks of informal re-testing, gut-feel sign-off, and hope. Teams with clear assertions “agent-authored PRs pass CI at X rate, plan rejection rate stays below Y%”, can validate a model version swap in days. That’s not a nice-to-have; it’s the difference between treating your agent like infrastructure and treating it like a demo.

None of these governance questions require all the answers before you start. But teams that don't ask them in the evaluation end up asking them in an incident post-mortem.

---

The AGENTS.md Question Is a Product Design Question

I want to end on this, because I think it’s under appreciated.

AGENTS.md (or CLAUDE.md, depending on your tool) isn’t a config file. It’s a product specification for how your agent behaves. It encodes:

• What the agent is allowed to do and what it isn’t

• How it should communicate its plans before acting

• What your codebase’s conventions are

• Where the dangerous zones are

Treating this like documentation written once, rarely reviewed produces an agent that behaves inconsistently as the codebase evolves and as the underlying model changes.

Treating it like a product artifact owned, versioned, reviewed when conventions change, informed by what agents got wrong in the last sprint produces an agent that gets better over time.

The builder framing: If you were building a new junior engineer role, you’d write an onboarding guide, a style guide, a list of “don’t touch without asking” systems, and a runbook for common tasks. AGENTS.md is all of that, for an agent. It deserves the same craft.

---

Coming in Part 3

We’ve covered the landscape (Part 1) and the evaluation framework (Part 2) above.

Part 3 is the rollout playbook: how to run a structured pilot with your engineering team, what governance to put in place before you scale, how to measure success beyond velocity, and the specific traps that burned teams in the first wave of agentic tool adoption.