A coding agent moves up the leaderboard. Another slips a few places. Should your team switch?

A ranking offers a reassuringly simple answer to a messy engineering decision. Your codebase, tasks and review process rarely fit neatly into that table.

This week, we’re looking at how much those rankings can actually tell us—and what evidence we need before changing the tools we trust with our work.

The question to keep in mind over coffee: better on the benchmark—but better for which job?

Imagine your team is choosing between tools such as Claude Code, OpenAI Codex and Cursor. One posts a higher benchmark score. Is that enough reason to switch?

A study published on September 15 analysed 254 public submissions across four SWE-bench splits, using results retrieved on July 30. SWE-bench tests systems on resolving real GitHub issues.

On Verified, the researchers’ statistical tests could not distinguish any of the 29 neighbouring pairs among the top 30 entries.

That does not mean all 30 systems are equally capable. It means those results do not establish every step up the ranking as an improvement. This was an audit of published results, not a fresh comparison of today’s products.

Among the ten leading entries, 285 of 500 tasks were shared successes, 51 were shared failures and 164 had differing outcomes.

A neat ranking can make small differences look decisive. Before acting on it, ask whether the evidence is strong enough for the decision you are making.

The study also found that scores depend on the model and its surrounding agent software together.

Think of the distinction between the Claude model family and Claude Code, the coding tool built around it. The surrounding software—often called the harness—controls how the model gathers context, uses tools and works through a task.

A model’s result in somebody else’s setup therefore does not automatically predict its performance in yours.

Use public benchmarks to help build a shortlist. Then compare candidates on representative work from your own environment.

For example, if your team is considering Claude Code and Codex, give both the same representative tasks: a dependency upgrade, a bug spanning several files and a feature that must follow your architectural conventions.

Use consistent acceptance checks, record each tool’s model and configuration, and inspect the resulting changes. Compare how much correction and review each needs, alongside completion time and cost.

A small trial can reveal practical fit and obvious weaknesses. It should not become another leaderboard claiming precision its sample cannot support.

The study’s observational design cannot cleanly separate model capability from the engineering around it, and it omits run-to-run variation. On the larger Test split, 14 of 23 neighbouring pairs were distinguishable.

Benchmarks still provide useful evidence. The problem is asking a narrow score gap to answer a broader engineering question.

Before switching agents for a higher rank, check whether the difference survives contact with your work.

For years, Shopify used React Native to share much of its mobile-app code between iOS and Android. The appeal is straightforward: build a feature once rather than maintain two separate implementations.

In a September 10 engineering account, Shopify described rebuilding its Shop app in Swift for iOS and Kotlin for Android in 12 weeks, assisted by coding agents. The existing app gave those agents a concrete reference for recreating screens and behaviour.

Shopify says AI reduced implementation effort enough to reconsider the shared-codebase approach. Engineers still needed native-platform expertise, testing and review to catch architectural and performance problems.

Why it matters: some architecture decisions depend on how expensive software is to build and maintain. When those costs change, earlier choices deserve another look. This migration does not prove that separate native apps will be cheaper over their full lifetime for every team.

A customer-service chatbot might explain how to correct a name on a ticket. Completing that correction requires more: checking eligibility, updating booking systems and issuing the revised ticket.

Air India’s expanded use of Salesforce Agentforce moves into that execution work. Its workflows also handle emails containing several requests, gathering information across systems and taking permitted actions, with human oversight where required.

The companies report that an earlier refund workflow reduced Air India’s internal processing time from approximately 14 days to four hours. That does not mean money reaches the customer in four hours: bank processing time remains unchanged.

Why it matters: useful automation needs access to business systems, clear eligibility rules and a route for human intervention. The reported speed gains are encouraging, but they are company-reported and do not independently establish accuracy or customer satisfaction.

An AI assistant may pause while waiting for a person’s approval or a response from another service. Keeping its computing environment running during that wait consumes resources, especially when thousands of assistants are active.

Google’s Agent Substrate, available on Google Kubernetes Engine from September 15, lets teams suspend an agent’s execution and restore it later, preserving its memory and files.

Think of pausing a workspace instead of keeping it running—or rebuilding it from scratch.

But open network connections do not survive suspension. If the agent was connected to a database or external tool, the application must reconnect after resuming.

Why it matters: this could reduce wasted resources for teams operating many agents that spend time waiting. Non-production use is available to all GKE customers; production support requires allowlisting. Recovery behaviour still needs testing.

This week, I’ve been preparing a session on AI engineering roles. One useful question kept coming up: what does each role actually take responsibility for?

Titles such as Applied AI Engineer and Forward Deployed Engineer can sound like entirely new professions. They become easier to understand when we connect them to a concrete problem.

Imagine a company building an AI assistant for customer support:

The work spans reliable AI behaviour, integration into the customer’s environment and reliable operation of the underlying services.

These are overlapping responsibilities, not rigid boundaries. One engineer may cover several, and employers may use different titles for similar work.

For someone deciding what to learn, that distinction matters. Start with the responsibility you want to handle, then identify the skills it requires: evaluating answers, integrating systems, understanding a domain or operating services.

A useful career question is: “Which part of making AI work can I confidently take responsibility for?”

That gives you a more concrete learning direction than a job title alone.

If your team is starting to delegate work to AI, read Anthropic’s report on measuring AI development and oversight, particularly its section on monitoring agents.

It offers three useful questions:

These questions make “we have oversight” concrete. Checking an action after it happens may be too late if its consequences cannot be undone.

Anthropic’s figures describe its own systems, and monitoring coverage does not prove detection quality. Still, the questions are useful well beyond frontier AI labs.

The next time a coding tool climbs a leaderboard, try it on a task you know well.

For example, a tool that drafts a fix in five minutes but needs an hour of corrections may save less time than one that takes fifteen minutes and needs only a quick review.

Have you seen something similar with Claude Code, Codex or Cursor?

Reply with one example of what helped—or created extra work. I’d like to hear what the benchmark scores didn’t tell you.