Earlier this month, Databricks published the results of an internal benchmark evaluating coding agents against real engineering tasks on their codebase containing millions of lines. It is one of the most rigorous public looks at how AI coding tools actually perform on production work, and if you lead an engineering team, it is worth reading in full.
For us at Qubika, reading it felt familiar. We have spent the last year and a half delivering software with coding agents for clients across healthcare, fintech, automotive, and media, sprint after sprint, across dozens of codebases that look nothing like each other. Along the way, we arrived at the same conclusions Databricks just confirmed with data, not by running a benchmark but by shipping production code.
This article covers what we learned in delivery, what we built because of it, and how Databricks’ findings map to what we see every day.
What agent assisted delivery taught us
Three lessons kept repeating across projects, regardless of stack or industry.
The most expensive model is rarely the right default. Most of what a delivery team ships in a sprint, including CRUD endpoints, config changes, test suites, and UI components that follow an existing pattern, does not need the most intelligent model available. It needs a model that knows the codebase conventions and a process that verifies the output. Teams default to premium models out of habit, not need, and the waste compounds silently across every task, every developer, every day.
The real cost of an agent is not the token price. It is repeated exploration. Watch an agent work on a real codebase, and you will see where the money goes: reading files it read yesterday, rediscovering the architecture it already traversed, and rebuilding context that evaporated when the session ended. The models became cheaper; the redundant exploration did not.
Speed without stability is a trap. Here is something we measured that surprised us: on frontend work, agents actually move faster when there are no finished designs to match. Freed from pixel perfect constraints, the model stops iterating against a mockup and just ships. But that raw speed comes with sharply lower stability. Two runs of the same ticket produce two different interfaces, conventions drift, and review costs eat the time you saved. Raw acceleration is easy to get. Stable acceleration is the actual engineering problem. No amount of model intelligence fixes it, because the variance does not live in the model. It lives in what the agent knows and what it is allowed to do on every single run.
The layer that decides everything: the harness
An AI model only generates text. The harness is the software around it that turns text into work. It decides which files the model sees, which commands it can run, what quality gates apply, and how much context gets sent again on every turn. Claude Code, Codex CLI, and Cursor are all harnesses. It is the same engine in a different car, and as it turns out, the car defines most of the fuel consumption.
Every harness on the market operates at the task level: one developer, one prompt, one change. But the logic generalizes. If the harness is the layer that controls what context the model sees and what it is allowed to do, the same discipline applies to the entire delivery cycle, from spec to merged pull request.
That is what we built.
QAF: a harness for the SDLC
The Qubika Agentic Framework, or QAF, is, in effect, a harness for the full software development lifecycle. It runs on top of Claude Code or Codex, across Anthropic, OpenAI, or any other model provider, because the leverage lives in the harness layer, not in any single model. It is implemented entirely in code, so you can audit, modify, and adapt every capability described below.
Each of the lessons above is a design decision in QAF:
Against repeated exploration: the agent knows the codebase before the first ticket. On initialization, QAF builds a Code Graph, a deterministic Tree Sitter map of every function, file, and dependency that is rebuilt in seconds before every run, and an LLM Wiki, an automatically generated, committed knowledge base of architecture and conventions that agents read instead of rescanning the repo. Context is precomputed once and shared by every developer instead of being rediscovered by every agent session.
Against expensive defaults: the model matches the task. QAF assigns specialized agents to each step of the cycle and routes every agent to the model tier its risk profile requires. For example, a frontier reasoning model plans and reviews high risk architectural changes, while a faster, cheaper model implements the code and writes the tests. The expensive model is a deliberate escalation, not a default.
Against instability: quality gates stand between generated code and an open PR. Linting, type checks, test generation with coverage thresholds, and a security review are run before a pull request opens. The developer reviews and merges; the framework does everything up to that point the same way, every time.
The results on a real project
We recently applied QAF from end to end on a healthcare platform we are building for a US client. We tracked estimated hours, based on traditional role based estimates, against actual hours logged per ticket, excluding bugs and newly added scope from the calculation.
In a recent sprint, the team delivered the planned work in less than half the traditionally estimated hours, representing a 2.1x aggregate acceleration. The split is where it gets interesting: our full stack lead ran at roughly 3.2x, while frontend ran at 1.4x, consistent with the pixel perfect effect described above. Our best increment, a persistence and resume feature spanning backend and frontend, shipped at roughly 4x, in about a quarter of its estimated time.
At the roadmap level, the foundation phase closed with 33% fewer hours than projected, and the first delivery phase finished on schedule and under plan while pulling work from the next phase forward into the same window.
Two structural consequences followed. Backend and frontend roles merged into full stack ownership of complete feature increments, and QA shifted toward a hybrid QA plus business analyst profile. The acceleration did not just change velocity; it changed the shape of the team.
One caveat we insist on with every client: these multipliers are not a constant you can plug into any roadmap. Every project is its own world. Acceleration depends not only on the complexity of the solution itself, but also on the integrations and external dependencies the team has to work through, including third party APIs, legacy systems, vendor sandboxes, and compliance constraints. An agent cannot accelerate a partner’s certification timeline. That is precisely why we measure per project instead of promising a number upfront.
Then Databricks published the data
Databricks’ benchmark, built from their engineers’ real merged PRs and graded by the original test suites rather than an LLM judge, reaches the same conclusions we reached through delivery. Four of them, specifically:
-
- Most engineering tasks don’t need the most capable model. Models cluster into capability tiers, and the default should be the cheapest tier that passes your quality bar — with the expensive model as an escalation.
- Cost per task is the metric — not cost per token. Token price is a poor predictor of what a completed unit of work actually costs.
- The harness is the dominant cost lever. What context the model sees, and how much of it gets re-fed on every turn, moves cost more than model choice does.
- Single-vendor lock-in is a new form of technical debt. No single provider covers the cost/quality frontier; the architecture must allow swapping models and harnesses.
Here is how each maps to their data and to ours.
They found that models cluster into capability tiers and that most daily engineering tasks do not need the top tier. Their task telemetry showed that roughly a quarter of what engineers asked coding agents to do involved low complexity work, while about 60% involved medium complexity work. Their takeaway is to push more work to cheaper models. This maps directly to QAF’s model routing, which we arrived at from the opposite direction.
They found that token price is a poor predictor of task cost. Sonnet 5 is about 1.7x cheaper per token than Opus 4.8, yet on their tasks it cost more per completed task, $2.09 compared with $1.94, while scoring six points lower because it worked longer and read more to get there. Cost per outcome, not cost per token, is the number that matters. It is the same reason we track hours per shipped feature rather than API spend.
Most strikingly, they found that the harness alone can change task cost by more than 2x in some cases, with identical quality. Running the same model through different harnesses, the efficient one sent roughly three times less context per turn because it kept a tighter working set instead of sending the model everything on every step. That is precisely the observation that led us to precompute codebase knowledge in QAF: the single biggest lever in agentic development is controlling what the model needs to see.
Their final conclusion matches a bet we made early in QAF’s design: model choice is only one piece of the puzzle, and locking into a single model or harness is a new form of technical debt. Their cost and quality frontier includes models from multiple providers; QAF is provider agnostic for exactly that reason.
What this means for your team
If there is one transferable idea in both their work and ours, it is this: measure agents on your own work, in your own context. Public benchmarks cannot tell you how a model performs on your codebase, your conventions, or your definition of done. Databricks measured on their merged PRs. We measure on client delivery, sprint by sprint. Both approaches beat adopting whatever model is trending.
And if standing up that measurement discipline, including the context layer, model routing, quality gates, and tracking, sounds like a project your team does not have room for, that is the problem we solve. It is the difference between buying tools and engineering outcomes.
See QAF in action
QAF is not a lab experiment. It is how our teams deliver production software for clients. Discover what agentic delivery could look like on your codebase.
Questions and answers:
What is a harness in AI coding agent development?
A harness is the software layer that sits between an AI model and a codebase. It decides which files the model sees, which commands it can run, what quality gates apply, and how much context gets resent on every turn. Claude Code, Codex CLI, and Cursor are all harnesses. The model inside them only generates text. The harness is what turns that text into shippable work
Why does the harness matter more than the AI model for coding agents?
Because the harness controls what the model knows and what it is allowed to do on every run, and that is what determines cost and stability, not raw model intelligence. Databricks’ internal benchmark found that running the same model through different harnesses changed task cost by more than 2x with identical output quality, since the more efficient harness sent roughly three times less context per turn. Qubika reached the same conclusion independently, through client delivery rather than a benchmark
What is the Qubika Agentic Framework (QAF)?
QAF is Qubika’s harness for the full software development lifecycle, not just a single coding task. It runs on top of Claude Code or Codex across Anthropic, OpenAI, or Google models, and assigns specialized agents to each step of delivery, from spec to merged pull request. It precomputes codebase knowledge through a Code Graph and an LLM Wiki so agents do not explore the same files again every session, and it is code-based, so its design decisions can be audited directly. QAF is part of how we deliver under AccelerateAI, our framework for AI assisted software delivery.
How much can AI coding agents accelerate software delivery?
On a recent healthcare platform project, Qubika’s team delivered a sprint’s planned work in less than half the traditionally estimated hours, a 2.1x aggregate acceleration. Results varied by role: the full stack lead ran at roughly 3.2x, frontend at 1.4x. Qubika treats these multipliers as specific to each project rather than a number that transfers, since acceleration depends heavily on third party integrations, legacy systems, and compliance constraints outside the agent’s control.
Why can a cheaper AI model cost more per completed coding task?
In Databricks’ benchmark, Claude Sonnet 5 was about 1.7x cheaper per token than Claude Opus 4.8, yet cost more per completed task, $2.09 versus $1.94, because it took longer and read more context to reach the same result. This is why cost per completed task, not cost per token, is the metric that actually predicts what a coding agent costs an engineering team.
Does an AI agent framework need to lock a team into one model or vendor?
No. QAF is designed to work with any provider and can run Anthropic, GPT, or Gemini models depending on the tier of intelligence a task requires, escalating to a frontier model only for architectural work that carries high risk. Qubika treats dependence on a single vendor as a form of technical debt, since no single AI provider currently covers the full cost to quality frontier. You can see how this plays out in our Agentic Factory work.
