Here's the short version, because it's the part that changed how we build. We gave two frontier models, Claude and Gemini, the same failing battery cell and the same raw cycling data, with open-ended code execution. The electrochemical reasoning was genuinely good. The output was a static, mis-scaled image we couldn't export, and rerunning the same prompt produced a different analysis every time.
To answer one question, the model rebuilt an entire analysis pipeline from scratch, every time it ran.
Then we changed exactly one variable. We took the weaker model (Gemini, which got about a third of the findings right on its own) and stopped letting it write code. Instead, it could only call a library of validated, domain-specific functions. On the same data, it went from 1-of-3 to 4-of-4, and the results were exportable and reproducible.
The lesson, stated plainly: in a domain where the answer feeds a real decision, the harness around the model matters more than the model. Below are the exact failure modes we hit, why the harness fixes each one, and a blueprint for building your own.
Optional cookies required for the embedded video
The on-page player uses a third-party YouTube embed. Accept optional cookies to load the video here, or open it directly on YouTube instead.
The test
Two cells from the same build, one healthy and one down on performance, plus their full charge/discharge and cycling data. The prompt was concrete and multi-part, the kind of thing a cell engineer does on a normal Tuesday:
- Compare the discharge profiles and capacity retention
- Plot the differential capacity (dQ/dV)
- Remove obvious outliers
- Explain why one cell is failing
Same prompt, same data, two frontier models, general-purpose code execution, no scaffolding. This is how most people use these tools today: open a chat, attach the data, describe the analysis, let the model write and run whatever it decides it needs.
This wasn't a benchmark. The question was narrower and more practical: drop a state-of-the-art model straight onto real lab data, is the result something you could put into an engineering workflow?
What worked: the electrochemistry was real
The reasoning surprised me, in the right direction.
The stronger model tied the capacity gap between the two cells to the effective C-rate and built a coherent internal-resistance story around it: mechanism, not correlation. If you read only the written explanation, you'd trust it and act on it. For anyone who's watched general LLMs hand-wave through domain questions, that's a real shift: the explanation is no longer the weak link.
Which makes where it breaks all the more instructive.
Where it broke: three failure modes worth memorizing
1. Correct, but not exportable
The deliverable came back as a static image with mis-scaled axes. Not adjustable, not exportable, not something you could rescale against a spec, drop into a report, or interrogate further. A picture of an answer instead of an answer you can work with.
In a lab that distinction isn't cosmetic. A plot you can't rescale is a plot you can't read against a spec. A result you can't export dies in the chat window. "Correct but unusable" isn't a partial win, for a working engineer it's a dead end with extra steps.
2. A different answer every run: the disqualifier
We reran the same task with slightly different wording and got a materially different path through the data, not a rephrasing of the same conclusion, a different analysis. Same cells, same numbers, different answer depending on how the question happened to be phrased and what the model chose to write that time.
This is the one that ends the conversation for a real workflow. "Validated" mostly means same inputs, same result, every time. An analysis that reasons itself fresh on every run is the opposite of a standard operating procedure. When the output feeds a scrap decision, a supplier conversation, or a design change, correct-but-irreproducible fails the same test a drifting instrument would.
3. The wrong-column bug: why domain context is non-negotiable
This is the most instructive failure, and every cell engineer will recognize it.
Cycler exports (Arbin, Neware, Biologic) often carry a column literally labeled Discharge Capacity that is actually cumulative discharge capacity across cycles, not per-cycle. The open-loop model grabbed that column at face value and built the capacity-retention plot on it, without checking. The result looked plausible and was completely wrong, and every downstream conclusion inherited the error. The same run also mishandled the voltage profile: it didn't enforce that the discharge capacity axis starts from zero, using native timestamps instead.
Neither is a reasoning failure. They're data-provenance failures: the model didn't know which column meant what, and nothing stopped it from guessing. A human expert catches both on reflex. An unconstrained model has no reflexes.
The fix, and more usefully, how it works
We took the weaker model and changed one thing: instead of writing arbitrary code, it could only call a library of vetted, domain-specific functions, validated implementations of the dQ/dV analysis, the capacity-retention calculation, the outlier handling, the resistance-trend logic. The model stopped writing the analysis and started selecting and composing correct analyses.
| Finding | Gemini, open-loop | Gemini + vetted tools |
|---|---|---|
| Voltage profile (start-from-zero) | ✗ wrong | ✓ correct |
| Capacity retention (right column) | ✗ wrong column | ✓ correct |
| dQ/dV curves | not attempted | ✓ correct |
| Failure-mode attribution | not attempted | ✓ correct |
| Score | ~1 / 3 | 4 / 4 |
Why each failure mode disappears, mechanically:
- The wrong-column bug can't happen. The vetted capacity-retention function already knows which column is per-cycle vs cumulative. The choice was made once, by a human, and tested.
- The output is a real artifact. The tool returns exportable data and a properly scaled figure, not a screenshot.
- It's reproducible. The same request routes to the same validated code, so the same inputs produce the same result regardless of phrasing or which engineer asks.
The uncomfortable takeaway: the constraint mattered more than the model. A frontier model is a very high-output engine. Run it open-loop and it spins freely. The harness, the fixed set of validated tools it dispatches to, is what couples that capability to repeatable, controllable, exportable work.
A blueprint: build the harness for your own domain
If you're deploying an agent anywhere the answer has consequences, here's the concrete version of the lesson. None of this is battery-specific.
- Inventory the analyses that must be correct. For each recurring, decision-feeding calculation, write a tested function with a fixed implementation. The model composes these; it doesn't reinvent them. Start with the 10 to 20 operations that cover most of your real work.
- Make the schemas strict: encode the domain into the interface. Types, units, and expected data semantics (which column is per-cycle vs cumulative, that capacity starts at zero, valid voltage ranges). The tool should reject or flag bad inputs rather than silently guess. This is where the wrong-column class of bug dies.
- Prefer retrieval/selection over free generation. Let the model choose from vetted functions (and, where you must generate code, generate against validated building blocks), rather than writing analysis from a blank editor.
- Build an eval harness with golden datasets. Keep a set of inputs with known-correct outputs, including the traps (a file with the cumulative-capacity column, an outlier-laden profile). Every tool and model change runs against it. This is what would have caught the wrong-column plot before a human trusted it.
- Handle coverage gaps loudly. When a request falls outside the library, the right behavior is to flag it for a human, not to silently freehand a novel method into a production answer. Make "I don't have a validated tool for this" a first-class response.
- Make results traceable. Log the exact tool calls and versions behind every output. Reproducibility isn't just "same answer twice," it's being able to point at the operation that produced a number six months later.
When you don't need this: the honest trade-off
Constraining a model has real costs: you build and maintain the library, and it can't do what you haven't pre-built. So be deliberate:
- Open-ended code-gen is fine for exploration, one-off questions, and low-stakes work where a wrong answer costs you nothing and you'll eyeball the result anyway. It's genuinely great for that.
- You need a harness the moment an answer (a) feeds a decision, (b) has to be reproducible, or (c) runs more than a handful of times. That's most of real R&D and all of production.
The mistake isn't using frontier models. It's handing one a blank code editor for a job in the second category and calling reasoning quality the finish line.
Why this generalizes: the harness is the durable part
Model choice is increasingly a commodity, the rankings churn quarter to quarter. The scaffolding is what's durable: strict tool schemas, a validated function library, retrieval over vetted code, and an eval harness. And a good harness is portable: it lifted a weaker model to parity here, which means it carries forward as the models underneath keep improving. You're not betting your workflow on one vendor's model; you're building the thing that makes any capable model behave.
How Niobia is built around this
This is the thesis Niobia AI runs on. Under the plain-language interface isn't a model turned loose on your data, it's a library of validated materials-science, electrochemistry, and process-analysis tools the model dispatches to, with the schemas, evals, and traceability above baked in. That's what lets it return results that are reproducible, exportable, and traceable to the exact operation that produced them, the same whether one engineer asks or another, today or next month. The model supplies reasoning and flexibility; the harness supplies trust. In a field where a wrong answer feeds a real decision, you need both, and the second is the part almost nobody is building.
Caveats
This is a scoped internal test on our own data and tasks, not a formal benchmark, and model behavior shifts across versions. The point isn't which model "won," it's where the leverage sits. Rerun it next quarter with newer models and we'd expect the reasoning to be even better, and the failure modes (unusable output, irreproducibility, data-provenance errors) to persist until something constrains them.
About the author
Dr. Gaurav Jha is the Founder of Niobia AI, which builds AI-powered process intelligence for battery manufacturing and advanced materials. His PhD focused on fast-charging niobium pentoxide (Nb₂O₅) nanostructured anodes, with broader research across gas sensors, ion sensors, and energy-storage materials. At Intel he worked on wet-etch defect reduction in 5nm and 7nm fabrication; he then developed one of the first large-scale lithium-sulfur cathode coatings at Lyten before moving to Sila Nanotechnologies to work on silicon-anode particles for high-energy-density, fast-charging cells across consumer and automotive programs.