// selected work, written up

Case
studies.

A few things I built end to end, told through the product decisions behind them. Less "look what I made," more why it is built the way it is, and what the users actually did.

Prefer the interactive version?

Explore the portfolio →

FounderProduct, design, full-stack2025 → now

Measuring what actually sticks.

Wivme is an audio-first learning app I built, used by a handful of students so far. This is not a tour of features. It is the product decisions behind one idea: measure whether learning survives the week, not whether a kid showed up.

Role

Founder

Scope

Product, design, code, GTM

Surfaces

Student, parent, teacher

Stack

Flutter, Next.js, Go, RAG

Visit the site ↗

The problem

School measures whether you could recall something on one day. It does not measure whether it stuck.

A student crams, passes the Friday test, and has forgotten most of it by the next unit. Everyone can see the marks. Nobody can see the forgetting, the slow leak between "covered in class" and "still knows it a month later." That gap is where learning actually fails, and it is invisible to the exact people trying to fix it.

Wivme did not need more content. Textbooks, videos and apps already flood a student. What was missing was a measurement: a way to see decay as it happens and act on it before the test, not after. Every decision below falls out of that one reframe.

The market

The category sells content and measures attention. Neither is the thing that actually fails.

Indian edtech confused engagement with learning. It measured what it could bill for, watch-time, streaks, completions, and hoped that was the thing that mattered. It was not. An app can hold a kid for forty minutes and teach her almost nothing. BYJU'S rode that model from a 22 billion dollar valuation to insolvency, and roughly 70 to 80 percent of Indian edtech users churn inside the first month.

Fig. the market on two axes

Video-first appsBYJU'S, Vedantu, Unacademy

More content, scored by watch-time. A streak is not a skill and a completion certificate is not competence.

Quiz and doubt appsQuizlet, Doubtnut

Questions, but no model of forgetting. Review runs on a fixed timetable, not on when a concept is actually about to slip.

Spaced-repetition toolsAnki

The right science, but DIY flashcards with no curriculum and no audio. Built for a motivated adult, not a Grade 8 student on a shared phone.

Language appsDuolingo

Retention done genuinely well, but for languages, not the school syllabus a kid is graded on.

Nobody was measuring retention of the school syllabus and acting on the forgetting before the test. That gap is the whole product.

The research

The problem was never a content shortage. It was that nobody measured decay.

What the evidence says

67%
of new material is forgotten within 24 hours without review. Roughly 42 percent is gone in the first 20 minutes.
10 to 30%
better retention from spaced practice than from cramming, across a meta-analysis of 254 studies.
~150%
better long-term retention when spacing is paired with retrieval practice, testing yourself, instead of re-reading.

Ebbinghaus 1885, replicated by Murre and Dros 2015; Cepeda et al. 2006; testing-effect research.

So the build had to be organised around the forgetting curve, not the syllabus. Work out what each student is about to lose, and put it back the day before they would. Every decision below falls out of that one move.

The decisions

Five calls that define the product, and what each one said no to.

A product is the sum of what it refuses to do. These are the five that matter.

01Decision

Measure retention, not attendance.

The core metric is not minutes or streak of days. It is decay, tracked per concept. A spaced-repetition scheduler (an SM-2 variant) models each concept's forgetting curve and schedules the next touch just before the student would lose it, year-round, weekends included. Reviews are driven by decay, not by a timetable.

Said no to

"Days used" or "lessons completed" as the headline number. Activity is easy to grow and tells you nothing about whether it stuck.

Fig. why timed review beats cramming

timed reviews (coral marks)cram, then forget

02Decision

Audio-first, not video.

Lessons are three-minute audio episodes. Audio survives a cheap phone, a shared device and a noisy house, and it slots into the dead time of a day: the commute, the chores, the walk. Lower friction is the whole game, because the habit only compounds if it actually forms.

Said no to

High-production video. It looks better in a demo and needs a good screen, full attention and data most of these students do not have to spare.

03Decision

The check is a sensor, not a grade.

A short Quick Check after each episode turns a passive listen into a measurable memory event that feeds the scheduler. I kept it deliberately low-stakes. The goal is to read the memory, not to stress it, so the number stays honest and the student is not tempted to game it.

Said no to

Scored, ranked tests. The moment a check decides a rank, kids optimise the check instead of the learning, and the signal rots.

04Decision

The parent app shows mastery, not minutes.

Most parent apps report screen time. That is theatre. Wivme's parent view leads with a mastery trend, not a minute count, the one number a parent can actually trust. If the product's one claim is "we measure what sticks," the surface a parent sees every day has to prove it, not decorate it.

Said no to

Minutes-watched and streak badges as the hero stat. They make parents feel good and tell them nothing about learning.

05Decision

Gamify the loop, not the content.

Streaks, teams and a class leaderboard drive the daily habit. I kept that game layer firmly away from the learning itself: no points for right answers, no rewards riding on the check. Motivation is allowed to pull a student to the door. It is never allowed to bribe the measurement.

Said no to

Rewarding correct answers. It is the fastest way to feel engaging and the fastest way to corrupt the exact signal the product exists to protect.

Then the signal came in thin. A few students stuck with it, but the broad pull I was chasing was not there yet, and that was the most useful result I got.

06 , When the data talked back

The pivot

Reading the signal instead of arguing with it.

Thin traction is a demand signal, not a detail to argue with. The decision was not to push harder on a motion that was not working. It was to find where the pull actually was: the sharpest pain was not the student's, it was the teacher's, in prep, checking, and knowing who really gets it. Same engine, re-aimed at teacher prep.

I would rather kill a good idea than defend a wrong one.

The through-line

Wivme is not a finished company. It is a working thesis: that the valuable layer in education is measurement, not more content. I built it end to end, three apps and a RAG engine, got it in front of real students, and stayed ready to redraw the roadmap when the signal said to.

The method underneath every decision here: find the invisible problem, turn it into something you can measure, instrument the honest path so using the product well produces your best data, and keep the nerve to follow that data even when it kills your first plan.

Back to

Case studies →

Inventor and builderArchitecture, engine, patent2026

AI tutoring you can reproduce.

Reactive AI tutoring is racing to zero. The version with a moat is the boring one: the engine that gives the same answer twice. RQSM turns any document into an identical, auditable study session on every replay. It is the basis of a published patent.

Role

Inventor, builder

Patent

IN 202641086881 A1

Core

Deterministic orchestration

Stack

Python, FastAPI, React

View the code on GitHub ↗

The problem

An LLM tutor is easy to demo and impossible to trust.

Ask it to teach the same chapter twice and you get two different lessons. You cannot audit it, cannot guarantee it, cannot build a curriculum on top of something that changes its mind every time you refresh. For a school, "usually good" is not a product.

The insight is that the moat is not the model, it is the orchestration. Take the control flow, the decision of what happens next, out of the model and put it in a deterministic layer. Reproducibility and auditability stop being features you bolt on and become properties you get for free.

The market

Every AI tutor on the market is a wrapper on a model that changes its mind.

Ask ChatGPT, Claude, or a school’s own GPT tutor to teach the same chapter twice and you get two different lessons. That is fine for a demo and disqualifying for anything a school has to stand behind.

Fig. the market on two axes

General LLM tutorsChatGPT, Claude

Fluent, but non-deterministic and un-auditable. Same prompt, different answer, every refresh.

Edtech LLM tutorsKhanmigo and similar

Guardrails bolted on top, but the control flow still lives in the model. You still cannot reproduce or defend a given session.

Classic rule-based tutorsolder ITS

Reproducible, but rigid and scripted, with none of the language a real lesson needs.

Nobody had both: real language, and a session you can replay line for line.

The research

Non-determinism is not a bug you can patch out. It is how these models work.

What is actually true about LLMs

By design
Sampling makes outputs vary. Even at temperature zero, floating-point and server-side batching cause drift.
Always
Hallucination is a direct consequence of the sampling objective, not a random glitch. Bounding it helps; removing it does not.
Auditable
Reproducibility is the prerequisite for trust. You have to be able to hand someone the log and defend every line.

LLM sampling, determinism and reproducibility research, 2023 to 2026.

So the bet was to move the control flow out of the model and into a deterministic layer. Reproducibility and auditability stop being features you bolt on and become properties you get for free. That bet is the basis of the patent.

The decisions

Four calls that make it a system, not a prompt.

Determinism is not an accident. It is a set of refusals.

01Decision

Control flow lives in a state machine, not the model.

The model generates language. A rule-based state machine decides which role speaks, when, and what comes next, driven by the document structure. Same input, same transitions, same output, run after run. The intelligence is in the orchestration; the model is a swappable part.

Said no to

Letting the LLM decide the next step. It is the one choice that makes everything downstream unreproducible.

Fig. reactive vs deterministic, three runs each

02Decision

A fixed cast of five roles, not a free agent.

Every session is run by the same five pedagogical roles. Fixing the cast is what makes the behaviour legible: a teacher can look at a transcript and know exactly which role produced which line, and why. An open-ended agent that invents its own behaviours is impressive in a demo and unaccountable in a classroom.

Said no to

An improvising "do anything" agent. Flexibility you cannot explain to a school is a liability, not a feature.

Explainer

breaks the concept down

Challenger

pushes back, probes gaps

Summarizer

locks in the takeaway

Example-Gen

grounds it in a case

Misconception

catches the wrong model

03Decision

Change roles only past a confidence gate.

An intent classifier reads each student turn (clarification, challenge, elaboration, misconception, topic-shift) and can reallocate roles. But it only switches past a confidence threshold, and hysteresis holds the new state so a single noisy message cannot make the tutor thrash between roles. Stability is a feature you design in, not hope for.

Said no to

Reacting to every message. A tutor that flips personality on one ambiguous reply feels broken, however smart each flip is.

Fig. hysteresis on the intent signal

04Decision

Auditable by construction.

Because control flow is explicit, every session is a trace: the exact sequence of roles, transitions and intents that produced it, replayable step for step. You can hand a school the log and defend every line. That is the difference between "the AI said it" and a system you can stand behind.

Said no to

Black-box generation. If you cannot explain why the tutor did what it did, you cannot sell it into a classroom.

Trace · session replay

01Explainersegment 1.1 introduced

02Example-Gengrounded in a worked case

03Misconceptionintent=misconception, conf 0.82 › switch

04Explainerre-explains the corrected model

05Summarizerlocks the takeaway

Why it matters

Anyone can wire an LLM to a chat box. The durable layer is the part that makes the output reproducible, stable and auditable, and that layer is orchestration, not the model underneath it. RQSM is a bet that the boring, deterministic version is the one schools can actually build on.

It is the same instinct as Wivme: the value is not more AI, it is a system you can trust and measure.

Back to

Case studies →

BuilderVR, gameplay, AI grader2025

A fire drill you can fail.

A real fire drill is theatre: everyone ambles out, nobody fails, nothing is measured. INFERNO is a VR fire drill with haptics and an AI grader that scores your escape against the actual protocol. You do not pass by reaching the exit. You pass by doing it right.

Role

Builder

Scope

VR, gameplay, grading

Model

Real building, to scale

Stack

Unity, C#, Blender

View the code on GitHub ↗

The problem

A drill that no one can fail teaches no one anything.

Stand up, file out, tick a box. A traditional fire drill has no stakes and no feedback, so it rehearses the walk but never the decisions: which door, how fast, what to avoid. When the real thing happens, muscle memory is a slow amble toward the nearest, sometimes wrong, exit.

The reframe: treat evacuation as a skill you can score. If you can measure a run, you can fail it, coach it, and watch it improve.

The market

Fire training is either a walk with no stakes or a slide deck nobody remembers.

A real drill has everyone amble out. Nobody fails and nothing is scored. Passive lecture or video training is worse, people keep about 5 to 10 percent of it. When a real fire happens, muscle memory is a slow walk toward the nearest, sometimes wrong, exit.

Fig. the market on two axes

Traditional fire drillsthe standard

A real building, but no measurement, no failure, no score. You pass by walking out.

Lecture, slide and videocompliance training

Cheap and passive. Retention sits around 5 to 10 percent.

Generic VR safety modulesoff-the-shelf

Immersive, but graded on completion, not against the building’s actual evacuation protocol.

Nothing let you fail, and nothing measured how you did against the real protocol.

The research

Training sticks when it has stakes, a real rubric, and room to get it wrong.

What the training research shows

up to 75%
more retained from active VR training than from passive lecture or reading, which holds about 5 to 10 percent.
80s to 54s
average evacuation time in a virtual-drill study, with safety scores climbing alongside it.
Not a swap
Real drills still matter. INFERNO adds what a drill cannot: failure, a score, and feedback that sticks.

VR safety-training and virtual evacuation-drill studies, 2020 to 2025.

So it had to grade against the building’s official evacuation protocol, and let you fail. You do not pass by reaching the exit. You pass by doing it right.

The decisions

Three calls that turn a walk into training.

Stakes, a real rubric, and feeling that makes it stick.

01Decision

You pass by getting it right, not by reaching the exit.

Reaching a door is not success. The grade is on the whole run: did you take a safe route, move at the right pace, avoid the smoke. Two people can both survive and score worlds apart, and the score is the point.

Said no to

A pass/fail on "did you get out." It rewards luck and teaches nothing about the choices that actually save you.

Fig. two survivors, two very different runs

02Decision

Grade against the real protocol, not vibes.

An AI grader scores each run on route, timing and efficiency, measured against the building's official evacuation protocol. It is not a feeling of "that went well." It is a number you can defend, compare, and hand to a safety officer.

Said no to

A vague completion screen. If the standard is not the real protocol, the training quietly teaches the wrong thing.

Grade · one run

03Decision

Use haptics only where they teach.

Haptics are not for spectacle. They fire exactly where a real body would learn: a hot door handle you should test before opening, the rising heat that tells you to stay low. Feeling the wrong choice is what makes the right one automatic under stress.

Said no to

Constant rumble for immersion. Noise everywhere means the one buzz that should change your behaviour gets ignored.

Haptic cues, where they matter

Why it matters

The moment a drill produces a score, it stops being a formality and becomes training you can improve. Same instinct as everything else I build: the value is not the simulation, it is the measurement it makes possible.

Back to

Case studies →

BuilderVR, therapist toolingUnity, C#, OpenXR

Facing fears, on a dial.

Exposure therapy works, but it depends on staging the exact thing a patient fears, at the exact intensity they can just about handle. Xposure is a VR tool I built to put that dial in a therapist's hands: the fear is virtual, so it can be turned up a notch at a time and measured throughout.

Role

Builder

For

Therapists, exposure therapy

Treats

Public speaking, agoraphobia, social anxiety

Stack

Unity, C#, OpenXR, Node

View the code on GitHub ↗

The problem

You cannot summon a crowd, a stage or an open square on demand in a therapy room.

Exposure therapy is one of the most effective treatments for phobias, and it hinges on graded exposure: face a controlled version of the fear, slightly harder each time, until the body stops treating it as a threat. But real-world stressors are expensive, unrepeatable and impossible to fine-tune. A therapist cannot dial a crowd from twenty people to fifty, or add just a little more noise.

The reframe: if the fear is virtual, its intensity becomes a control the therapist can turn.

The market

Exposure therapy works. Staging the exact fear at the exact intensity is the hard part.

The treatment depends on giving a patient the specific thing they fear, at a level they can just about handle, and nudging it up as they settle. Real life does not come with a dial.

Fig. the market on two axes

In-vivo exposurereal-world, gold standard

The benchmark, but expensive, unrepeatable and impossible to fine-tune. You cannot turn a crowd from twenty people to fifty, or add just a little more noise.

Imaginal exposurein the mind

Controllable, but only as vivid as the patient’s imagination. Weak exactly where it matters.

Fixed VR levelslevel 1 to 5

Immersive, but a menu the patient picks, not a dial the clinician turns in the moment.

Nobody handed the therapist a real-time intensity dial, with the session measured.

The research

If the fear is virtual, its intensity becomes something you can turn.

What the evidence says about VR exposure

As effective
VR exposure matches in-vivo exposure head to head, with no significant difference in outcomes.
Large effect
Strong results against waitlist controls, and often more acceptable to patients than real-world or imagined exposure.
Control
The documented edge of VR is the ability to control the quality, intensity, duration and frequency of exposure.

VR exposure-therapy meta-analyses for anxiety and phobia.

So the tool is built around the gradient: crowd size, noise, open space, each a dial the therapist turns live, with the session tracked. The patient is inside the fear; the clinician stays in control of exactly how much of it there is.

The decisions

Three calls that make it a controllable tool, not a VR demo.

The therapist steers, the software measures.

01Decision

The therapist holds the dial, not the patient.

Crowd size, background noise, open space: the therapist can adjust each live, mid-session, to read the patient and nudge the intensity up or down. The patient is inside the fear; the clinician stays in control of exactly how much of it there is.

Said no to

A fixed "level 1 to 5" the patient picks. Exposure has to track a real person's tolerance in the moment, not a menu.

Live intensity controls

02Decision

Grade it, do not just present it.

The point is not to drop someone into their worst fear. It is systematic desensitization: start where they can cope, hold until the anxiety fades, then add a little. The tool is built around that gradient, so intensity climbs across sessions only as fast as the patient's tolerance does.

Said no to

Shock-factor immersion. Overshoot once and you reinforce the phobia instead of shrinking it.

Graded exposure across sessions

03Decision

Measure the response, not just the minutes.

Every session logs how long the patient stayed in the environment and how they responded, so progress is something you can see, not just feel. Anxiety trending down while time-tolerated trends up is the whole treatment, made legible for patient and therapist alike.

Said no to

"That felt better." Recovery you cannot chart is recovery you cannot steer or prove.

Session tracking

Why it matters

Xposure does not replace the therapist; it hands them a slider for something that was never adjustable before. Same instinct as everything else I build: take a thing people only judge by feel, and turn it into something you can control and measure.

Back to

Case studies →

Builder, maintainerOpen-source dev tooling2026

A prototype is easy now. A product you can actually ship is not.

Prompt-to-app tools hand you a demo, then leave you at the wall: no spec, no plan, no proof it works, and the project locked to their platform. Keel is an open-source AI founding team that does the research, the spec, the design and the architecture, gates the plan on feasibility, then drives Claude Code or Cursor to write and ship the real, tested, secured code.

Role

Builder, maintainer

Core

An AI founding team, idea to ship

Method

Research, gate, build, prove

Stack

Python, Claude Code, MCP, Z3, MIT

View the code on GitHub ↗

The problem

AI writes the code now. It still guesses at everything around the code.

A coding agent will generate a working screen in a minute. What it will not do on its own is tell you whether the thing is worth building, what the real spec is, what to build first so nothing depends on a part that does not exist yet, or whether "done" is actually done. It writes fast and guesses at the rest.

Prompt-to-app tools go the other way. They hand you an impressive prototype, then the wall: no product spec, no build plan, no proof the thing works, and the whole project locked inside their platform. Either way the judgment that decides whether a build ships, the part a founding team does, is the part that gets skipped.

That gap is where most software dies: built without knowing what to build, shipped with mock data and broken permissions, or never shipped at all.

The market

The space is crowded at both ends and empty in the middle.

Prompt-to-app builders like Lovable, Bolt, v0 and Replit optimise for the first five minutes: a demo from a sentence. Raw coding agents like Claude Code, Cursor and Codex write code fast but take no view on what to build or whether it is right. Spec-driven kits add a plan, then stop before the build and the proof. Nobody runs the whole path, research to shipped, on the tool you already pay for, on code you keep.

Fig. the build space on two axes

Prompt-to-app buildersLovable, Bolt, v0, Replit

A demo from a sentence, then the wall. No real spec, no plan, no proof it works, and the project is locked to their platform.

Raw coding agentsClaude Code, Cursor, Codex

Write code fast, and guess at what to build and whether it is right. No feasibility gate, no written definition of done.

Spec-driven kitsGitHub Spec Kit and similar

Add a plan before the code, a real step forward. They stop there: no business case, no design, no security proof, no build loop that ships.

The empty middle is the whole path run end to end, on the tool you already own, on code you keep. That is where Keel sits.

The research

The expensive failures are not typos. They are building the wrong thing, and shipping code that leaks.

What the research shows

35%
of failed startups cite no market need for what they built, among the most common reasons in a teardown of 110-plus post-mortems.
~36%
how often a 20-step build at 95 percent per-step reliability completes correctly. Order and dependency are not optional; error compounds.
#1
broken access control is the top category on the OWASP Top 10, and the one class of bug ordinary scanners cannot catch, because it needs to know who is supposed to see what.

CB Insights, The Top Reasons Startups Fail (2021); OWASP Top 10 (2021), A01 Broken Access Control; per-step reliability is a compounding-probability calculation.

So the design was never "generate code faster". It was to make the founding-team judgment, what to build, in what order, and proof that it holds, the enforced part, and let the AI tool do the typing.

The decisions

Four calls that make it a founding team, not a code generator.

The value is not the code. It is the judgment around the code.

01Decision

Drive the tool you already own. Do not become another platform.

Keel is layered: the rulebooks, specs and pipeline are just Markdown; the tracker, the quality gates and the security proof are just Python and run with no AI at all; only the top layer binds to a tool, and it ships an AGENTS.md so it works with Cursor, Codex, Aider or Claude Code. Best in Claude Code, locked to none.

Said no to

A hosted app-builder. Lock-in is the business model there, and the moment you outgrow the demo you cannot take the project with you.

yours to keep, MIT

02Decision

Gate the plan on feasibility before a line of code.

A feasibility-auditor reads the business, product and technical plans together and returns GO, REVISE or NO-GO: is the product coherent with the business, is it buildable with your tools, can you afford to run it. On REVISE it names exactly what to fix. The build does not start until all three plans agree.

Said no to

Build-first, fix-later. The cheapest bug to fix is the feature you decided not to build.

GO / REVISE / NO-GO

03Decision

Enforce production quality with gates that fail the build.

Most AI tools hand back mock data, TODOs and happy-path functions and call it done. Keel refuses, through gates that fail the build: no placeholders, no cross-boundary imports, no code reaching into what it does not own, and a code-reviewer that catches the missing error state a linter cannot. Every weakness it finds arrives with the fix that closes it.

Said no to

Trusting the model to police itself. Quality by good intentions is quality you cannot ship.

demo code vs ship code

04Decision

Prove security. Do not scan for it.

Keel ships trespass, a formal analyzer for Postgres and Supabase row-level security. It compiles every policy into logic and either proves no user can reach another user’s rows, or hands you the exact query that shows they can. Written on the standard library, validated against Z3 and a real Postgres, it fails the build when a policy leaks.

Said no to

A pattern-matching linter. Access control is about who is supposed to see what, which a grep cannot know. A proof can.

a proof, not a guess

The finding

Building three products this way taught the same lesson three times: the code was never the bottleneck.

Keel is the process I wished I had while building the other things on this site. Wivme, Plumbline, Beachhead, each one, the hard part was never getting the AI to write code. It was deciding what to build, in what order, and proving it actually worked before anyone relied on it. Keel is that judgment, made repeatable.

What it is, honestly

Not magic
It does not remove judgment. It structures the founding-team judgment and refuses to let you skip it.
Only as good as
the AI tool it drives and the decisions you feed it. Keel is the scaffolding and the gates, not the intelligence.
Early
Published and MIT-licensed on PyPI as keel-kit, and young. It needs more real projects run end to end before its gates have seen everything they should.

Keel: open source, MIT, PyPI keel-kit, github.com/Bhargs24/keel.

The through-line

What it is, and honestly what it is not.

Anyone can generate an app now. Almost no one ships a real product from a sentence.

That gap, between a demo and something you can put in front of real users, is what Keel is built to close: the research, the spec, the plan, the gates and the proof, wrapped around the coding tool you already use, on code you own. It is not a hosted platform and not a magic button. It is founding-team judgment made repeatable, and given away.

Back to

Case studies →

Builder, researcherLLM agent reliability2026

The check your agent skips when you reword the request.

Your agent passed its evals, which means it gave acceptable answers on a fixed test set. It does not mean it will still run your duplicate-payment check when the same request arrives worded differently. Plumbline measures that, and names the step where it breaks.

Role

Builder, researcher

Core

Metamorphic conformance testing

Method

Declare, perturb, localise, certify

Stack

Python, 2,082 trajectories, Apache-2.0

View the code on GitHub ↗

The problem

An LLM agent is easy to demo and impossible to trust twice.

Evals score an agent on a fixed set of inputs. They do not test whether it still runs its critical controls when the same request is reworded, when a tool blips, or when an irrelevant sentence is added. The failure this is built around: an accounts-payable agent pays an invoice without running the duplicate check. The payment succeeds. The confirmation is byte-identical to a correct run.

Nothing in an output-level eval will ever show it. The damage is invisible at the output layer, so the test has to move down to the trajectory and name the step where the control was skipped.

The market

The agent-evaluation space is crowded and heavily funded. It is also solving a different problem.

Patronus AI raised 50 million dollars for world models that stress-test agents in simulated environments. Braintrust raised 80 million, Arize has raised over 130 million, and Cisco folded Galileo into Splunk. Every one of them does scenario generation, finding new inputs that break you, and trace scoring on fixed inputs. None of them documents the opposite test: whether you break on inputs that should not break you.

Fig. the space on two axes

Eval platformsPatronus, Braintrust, Arize

Scenario generation and trace scoring on fixed inputs. They find new inputs that break you. They do not test invariance to inputs that must not.

Governance frameworksAgent Certified, ISO 42001

A layer above, feeding audit and insurance. They need an instrument to produce the evidence. Closer to a consumer than a rival.

Success and output metricsthe default

Compare the final answer, or which tools ran. A payment of 48,200 instead of 4,820 runs the identical tools in the identical order. Invisible.

Nobody had declared-invariant conformance with argument-level comparison against ground truth. That gap is the whole contribution.

The research

Rewording a request, something a correct agent should be indifferent to, degrades it measurably.

What the research shows

96.9 to 88.1%
the drop in agent success from a minor task rewording, measured on a reliability benchmark.
~36%
how often a 20-step workflow at 95 percent per-step reliability completes correctly. Compounding is worse than it looks.
Aug 2026
the EU AI Act accuracy and robustness rules for high-risk systems become enforceable, with harmonised standards still unfinished.

ReliabilityBench (arXiv 2601.06112); Consistency as a Testable Property (arXiv 2605.10516); EU AI Act Article 15.

So the design goal was never another accuracy score. It was to hold the scenario fixed, vary only the surface form, and certify what a run must never do, with the exact step named when it does.

The decisions

Four calls that make it an instrument, not another benchmark.

A test is only as honest as what it refuses to score.

01Decision

Declare the invariants. Do not infer them.

The spec is the contract a run cannot violate: MustCall the duplicate check, Ordering the match before the payment, CallAtMost one payment, ArgEquals the amount to the one in context, each with a declared severity.

Said no to

Scoring the final answer. A correct-looking confirmation can sit on top of a skipped control, byte for byte.

declared, not learned

02Decision

Perturb what must not matter.

Reword the request, with an independent model verifying the meaning is preserved. Inject a transient tool fault. Add irrelevant context, add decoy tools, sweep the temperature. The scenario is held fixed; only the surface form varies.

Said no to

Generating new scenarios. That is a different axis. This tests whether the agent is indifferent to changes it should be indifferent to.

invariance, not coverage

03Decision

Compare the arguments, not the tool names.

Sequence alignment recovers the real edit script, so a skipped control reports as a skip, not a substitution of whatever ran next. Arguments are compared field by field against ground truth. This is the technical contribution.

Said no to

Comparing which tools ran. A payment of 48,200 instead of 4,820 runs the identical tools in the identical order and returns an identical receipt.

the invisible failure

04Decision

Certify a number you can defend.

Report the 95 percent lower confidence bound on critical-invariant conformance under the worst perturbation, with provenance and an evidence hash, rebuildable from the stored traces.

Said no to

A pass or fail stamp. An agent is only as certified as the perturbations someone thought to apply, and the report says so.

a bound, not a badge

The finding

The first result was a clean, significant lie. I almost shipped it.

I built this to measure what moving an agent’s control flow out of the model actually buys. Under an injected transient tool failure, the deterministic executor reached the correct outcome 81.2 percent of the time against the free-form agent’s 98.2 percent. A 17-point gap, p = 0.0029. Determinism looked worse.

Then the obvious. No production finance system treats a single 503 as fatal; every RPA platform has a retry policy. My executor had none. I added one and re-ran the identical 768 trials. The gap vanished, 100 percent against 98.2 percent, not significant. Three lines of retry closed the whole thing.

What it is actually evidence for

Not determinism
It is evidence about benchmark construction, not about whether determinism wins or loses.
The stats were right
The confidence intervals, the permutation test and the control condition were all correct. The baseline was the strawman.
Only domain knowledge
A plausible, well-visualised 17-point effect was entirely an artifact of an unrealistic baseline. Statistical rigour does not catch that. Product knowledge does.

Plumbline study: 768 trials per arm, 95% Wilson intervals and permutation tests, 2,082 trajectories committed.

The through-line

What it is, and honestly what it is not.

Every published agent benchmark has a baseline somebody chose.

This is what it looks like when that choice is the whole result. Plumbline is not a startup and not a finished product. It is an instrument that produced a finding, kept the strawman as a permanent control arm so the effect can be reproduced and attributed, and is published by the person holding a deterministic-orchestration patent that the result complicates. That is discussable, and it is not dismissible.

Back to

Case studies →

Casestudies.

Measuring what actually sticks.

Measure retention, not attendance.

Audio-first, not video.

The check is a sensor, not a grade.

The parent app shows mastery, not minutes.

Gamify the loop, not the content.

AI tutoring you can reproduce.

Control flow lives in a state machine, not the model.

A fixed cast of five roles, not a free agent.

Change roles only past a confidence gate.

Auditable by construction.

A fire drill you can fail.

You pass by getting it right, not by reaching the exit.

Grade against the real protocol, not vibes.

Use haptics only where they teach.

Facing fears, on a dial.

The therapist holds the dial, not the patient.

Grade it, do not just present it.

Measure the response, not just the minutes.

A prototype is easy now. A product you can actually ship is not.

Drive the tool you already own. Do not become another platform.

Gate the plan on feasibility before a line of code.

Enforce production quality with gates that fail the build.

Prove security. Do not scan for it.

The check your agent skips when you reword the request.

Declare the invariants. Do not infer them.

Perturb what must not matter.

Compare the arguments, not the tool names.

Certify a number you can defend.

Case
studies.