AI Agent Consulting
KeppyLab Consulting
Verifiable agents for companies that aren't AI companies.
A production agent on your actual workflow in 2–4 weeks, with an eval harness included, so you know whether to trust it.
The Problem
You already suspect an AI agent could take real hours out of your week: document intake, ops triage, reporting, support. But most of what's for sale is either a demo that works once, or an agency retainer that never ends. And standing up an ML team to find out is overkill.
The Offer
1 · 2–4 weeks
The Pilot
Fixed scope. Pick one workflow. You get:
- A working agent wired into your actual tools, not a slide deck
- An eval harness: an automated test suite that scores the agent against your real cases, so you know its accuracy before you rely on it
- A plain-English score report and handoff docs
- A go/no-go recommendation, including "don't automate this" if that's the honest answer
2 · 6–10 weeks
The Full Build
A production agent system for a bigger surface.
- Multiple connected workflows, built to run unattended
- 30 days of post-launch tuning included
3 · Monthly
Fractional AI Engineering
Ongoing advisory plus hands-on build.
- A senior AI hand for teams that don't need a full-time one
- PE firms: this shape spans multiple portfolio companies
All engagements are fixed-scope and quoted after a 30-minute scoping call.
Why The Eval Harness Is The Whole Point
Every AI demo works once. The question that matters is: out of a hundred real cases, how many does it get right — and does it know when it's unsure? An eval harness answers that with numbers instead of vibes. It's the difference between a party trick and a system you can put in front of customers. Most agencies don't ship one. KeppyLab doesn't ship without one.
So, here's a real example. Everything below came out of gonogo, the harness I use on implementations, which I've open sourced. I pointed it at a plain baseline classifier and 250 customer messages from Banking77, a public dataset of real support requests to a retail bank. Nothing here is client work, and the classifier is nothing special on purpose.
Watch what happens to it: seventy-seven percent sounds like a decent start. Then the interval tells you the real rate might be 71.6%, and once you respect the interval, no confidence threshold clears a 95% bar anywhere you put the cut. Verdict: don't ship this yet. It's better to hear that in week two than the week after it starts touching customers.
Route customer message to the right queue
ASSIST ONLY
Use it to draft, keep a human on every case.
Pass rate 77.2% [71.6%, 82.0%] is well short of the 95% target and no confident subset reaches it; useful as a draft-generator, not as an unattended step.
- Cases evaluated
- 250
- Passed
- 193
- Pass rate
- 77.2% [71.6%, 82.0%]
- Target
- 95%
- Calibration error
- 0.42 (ranks cases, scale unreliable)
The pass rate carries a 95% confidence interval. With 250 cases the true rate could plausibly sit anywhere in that range, which is why the interval and not the headline number drives the decision.
| Confidence floor | Handled | Precision | To review |
|---|---|---|---|
| 0.03 | 100% | 77.2% [71.6%, 82.0%] | 0 |
| 0.10 | 85% | 81.7% [76.0%, 86.3%] | 37 |
| 0.18 | 70% | 85.1% [79.1%, 89.7%] | 75 |
| 0.26 | 57% | 88.0% [81.7%, 92.4%] | 108 |
| 0.39 | 40% | 90.0% [82.6%, 94.5%] | 150 |
| 0.51 | 27% | 94.1% [85.8%, 97.7%] | 182 |
| 0.66 | 14% | 97.2% [85.8%, 99.5%] | 214 |
| 0.96 | 0% | 100.0% [20.7%, 100.0%] | 249 |
| Stated confidence | Cases | Mean confidence | Actual accuracy |
|---|---|---|---|
| 0.0-0.2 | 83 | 0.11 | 61% |
| 0.2-0.4 | 74 | 0.30 | 78% |
| 0.4-0.6 | 50 | 0.51 | 84% |
| 0.6-0.8 | 30 | 0.70 | 97% |
| 0.8-1.0 | 13 | 0.87 | 100% |
Caveats
- Calibration error 0.42 exceeds 0.15: the confidence score ranks cases usefully but its scale is not a probability. Treat any threshold below as an opaque cut point, and recalibrate before reading it as a percentage.
The full report lists the cases it got wrong. Your pilot ends with one of these, run against your work instead of a benchmark.
So we swapped in a frontier model
The obvious next move, and the one most teams make: throw a better model at it. I pointed Claude Opus 5 at the identical 250 messages, scored the same way, and ran the harness again. Accuracy went from 77.2% to 81.6%. On most dashboards that is the end of the story: a win, write it up.
But because both models saw the same cases, the comparison can be tested properly, and the honest read on those 4.4 points is that they're inside the noise: the difference carries a 95% interval of −0.8 to +9.6 points and fails significance at p = 0.14. Roughly seven hundred cases would be needed to say something meaningful. A team reporting "we upgraded and accuracy improved" would be reporting a coin flip. The same model rerun on the same 250 cases scored 80.8% one day and 81.6% the next, so a fifth of the reported "improvement" was the model failing to be deterministic.
| Baseline classifier | Claude Opus 5 | |
|---|---|---|
| Pass rate | 77.2% [71.6%, 82.0%] | 81.6% [76.3%, 85.9%] |
| Calibration error | 0.42 | 0.06 |
| Handled automatically at ~98% precision | 14% | 50% |
Accuracy is a wash. The share of the queue you can safely hand over more than triples.
The gain is in the bottom two rows, and neither is accuracy. The baseline's confidence score was close to meaningless — it would say 0.11 and be right 61% of the time. Claude's is honest almost to the decimal: it says 0.92 and is right 93% of the time. Honest confidence is what lets you draw a line and route everything below it to a person. So the same measurement that found no provable accuracy gain found a 3.6× increase in how much of the queue you can automate at the same error rate.
If we measure and report properly, we can get a better understanding of the decision surface we are dealing with. That is the entire argument for measuring properly. The number everyone reaches for moved by an amount that means nothing. The number that decides how many hours actually leave your week moved by a factor of three. You cannot see either one without the interval and the coverage curve, and you would have shipped the wrong conclusion about both.
Neither run cleared a 95% bar, so neither ships unattended — and the honest report says so. The closest Claude gets is 97.6% precision on half the cases, where the lower bound lands at 93.2% and misses the target by under two points. Very likely good enough. Not yet provable on 250 cases.
How It Works
-
Scoping call 30 min, free
We pick the one workflow with the best effort-to-value ratio.
-
Fixed quote
Scope, timeline, and price in writing. No hourly meter.
-
Build
Weekly demos on your real data, not mock data.
-
Eval + handoff
Score report, docs, training for your team, and honest next-step advice.
About
James Dominguez — KeppyLab is the applied-AI practice of James Dominguez, a senior AI/ML engineer and product leader.
- Ran the technical org at a venture-backed startup as Head of Product & AI: post-training, evaluations, synthetic data, and agentic systems, shipped in production
- Designed a reasoning-RL post-training program for a Qwen3-based model — LLM-as-judge evaluation harness, ~60% benchmark gains
- Built agent pipelines generating synthetic training data and curriculum (14K examples at 85% compression)
- Previously: production AI for a Johnson & Johnson–funded medical-imaging company; EIR at the AI2 Incubator
- 10+ years full-stack engineering (Python / Go / Rust / TypeScript)
- Open research in AI evaluation: github.com/keppy