LLM Evaluation Audit // Fixed-Price Offering
Led by Ernesto Rodriguez — Senior Product Manager at BCG (2022–2024) and AI Product Lead at Ascension Health (2024–2026), where he set AI use-case strategy and roadmap on GCP/Vertex AI and shipped agent-based workflows — patient messaging engagement up 10%. Now head of Aladdin Studio's frontier-model data operations and evaluation practice, including ongoing evaluation-data work for a frontier AI lab, under NDA. The people who build training and evaluation data for a living are the people scoring your feature.
[01] // The Audit
A fixed-price, fixed-scope engagement. A report you can act on — plus the benchmark asset that makes every future release measurable.
[DELIVERABLES]
[FROM YOU]
[02] // Pricing
Launch pricing is in effect for our first engagements in this format — standard rates apply once the launch cohort fills.
STARTER
$1,500 $995
STANDARD
$4,500 $3,250
ADVANCED
$9,000 $6,600
An audit is a snapshot — models drift, prompts change, traffic shifts. Monthly benchmark re-runs with drift reports, new failure cases folded in from your production logs, and ship/no-ship reads on release candidates keep quality measured.
from $2,500Per month · Month-to-month
Select your tier in the request form below. Timelines run from benchmark approval (US Central): Starter 5 days, Standard 14 days, Advanced 28 days. Benchmark approval is a walk-away gate — if the review doesn't convince you the test is the right one, you stop there: no charge beyond the benchmark stage, and the benchmark is yours to keep. One report revision round is included in every tier. If intake shows the engagement doesn't fit the tier, we tell you before work starts and requote — you approve any change before it happens. Direct engagements: 50% to start, 50% on delivery.
[03] // Process
Four steps. No surprises about what "good" means — you approve the benchmark before scoring begins.
[04] // Methodology
Every test case is scored 0–4 per dimension; a case passes a dimension at 3 or better. Standard and Advanced add dual scoring with agreement analysis — you see where judgment is consistent, not just a number.
Is the output right? Factual and task-level correctness against the expected result.
Is it supported? Claims traced to source material; fabrication and hallucination counted, not vibes.
Did it do what was asked? Format, constraints, tone, and task completion.
Does it stay inside the lines? Refusal calibration, PII handling, prompt-injection resistance.
What does quality cost? Response-time distribution and per-request cost at your volumes.
Does it hold up? Repeated and paraphrased runs — variance measured, not assumed.
Benchmarks are built from your traffic patterns and approved by you before scoring. Failures are classified into a nine-code root-cause taxonomy, so a retrieval problem is never mistaken for a prompting problem. The full methodology — rubric anchors, sampling protocol, taxonomy — is published: read the methodology and a complete sample audit report before you engage. Every engagement also opens with a calibration gate: the lead evaluator hand-labels a subset, judge agreement (Cohen's kappa, sensitivity/specificity) is measured, and the agreement numbers are reported in your audit.
[05] // FAQ
Prompt testing checks whether outputs look good. An audit measures whether outputs are good — against a defined benchmark, with failures classified by root cause and fixes ranked by impact. You also keep the benchmark: the asset that makes every future release measurable.
Model- and framework-agnostic: OpenAI, Anthropic, Google, open-weight models, and any RAG or agent stack — as long as we can run inputs through it, via API, a staging app, or batch file exchange.
Yes. We construct the benchmark from your use-case scenarios, product documentation, and any available support tickets or logs. Real inputs improve precision — and we'll tell you honestly where synthetic data limits confidence.
No. Your inputs, outputs, and materials are never used to train models and never shared with third parties. Everything is treated as confidential, used only for the audit, and deleted from our working systems within 30 days of delivery. NDAs welcome.
Then the audit did its job — you found out before your users did, with the failure breakdown and fix order in hand. Most first audits surface two or three fixable root causes behind the majority of failures.
Tools execute measurement — they don't decide what to measure. The hard part of evaluation is the benchmark: which cases, what "good" means for each one, and a failure taxonomy your team trusts. That's what the audit builds, and it hands the asset over in open formats (CSV/JSON benchmark, documented rubric) so your team can re-run it in any tool afterward. If you already have a trusted benchmark and criteria, you don't need us.
The audit is led personally by Ernesto Rodriguez: intake, benchmark approval, calibration, and the readout are his. Scoring runs execute on an automated harness under his direction; he hand-labels the calibration set, reads every failing case, and signs the report. Scoring is never subcontracted.
We work on de-identified data: inputs should be de-identified before transfer, and we offer de-identification assistance at intake if your team needs help. On that model we don't sign BAAs and we make no HIPAA or SOC 2 attestation — if your compliance requires a BAA over raw data, we'll tell you on the intake call that we're not a fit, before you've paid anything. Working files are deleted within 30 days of delivery, with a file inventory available on request.
[06] // Request
Tell us what your AI feature does and what concerns you about its output. We reply with a fit assessment and a tier recommendation.
