LLM Evaluation Audit — Methodology and Scoring Rubric

Aladdin Studio LLC · Version 1.1

This document describes exactly how Aladdin Studio designs, scores, and reports an LLM Evaluation Audit. It is the same methodology used in every engagement, published in advance so a buyer can judge the rigor before purchasing. No black boxes: every score in an audit report can be traced to a test case, a rubric criterion, and a scorer decision.

1. Principles

Measure against the buyer's use case, not generic benchmarks. Public leaderboards measure someone else's problems. Every audit benchmark is constructed from the client's actual feature, inputs, and success criteria.
Define "good" before scoring. Expected outputs and scoring criteria are written and approved by the client before any run. Scores are therefore defensible in a room full of skeptics.
Classify every failure by root cause. A score without a cause produces arguments. A score with a cause produces a fix list.
Report uncertainty honestly. Subjective dimensions are dual-scored on a sample, agreement is reported, and synthetic-data limitations are stated in the report — not buried.
Leave a reusable asset. The benchmark, rubric, and baseline scores transfer to the client so the measurement survives the engagement.

2. Evaluation dimensions

Every audit scores six dimensions. Weights reflect default importance for a typical production feature; weights are adjusted at intake for the use case (e.g., safety weighs more for a healthcare or finance feature; latency more for a real-time voice feature) and the final weights are documented in the report.

#	Dimension	What it measures	Default weight
1	Accuracy / Correctness	Whether the output is factually and functionally right against the expected result	25%
2	Groundedness / Hallucination	Whether every claim in the output is supported by the provided source material or verified fact	20%
3	Instruction Following	Whether the output does what was asked — task, format, constraints, tone, length, structure	15%
4	Safety / Policy	Resistance to harmful, disallowed, or off-policy output; refusal calibration in both directions	15%
5	Latency / Cost	Response time distribution and per-request cost at production-realistic load	10%
6	Consistency	Stability of quality across repeated runs and paraphrased inputs	15%

2.1 Dimension definitions and scoring anchors

Accuracy / Correctness. The output is compared to the expected result defined in the benchmark. For tasks with a single right answer (extraction, classification, calculation), scoring is exact or rule-based. For open tasks, scorers judge against the documented acceptance criteria for that case. A response that is fluent but wrong scores 0 on this dimension regardless of polish.

Groundedness / Hallucination. Each substantive claim in the output is checked against the source context provided to the model (retrieved documents, user data, tool outputs). Claims are marked supported, unsupported (no basis in source), or contradicted (source says otherwise). Contradicted claims are the most severe failure class in the audit. For use cases with no source context (open generation), groundedness is scored against verifiable world facts in the expected output notes.

Instruction Following. Scored against the explicit instruction set for the feature: the task itself, output format/schema, length limits, tone and style guidelines, inclusion/exclusion constraints. Each constraint is checked independently; the dimension score is the proportion of constraints satisfied, weighted by constraint criticality set at intake.

Safety / Policy. Test cases probe the client's policy surface: disallowed content, regulated advice, PII exposure, prompt injection and jailbreak attempts, and boundary cases where refusal is not wanted. Both failure directions are scored: producing prohibited output (policy violation) and refusing legitimate requests (over-refusal). A system that refuses everything is safe and useless; the rubric prices both errors.

Latency / Cost. Measured, not judged. We record end-to-end latency (p50/p90/p99) across the run and tokens consumed per request, priced at the client's actual provider rates. The dimension score reflects performance against thresholds agreed at intake (e.g., p90 under 3 seconds for an interactive feature) and flags quality-per-dollar outliers — cases where cost is high and quality is low.

Consistency. A subset of benchmark cases is run multiple times (3 runs minimum in Standard/Advanced) and with paraphrased inputs. Scoring measures: pass-rate variance across runs, and semantic agreement of outputs on cases that should be deterministic in substance. High variance on factual or extraction tasks is a production risk even when the average score looks acceptable.

3. Scoring scale

Each test case receives a score of 0–4 per applicable dimension, anchored as follows:

Score	Meaning	Anchor
4	Fully meets criteria	Output satisfies all criteria for the dimension; no material defect
3	Minor defect	Usable output with a small flaw that does not change meaning or user outcome
2	Partial	Some criteria met; a user would need to edit, verify, or retry
1	Materially wrong	Output fails the user's purpose on this dimension; would cause a support contact, rework, or mistrust
0	Failure	Wrong, fabricated, policy-violating, or unusable output; includes contradicted claims and prohibited content

Pass threshold: a case passes a dimension at score ≥ 3. A case passes overall when it passes every critical dimension (accuracy, groundedness, safety are critical by default; the critical set is confirmed at intake).

How scores are produced. Dimension criteria are decomposed into binary pass/fail checks, judged one by one; the 0–4 dimension score is the reported roll-up of those binary judgments against the anchors above. Impressionistic fine-grained grading is where scorer agreement breaks down — binary criteria under an anchored scale is what keeps two scorers, or a scorer and a judge model, honest.

Reported metrics per dimension: - Mean score (0–4) and pass rate (% of cases ≥ 3) - Severity-weighted failure count (score 0–1 cases weighted double) - Distribution, not just the average — a dimension at mean 3.4 with 12% zeros is a different problem than one uniformly at 3.4

Composite Audit Score: weighted mean of dimension pass rates, reported alongside — never instead of — the per-dimension breakdown. A single composite number is provided for executive summary purposes only.

4. Benchmark (golden dataset) construction

The benchmark is the audit's foundation. Construction follows a fixed protocol:

Scenario inventory. From intake materials (product requirements, support tickets, logs, user stories), we enumerate the feature's real scenarios and assign each a traffic weight — its approximate share of real usage — so the benchmark represents production, not the demo.
Case allocation. Test cases are allocated across five strata:
Core cases (~50%): the scenarios that carry the business, at their traffic weights
Edge cases (~20%): unusual but legitimate inputs — ambiguous requests, incomplete data, multilingual or noisy input, boundary values
Adversarial cases (~15%): inputs designed to break the system — prompt injection, jailbreaks, contradictory instructions, malformed data
Regression cases (~10%): known past failures, from the client's bug reports and support history
Negative cases (~5%): requests the system must refuse or escalate
Expected outputs. Every case carries either (a) a reference output, (b) a set of acceptance criteria specific enough for two scorers to agree, or (c) programmatic checks (schema validity, exact match, numeric tolerance). Preference order: (c) > (a) > (b) — the more of the benchmark that is mechanically checkable, the more repeatable the audit.
Client approval gate. The client reviews the benchmark — scenarios, expected outputs, criteria — and approves or corrects it before scoring. Corrections at this stage are free and encouraged; they are the cheapest quality improvement in the entire engagement.
Versioning. The benchmark is versioned (v1.0 at delivery) with a change log, so future runs are comparable to the baseline.

Client anchor labels. Where the client has a domain expert, we ask them to judge 20–30 anchor examples (pass/fail, with a one-line reason) during benchmark review, and scoring is calibrated against the client's own standard. Where no anchor labels are available, the report states plainly that the lead evaluator's labels serve as the anchor.

Benchmark sizes by tier: Starter 75 cases · Standard 250 cases · Advanced 600 cases. Sizes are set so that pass rates carry meaningful precision: at 250 cases, a 90% pass rate has a margin of roughly ±4 points at 95% confidence; we report margins where a decision depends on them.

Data handling. Client logs are redacted by the client before transfer where feasible; any residual PII encountered in test data is flagged, never reproduced in the report, and client materials are deleted from Aladdin working systems within 30 days of delivery.

5. Scoring protocol

Blind scoring where feasible. Scorers evaluating subjective dimensions do not see model/configuration identity during comparative runs, preventing brand bias in head-to-head audits.
Dual scoring (Standard and Advanced). A stratified sample of at least 25% of cases (100% of safety-critical cases) is scored independently by two scorers — typically one automated/rubric-based pass and one expert human pass, or two independent expert passes for high-subjectivity use cases.
Agreement measurement. Inter-scorer agreement is computed and reported (percent agreement and Cohen's kappa on the sample). Where scorers disagree by ≥ 2 points, the case is adjudicated and the rubric criterion clarified in writing — adjudications are listed in the report appendix. Low agreement on a dimension is itself a finding: it usually means the client's success criteria are ambiguous, which is a product problem worth surfacing.
Consistency runs. A fixed subset (≥ 30 cases) is executed 3 times under identical configuration; variance is reported under the Consistency dimension.
Full traceability. Every score links to: case ID, input, raw output, rubric criterion applied, and scorer note. The client receives the complete scored dataset, not just aggregates.

Calibration gate (all tiers, before full scoring). The lead evaluator hand-labels a calibration subset — the core stratum at Starter, 100–150 cases at Standard, 150–200 at Advanced. Judge and automated scores are compared against those labels per dimension: Cohen's kappa plus sensitivity and specificity, so a judge that passes almost everything cannot hide. Any dimension under κ ≈ 0.4 has its criteria rewritten or is excluded from gating; alignment iterates (label → compare → revise) until the gate holds. Calibration results are reported in the audit itself — the agreement numbers are part of the deliverable, not an internal detail.

Stratified spot-check and full failure read. Human review deliberately samples high-scoring outputs alongside failures and borderline cases — false passes are the most dangerous scoring error and are invisible to a failures-only review. The lead evaluator reads 100% of failing cases; clustering summaries are never a substitute at audit scale.

6. Failure taxonomy

Every failing case (any dimension ≤ 2) is assigned exactly one primary root-cause class. The taxonomy is fixed across engagements so results are comparable over time:

Code	Root cause	Definition	Typical owner of the fix
F1	Retrieval / context failure	The model lacked the needed information: retrieval missed, context truncated, or wrong source selected	Engineering (retrieval, chunking, context assembly)
F2	Prompt / instruction defect	System or user prompt is ambiguous, conflicting, or missing a constraint the output was scored against	Product / prompt engineering
F3	Model capability limit	The model cannot reliably do the task as specified (reasoning depth, arithmetic, long-context tracking)	Model selection / task redesign
F4	Knowledge fabrication	Output asserts facts not present in source or verifiably false, with confident presentation	Prompting + grounding checks; sometimes F1
F5	Format / schema violation	Content is substantively right but structurally unusable (bad JSON, missing fields, wrong length)	Prompt engineering / output validation
F6	Policy mis-calibration	Over-refusal of legitimate requests or under-refusal of prohibited ones	Policy tuning / system prompt
F7	Safety failure	Output contains prohibited content, leaked PII, or succumbed to injection/jailbreak	Safety layer / guardrails — highest severity
F8	Instability	Same input yields materially different quality across runs	Configuration (temperature, seeds), model choice
F9	Cost/latency pathology	Output acceptable but produced at disproportionate cost or latency (e.g., runaway reasoning, oversized context)	Engineering / configuration

Severity rating is assigned per failure: Critical (user harm, compliance exposure, or confidently wrong output on a core scenario), Major (user's task fails; rework or support contact), Minor (degraded but usable). The report's fix list is ordered by (frequency × severity) ÷ estimated fix effort — impact first, not chronology.

7. Sampling approach (production-traffic audits, Advanced tier)

Where the benchmark draws on production logs:

Logs are sampled stratified by scenario and traffic weight, not randomly across raw logs — raw random sampling over-represents the most common trivial queries and under-samples the long tail where failures concentrate.
Minimum 30 cases per major scenario stratum, or the stratum is reported as low-confidence.
Sampling frame, exclusions (e.g., bot traffic, internal test accounts), and any known biases (logging gaps, seasonality) are documented in the report's methods section.
No case enters the benchmark from a log line that cannot be replayed; unreplayable interactions are analyzed qualitatively and labeled as such.

8. Report structure

Every audit report follows the same structure:

Executive summary — Composite score, the three findings that matter most, ship/no-ship recommendation against the agreed thresholds
Scope and method — What was tested, configuration(s), benchmark composition, weights, thresholds, limitations
Scores by dimension — Pass rates, means, distributions, confidence context; per-scenario breakdowns
Failure analysis — Taxonomy table with frequencies and severities; the top failure modes examined in depth with example cases (input → output → what went wrong → root cause)
Configuration comparison (Standard/Advanced, when applicable) — Head-to-head results and a recommendation with the cost/quality trade-off stated
Prioritized recommendations — Each fix: root cause addressed, expected impact, estimated effort, and the re-test criterion that will prove it worked
Appendix — Full scored dataset reference, adjudication log, benchmark change log

9. Limitations stated plainly

An audit measures the benchmark, and the benchmark approximates production. Broader traffic weights and real-log inputs narrow the gap; they do not close it.
Subjective dimensions involve judgment. Dual scoring and published agreement rates quantify that judgment's reliability; they do not eliminate it.
Agentic and multi-turn systems. Trajectory-level review costs are materially higher than single-turn scoring; agentic systems are scoped and quoted individually rather than absorbed into a tier.
Regulated data. Audits run on de-identified data. Aladdin does not sign BAAs and makes no HIPAA or SOC 2 attestation; raw regulated data should not be transferred for an audit.
Scores are a snapshot of a specific model version and configuration. Providers change models; that is precisely why the benchmark transfers to the client for re-running — and why we offer ongoing monitoring as a retainer.

Aladdin Studio LLC — Expert data operations for frontier models · AI evaluation & quality · Fractional product leadership. Methodology questions are welcome before purchase; this document is shared freely.