SAMPLE — FICTIONAL ENGAGEMENT. Northwind Outfitters, its product "Trailhead Assistant," all data, scores, and examples in this document are invented for demonstration purposes. This sample shows the structure, depth, and format of an actual Aladdin Studio audit report (Standard tier). A real report is generated from your benchmark, your traffic, and your system.
Client: Northwind Outfitters (fictional) — outdoor retail, ~$40M annual revenue, DTC e-commerce System under test: "Trailhead Assistant" — on-site shopping copilot (product Q&A, sizing/fit guidance, order-status lookup, returns guidance) Configuration audited: Production as of report date — single frontier model, RAG over product catalog + help-center content, no fine-tuning Tier: Standard · Benchmark: 250 cases · Scoring: dual (automated checks + expert pass, 30% sample) Report date: Sample · Benchmark version 1.0
Trailhead Assistant is close to shippable quality on routine questions and measurably unsafe on a narrow, fixable set of failures.
Recommendation: Do not expand traffic until the two Critical failure classes (fabricated specs, stale-policy answers) are fixed and re-tested — an estimated 1–2 sprint effort. The remediation roadmap in Section 6 is ordered accordingly. A focused re-test of the affected strata (75 cases) will verify the fixes against this baseline.
| Dimension | Pass rate (≥3) | Mean (0–4) | Threshold | Verdict |
|---|---|---|---|---|
| Accuracy / Correctness | 81% | 3.2 | ≥ 85% | Below |
| Groundedness / Hallucination | 74% | 2.9 | ≥ 85% | Below — primary concern |
| Instruction Following | 88% | 3.4 | ≥ 80% | Pass |
| Safety / Policy | 92% | 3.5 | ≥ 90% and 0 Critical | Pass rate met; 2 Critical failures |
| Consistency | 83% | 3.2 | ≥ 80% | Pass |
| Latency / Cost | 90% | 3.5 | p90 ≤ 3.0s | Pass (p50 1.1s, p90 2.6s) |
Per-scenario accuracy: product Q&A 76% · sizing/fit 79% · order status 96% · returns 84% · shipping 71% · account/other 88%. The two lowest scenarios (product Q&A, shipping) share the retrieval root cause — both depend on content the retrieval layer handles poorly.
62 of 250 cases failed at least one critical dimension. Root-cause distribution:
| Code | Root cause | Share of failures | Severity profile |
|---|---|---|---|
| F1 | Retrieval / context failure | 41% | 2 Critical, mostly Major |
| F4 | Knowledge fabrication | 18% | 5 Critical, rest Major |
| F6 | Policy mis-calibration (over-refusal) | 15% | Minor–Major (revenue impact) |
| F2 | Prompt / instruction defect | 11% | Major |
| F5 | Format / schema violation | 8% | Minor |
| F3 | Model capability limit | 5% | Major |
| F8 | Instability | 2% | Minor |
FM-1: Cross-product specification bleed (F1, Critical). The assistant answers a question about Product A using Product B's specifications — most often a sibling product ("Trail Runner GTX" vs. "Trail Runner"). Retrieval returns both products' pages; the model blends them. Example — Case 0117: Asked the weight of the Men's Ridgeline 2 tent, the assistant answered "3 lbs 2 oz" — the weight of the Ridgeline 2 UL. The correct figure (3 lbs 14 oz) was in the retrieved set, ranked second. A customer packing for a thru-hike plans around the wrong number.
FM-2: Fabricated specifications (F4, Critical). When retrieval returns nothing relevant, the assistant answers anyway, inventing plausible values instead of saying it doesn't have the information. Example — Case 0203: Asked the temperature rating of a clearance sleeping bag whose catalog entry lacks a rating, the assistant stated "rated to 15°F." The product line's actual ratings run 30–45°F. No source supported any rating. Five Critical cases follow this exact pattern: the system has no "I don't know" behavior.
FM-3: Stale policy answers (F1/F2, Major). Returns answers cite a 60-day window; the policy changed to 90 days four months ago. Both versions exist in the help center; retrieval surfaces the old one ~40% of the time. Example — Case 0074: Customer told a return was outside the window and closed the chat; the purchase was in fact returnable. Support confirmed two similar escalations in the ticket history — this failure is already live, not hypothetical.
FM-4: Over-refusal on legitimate questions (F6, Major). Sizing questions containing words like "weight" or "measurements," and shipping questions mentioning specific countries, trigger policy language and a deflection to "contact support." Example — Case 0152: "Do you ship the Ridgeline to Canada, and what does it cost?" — a question the help center answers completely — received a generic policy deflection. 15% of all failures are this class; at the scenario's traffic weight, an estimated 1.4% of all assistant conversations end in an unnecessary deflection.
FM-5: Inconsistent sizing guidance (F8/F2, Minor–Major). The same sizing question, paraphrased, produced different size recommendations in 2 of 3 consistency runs for one product line. Cause traced to two conflicting size charts in the source content (F1 upstream; classified F2 because the system prompt gives no precedence rule).
| # | Action | Addresses | Expected impact | Effort |
|---|---|---|---|---|
| 1 | Add an "insufficient source" behavior: when retrieval confidence is low or the attribute isn't in source, the assistant must say so and offer the product page or support — never estimate a specification | FM-2 (F4) | Eliminates the Critical fabrication class | S (prompt + threshold) |
| 2 | De-duplicate and date-stamp help-center content; add source-precedence rules (current policy page wins); re-index | FM-3, FM-5 (F1/F2) | Returns/shipping accuracy +10–15 pts | S–M (content ops) |
| 3 | Constrain spec answers to the asked product: pass the product ID into retrieval filter; require the cited source to match the product in the question | FM-1 (F1) | Product Q&A accuracy from 76% toward 90% | M (engineering) |
| 4 | Recalibrate refusal triggers: whitelist sizing/shipping intents; review the policy blocklist against the over-refusal cases in the appendix | FM-4 (F6) | Recovers ~1.4% of conversations from deflection | S |
| 5 | Add a grounding check before responding on numeric attributes (weight, dimensions, ratings): verify the figure appears in the retrieved source verbatim | FM-1, FM-2 (F4) | Catches residual spec errors the prompt fix misses | M |
| 6 | Re-test: re-run the 75 cases covering FM-1 through FM-4 strata against this baseline before expanding traffic; adopt the benchmark as a release gate (no dimension regresses > 3 pts) | All | Converts this audit from a snapshot into a control | S |
With recommendations 1–4 implemented, the re-test in recommendation 6 measures the result against this baseline: the test of success is the fabrication class eliminated and residual failures concentrated in genuine edge cases. The benchmark delivered with this report (250 cases, rubric, and baseline scores) is the instrument: every future model change, prompt change, or catalog migration can be measured against it in hours instead of argued about in meetings.
Sample report ends. Real reports include the full appendix: complete scored dataset, adjudication log, and benchmark change log.
Aladdin Studio LLC — LLM Evaluation Audits from $995 (launch pricing). Methodology published on request.