LLM products fail quietly. I build the evals that catch them.

I'm Zoeb Nomi. At Instead — an AI-native tax research and planning platform, the first new entrant to clear IRS e-filing approval alongside incumbents — I own end-to-end output quality for a production tax-research LLM: citation accuracy, model benchmarking, and the evaluation loops that catch regressions before release. In tax research, a wrong citation isn't a UX bug. It's a compliance risk.

Book a call Email LinkedIn

0.994

citation precision under strict citation discipline (CrossSource v0.1; baseline 0.981)

100% (15/15)

blind human–judge agreement validating the LLM judge

~95%

citation accuracy held on a production golden set

CrossSource

An open evaluation harness for citation accuracy in legal RAG. 22 public court opinions, a 25-question golden set, an LLM judge validated blind against a human — and a real harness bug caught by that validation.

Read the case study
Table 1 — baseline vs. strict
Dimension BaselineStrict
Citation precision 0.9810.994
Citation recall 0.7600.760
Faithfulness 1.0001.000
Answer relevance 0.9800.960

THE FINDING THAT MATTERS

Prompting discipline buys precision — it cannot buy recall. 0.760 in both configurations, because every missing-authority failure traces to retrieval, not generation.

  • Instrument

    Flag → classify by failure mode → weekly review, scored on a four-dimension rubric: answerability, accuracy, citation quality, actionability.

  • Taxonomize

    A failure taxonomy instead of a single score — so you know whether the fix belongs to prompting or to retrieval.

  • Validate the judge

    An unvalidated eval reports wrong numbers with full confidence — precisely the failure mode the eval exists to catch.

If you're shipping an LLM product, let's talk about what your evals miss.

Book a call