AI Eval measures whether your AI features actually work. You write down what a good answer looks like, and the module asks your agent or model the questions, scores every response, and reports a pass or fail verdict against a quality gate you configured. The same evaluation you run from the browser also runs from Drush, so a prompt change that quietly breaks your assistant can fail your CI pipeline instead of reaching users.

It evaluates in two modes. Agent mode exercises AI Agents plugins end to end, the way a real request would. Chat mode sends prompts directly to any provider supported by the AI module, so you can compare models and prompts without an agent framework. Both modes share the same datasets, graders, scoring, and reporting.

One loop, not a pile of tools

Everything in the module belongs to a single quality loop. A target names the system under test, its dataset of questions, its graders, and its gate. Running it produces per-question scores and a verdict. From there you work on the two questions that follow any measurement: why does it fail, and can the scores be trusted?

For the first, you annotate failing answers with a verdict, a note, and a failure mode from a taxonomy you grow as you learn (for example hallucinated_fact), and the module tracks failure rates per mode. You can also import real production conversations as OpenTelemetry GenAI traces, review them grouped by conversation, label them, and promote the failures into your datasets, which turns every production incident into a permanent regression test.

For the second, the module takes LLM-as-judge skepticism seriously. Every LLM judge has a gold-labeling screen where you label real traces from its domain yourself, and a validation run that measures how often the judge agrees with you, separately on positives and negatives (TPR and TNR). Judges earn a trust state that goes stale as it ages, and each validation records the prompt version it was measured against so you know to re-validate after editing a judge. An untrusted judge never blocks anything, but every verdict it contributed to is marked provisional wherever it appears, including in exported results.

Once measurement is trusted, an automated optimizer closes the loop: it reads the failures, proposes an improved system prompt, evaluates the candidate over repeated runs against a noise margin, and applies it or queues it for human review.

What you get

A single admin section under Configuration > AI with a portfolio view of every target's health, an htmx-driven UI for running evals with live progress, dataset and rubric authoring (YAML files, content entities, or module-shipped config), synthetic question generation, trace review and promotion, per-judge validation and prompt tuning, failure-mode analytics, and an optimizer review queue. Alongside it, a set of Drush commands covers running, optimizing, trace import, judge validation, and export, with exit codes meant for CI gating.

The statistics are kept honest. Pass rates always carry Wilson 95% confidence intervals, runs warn when a dataset is below the Cochran sample size floor for the claims you are making, and when a single trusted judge graded a run the module reports a bias-corrected pass rate that accounts for that judge's known error rates. Results export as portable EEE (Every Eval Ever) JSON envelopes for comparison across sites, and the datasets, rubrics, and judges you author have a portable form of their own: the module ships JSON Schemas (Draft 2020-12) that define the eval-content format, so the same dataset validates across sites and can be checked in CI before a run.

Graders are plugins: five LLM judges (accuracy with optional ground-truth facts, relevance, completeness, actionability, fact match) and two deterministic checks (format, tool usage) ship in the box, and your module can add domain-specific graders with one attributed class.

Getting started

Requires Drupal 11.3, PHP 8.3, and the AI module. AI Agents is only needed for agent mode; trace review additionally uses the AI module's ai_observability submodule with the OpenTelemetry module. The README walks through every surface, and docs/integration.md in the repository is a step-by-step guide from provider setup to cost estimation. Development happens in the issue queue under the 1.0 roadmap.

The design follows the eval methodology of Hamel Husain and Shreya Shankar: error analysis before metrics, and validate the validators before you believe them.