AI Eval helps you find out whether AI features on a Drupal site are getting better or worse. Describe a good answer, run a fixed set of questions, score each answer with judges and checks, then compare the result with a quality gate. Agent mode tests an AI Agents plugin from end to end. Chat mode tests a provider, model, and prompt without an agent framework.

Built by a human using an AI assistant: 🤖 ➕ 🧠

Most of this module's code was generated with an AI assistant under human direction, review, and testing, using Strikethroo & Kenkeep for improved Drupal code generation.

Start with a guided test

Create a test in three steps. First choose the AI, graders, and quality gate. Next add examples or choose an existing dataset. Finally review the test, then save it, or save it and go straight to the run button. Drafts remain available for seven days. When several tests share a dataset, you can update the shared dataset or create a copy for one test.

Read a result

Each result starts with “N of M examples met your expectations.” Questions that need attention appear first. Run history records completed, cancelled, errored, and interrupted runs. You can resume an interrupted run or start a new one.

Compare two runs of the same test. When both runs used the same judge, prompt, and settings, the page reports a direct comparison. When something differed, it names what changed, so read the difference with care. When a run did not record its settings, the page shows the two results side by side without a verdict.

Know whether to trust the judge

Validate each LLM judge against answers labeled by people. AI Eval measures how often the judge agrees with people on good answers and on bad answers. A trust badge on the Dashboard shows the current state. Results warn you when a quality gate used a judge that has not passed validation.

Find out why it fails

Import traces, the recorded conversations between users and your AI feature, review them, and label what happened. Add failure modes to group recurring problems. Promote a conversation into a dataset so the same failure becomes a question in later runs.

Graders and rubrics

Twelve graders ship with the module. Seven LLM judges measure relevance, completeness, accuracy, whether the answer gives the reader useful next steps, fact match against verified facts, whether claims are supported by source material, and a decision rubric. Five deterministic graders check the answer format, the rubric checks described below, which tools the agent called, recall@k, and reciprocal rank.

Rubrics let many questions share the same expectations. The editor provides form fields for the two text-list check kinds and JSON editing for every other kind. The rubric page shows the whole rubric and provides an Edit button.

Improve the prompt

The optimizer reads failed answers and asks a model to propose a better system prompt. Review a proposal before you apply it. The optimizer repeats the current prompt and each candidate because judge scores can vary between runs. It accepts a candidate only when the measured improvement clears the configured margin.

Import and export

Export completed runs as Every Eval Ever (EEE) 0.2.2 JSON files. Import EEE files into the results table to inspect results produced elsewhere and compare results from different test tools. EEE files contain aggregate metrics and run details, not question or answer text. You can also import OpenTelemetry AI traces from a JSON file or directory.

Drush commands

Requirements

Agent mode also requires AI Agents. Trace review can use AI Observability and OpenTelemetry. These features are optional.

Documentation

Read the published documentation for installation, configuration, usage, and extension guides. The repository README provides an overview and quick start.