> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fetchhive.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiment evaluators

> Current evaluator status and planned evaluator types for experiments

Evaluator execution is not enabled in the current experiments release.

Datasets can already store `expected_output`, but Fetch Hive does not yet run automatic scoring or pass/fail checks from it.

## Current behavior <a href="#current-behavior" id="current-behavior" />

When you upload a dataset, `expected_output` is stored with each row.

Use it during manual review and row comparison.

No exact-match evaluator runs automatically today.

## Planned evaluator types <a href="#planned-evaluator-types" id="planned-evaluator-types" />

Future evaluator support may include:

| Evaluator type    | Use case                                                                                 |
| ----------------- | ---------------------------------------------------------------------------------------- |
| Exact match       | Strictly compare output with expected output                                             |
| Contains          | Check whether output includes required text                                              |
| Regex             | Check output against a pattern                                                           |
| JSON field match  | Compare specific fields in structured output                                             |
| Schema validation | Confirm output follows a required JSON schema                                            |
| LLM judge         | Score semantic correctness, reasoning quality, instruction following, or task completion |
| Custom evaluator  | Run workspace-defined evaluation logic                                                   |

## How should I prepare datasets for evaluators? <a href="#prepare-datasets" id="prepare-datasets" />

Add `expected_output` when you have a known answer.

Use `metadata.*` columns to group rows by topic, priority, source, language, or case id.

Keep expected outputs concise when you expect exact or contains checks.

Use structured JSON in `expected_output` when future field-level checks will be useful.

Example:

```csv theme={null}
question,expected_output,metadata.case_id,metadata.topic
"Return the country code for Japan.","JP","locale-001","localization"
```

See also: [Datasets](./datasets) and [Review results](./results)