expected_output, but Fetch Hive does not yet run automatic scoring or pass/fail checks from it.
Current behavior
When you upload a dataset,expected_output is stored with each row.
Use it during manual review and row comparison.
No exact-match evaluator runs automatically today.
Planned evaluator types
Future evaluator support may include:| Evaluator type | Use case |
|---|---|
| Exact match | Strictly compare output with expected output |
| Contains | Check whether output includes required text |
| Regex | Check output against a pattern |
| JSON field match | Compare specific fields in structured output |
| Schema validation | Confirm output follows a required JSON schema |
| LLM judge | Score semantic correctness, reasoning quality, instruction following, or task completion |
| Custom evaluator | Run workspace-defined evaluation logic |
How should I prepare datasets for evaluators?
Addexpected_output when you have a known answer.
Use metadata.* columns to group rows by topic, priority, source, language, or case id.
Keep expected outputs concise when you expect exact or contains checks.
Use structured JSON in expected_output when future field-level checks will be useful.
Example: