A reproducible model evaluation starts with a run record
A useful evaluation record explains what was run, under which conditions, against which inputs, and with which judging criteria. Save that record before interpreting a result.
Minimum run record
| Item | Record |
|---|---|
| Model | Exact identifier, revision, weight variant, or endpoint |
| Environment | Runtime, dependencies, device, and relevant resource limits |
| Inputs | Dataset revision or checksums and sample identifiers |
| Configuration | Prompts, preprocessing, decoding settings, and seed where supported |
| Outputs | Original generated files and machine-readable logs |
| Evaluation | Metric version, judging criteria, reviewer process, and exclusions |
Use relative identifiers in shareable records and keep private inputs out of public repositories. Record unavailable version information explicitly.
Keep the procedure stable
Define the evaluation set and acceptance criteria before running candidates. Run comparable conditions and retain failures, timeouts, and retries. If you change the procedure, create a new run record rather than overwriting the old one.
Report the limit of the result
State the tested population, number of samples, configuration, and unresolved failure modes. A small evaluation supports a bounded observation. It does not establish universal performance or causality.
This page proposes a run-record protocol; it contains no completed experimental results. See Hugging Face Evaluate for an evaluation-tooling reference, and our speech protocol for an example application of the method.