Check multimodal inputs before comparing model outputs
Before attributing an output difference to the model, check what the model actually received. Input preparation can change the task, especially when multiple modalities are involved.
Preserve the original and the submitted input
Save the original asset, the transformation settings, and a checksum of the final submitted file. Keep the prompt and the ordering of attachments alongside the record. For a hosted service, note any documented server-side transformations you cannot control.
Inspect each modality
| Modality | Preflight questions |
|---|---|
| Images | Were crop, orientation, resolution, and color handling preserved? |
| Audio | Were sample rate, channel mixing, trimming, and normalization recorded? |
| Video | Which frames and time intervals were retained? Was audio included? |
| Text | Did truncation, formatting, or encoding change the task? |
Treat unsupported combinations as unsupported until documented or tested. Do not infer video reasoning from isolated image support, or temporal consistency from a single output.
Use controls
Include an input with an unambiguous expected observation and an input that should expose a known boundary. Inspect submitted files before running the full set. When a preprocessing change alters results, report it as a separate condition.
For emerging world-model workflows, also define the state, action, time horizon, and observation used by the task. The protocol must match the actual interface; the term world model alone does not specify one.
Continue with the evaluation run record to preserve the full procedure. This is a proposed protocol, not a completed experiment.