---
title: A practical protocol for comparing text-to-speech outputs
slug: tts-evaluation-protocol
date: '2026-10-27'
description: A proposed evaluation procedure for pronunciation, intelligibility, voice
  behavior, and timing.
author_type: Organization
author_url: /editorial-method/
category: library
---
Use a fixed script set and a documented listening procedure before comparing text-to-speech systems. Keep quality judgments separate from runtime measurements.

## 1. Prepare the script set

Include ordinary sentences, numbers, abbreviations, named entities, punctuation, and a longer paragraph. Add the languages and speaking styles required by your application. Record script identifiers and do not modify text between candidates without logging the change.

## 2. Record the generation conditions

Save the model identifier, revision or endpoint, voice setting, reference audio if applicable, sample rate, decoding settings, and output format. Use reference recordings only with permission for the intended purpose. Retain the original outputs before loudness adjustment or other post-processing.

## 3. Listen against separate criteria

Judge intelligibility, pronunciation, prosody, artifacts, and speaker consistency separately. If speaker similarity matters, define the authorized reference and a separate comparison process. Randomize clip order and hide model names from listeners where practical.

## 4. Measure timing with clear boundaries

For streaming, define when the request starts and when the first usable audio arrives. Also record total generation time and audio duration. Note device or service conditions, network conditions where relevant, warm-up handling, and concurrency. Report failures and retries alongside successful runs.

## 5. Report bounded conclusions

Publish the scripts you can share, the procedure, sample count, settings, and known limitations. Keep listening judgments distinct from objective timing. Do not turn a small test into an overall model ranking.

This page proposes a protocol. No generated samples or benchmark results are claimed. Use [the general run record](/reproducible-model-evaluation/) to organize the evidence.