Time to First Token Explained

Teams keep buying a faster model and watching time to first token barely move.

Most of that number is not generation. It is connection, admission, queueing, and reading the prompt.

TTFT is the wait from sending a request until the first usable chunk arrives. It is not output speed, and it is not full application latency. We treat it as one timing boundary, next to output rate, completion time, and task success.

Measured without a stated boundary, the number means very little.

Most of the wait is not generation

Every request crosses several stages:

1. the client opens or reuses a connection,
2. the provider authenticates and admits the request,
3. a scheduler queues it for compute,
4. the model processes the input prompt, often called prefill,
5. the model produces its first output, and
6. the serving layer sends the first usable chunk back to the client.

Only stage five is the model writing anything. Everything above it is transport, admission, scheduling, and reading the prompt, which is why a team that chases TTFT by swapping models often finds the number barely moves.

TTFT normally includes all six from the client's point of view. Server-side metrics may exclude DNS, connection setup, or internet transit. Providers often report model time measured after queueing. Neither convention is wrong, but the labels are not interchangeable, and a vendor comparison that mixes them is measuring two different things.

Our speed dashboard uses the runtime feed's first-answer measurement. Do not read it as full voice latency, browser render time, or a guarantee for another region.

Four numbers get called speed

Most latency arguments are two people using one word for different quantities. Four terms cover almost all of it.

Table 1 Download CSV

Metric	What it times	Relationship	Dominates when
TTFT	Request sent to first usable chunk	—	The user is waiting in silence
TPOT	Average gap between output tokens	1 ÷ output rate	The answer is long and visible
Output rate	Tokens emitted per second after the first	1 ÷ TPOT	Reports, drafts, transcripts
End-to-end	Request sent to last token, or to first audio	TTFT + generation	The app cannot act on a partial answer

TPOT and output rate are one measurement wearing different units. Fifty tokens per second is about twenty milliseconds per token. Vendors quote whichever framing flatters the result, so convert before comparing.

End-to-end is the row product teams actually feel, and the only row no provider can publish on your behalf, because it includes your prompt, your network, and your retrieval step.

Suppose Model A begins in 0.4 seconds and produces 40 tokens per second, while Model B begins in 2.0 seconds and produces 300 tokens per second. Both figures are invented for the arithmetic. The shape they describe is not.

For a ten-token classifier response, Model A finishes first. For a 2,000-token report, Model B overtakes it after the opening delay. Crossover depends on output length:

estimated completion time = TTFT + output tokens ÷ output tokens per second

That formula ignores stop time, network jitter, variable decoding rate, and reasoning tokens, but it exposes the trade. One "fastest model" label hides two clocks.

Table 2 Download CSV

Workload	Metric that usually dominates	Why
Voice turn	End-of-speech to first audio	Silence feels broken before a long answer exists
Chat answer	TTFT and readable streaming	Early text confirms that work started
Router or classifier	Full completion time	The application needs the short result before acting
Long report	Output rate and completion time	Most time is spent after the first token
Background batch	Throughput and cost	No user is watching the initial wait

Fast models are rarely the strong ones

Current first-answer shortlist, rebuilt from the runtime and model catalogs on every deploy:

Table 3 Download CSV

Model	Latency (first answer)	Output speed	Type	Overall score
Gemini 3 Flash	0.75s	199 t/s	Non-Reasoning	60
GPT-5 (high)	1.33s	66 t/s	Reasoning	59
Claude Opus 4.5	1.42s	52 t/s	Non-Reasoning	63
GLM-4.5	1.45s	51 t/s	Non-Reasoning	58
Claude Sonnet 4.6	1.60s	47 t/s	Non-Reasoning	64
Claude Opus 4.6	2.37s	44 t/s	Non-Reasoning	68
Kimi K2.6	2.88s	40 t/s	Reasoning	60
GPT-5.4 nano	3.64s	191 t/s	Reasoning	66

Read that table twice. Once top to bottom, which is first-answer order. Then re-sort it mentally by the overall score column.

Those two orderings barely agree. Quick models cluster low in the quality column, strong models tend to sit in the slower half, and the rows worth your attention are the few that appear early in both. Exceptions like that are worth more than any single latency record, because they are the only models that let a team stop trading one property for the other.

So we do not publish a "fastest model" verdict. Speed ranks one column. Intersection is what matters, and it moves whenever a provider ships a new serving stack.

Treat the table as a screen, not a service-level promise. Providers reorder it without changing a model name. Region, account tier, concurrency, prompt length, reasoning setting, and cache state all shift the request.

Use the full speed dashboard to sort by first answer, output rate, price, and score. Then repeat the measurement on the route you actually deploy.

An estimated latency is worse than a gap

Our renderer drops any model whose runtime feed carries no first-answer measurement, and any model below a quality floor.

Both refusals cost us coverage. Readers looking for a model we have not timed find nothing rather than an approximation, and we field the occasional note asking why a well-known name is missing. We keep the behavior anyway, for two different reasons.

We do not estimate latency, because a plausible interpolated number is worse than an absent one. Someone will build a budget on it, and no caveat travels as far as a figure in a table.

We set the quality floor because first-answer speed has a degenerate optimum. Without it, the top of the table fills with small, weak, or unverified models that answer quickly and badly, and a screening table recommending those has failed at the only job it has.

Neither rule makes our data better. They make the omissions honest, which is a smaller and more defensible claim.

Long prompts hide the real cost

Before decoding output, a transformer processes the input tokens and builds the state needed for generation. Longer prompts increase that prefill work. Large tool schemas, retrieved documents, conversation history, images, and system instructions all belong to the input even when the user typed one sentence.

Teams usually notice this late. Your agent felt responsive in week one and drags by week six, and nothing about the model changed. What changed was the prompt: three more tools, a longer system instruction, and a retrieval step that now returns eight documents where it used to return three.

Prompt caching reduces repeated work, but cache rules differ between providers. Requests miss because one early token changed, the cached prefix expired, the region changed, or the provider routed to another model version. Record cache status rather than averaging hits and misses into one unexplained number.

Reasoning settings add another variable. Some systems perform hidden or visible reasoning before a usable answer appears. First network events often carry metadata rather than content the application can show. Define "first token" as the first usable answer chunk when that is what the user experiences.

One request is a demo

One request proves that the endpoint worked once.

For a useful test, build a fixed set of prompts at several input and output lengths. Warm the client connection separately from cold starts. Send requests at the concurrency the application expects, from the target region, through the same provider and API path production will use.

Report at least p50 for the typical request, p95 or p99 for the slow tail, output tokens per second after the first chunk, full completion time, input and output token counts, errors and rate-limit responses, and task success for the answer that came back.

Keep failed requests in the operational report. Deleting timeouts makes a surviving latency distribution look excellent while the product is unusable.

Most teams need a small measurement, and writing it beats trusting anyone's published figure, ours included:

import time, statistics
from openai import OpenAI

client = OpenAI()

def ttft_once(prompt, model="gpt-4.1"):
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta
        # Wait for the first chunk carrying visible content.
        # Early chunks often hold role or tool metadata only.
        if getattr(delta, "content", None):
            return time.perf_counter() - start
    raise RuntimeError("stream ended with no content")

samples = sorted(ttft_once("Summarize the causes of the 1907 panic.") for _ in range(30))
print(f"n={len(samples)}")
print(f"p50={statistics.median(samples):.3f}s")
print(f"p95={samples[int(len(samples) * 0.95) - 1]:.3f}s")
print(f"max={samples[-1]:.3f}s")

Three details there do the real work.

It waits for content rather than the first network event, which is the most common way a benchmark reports a number lower than anything a user experiences. It runs thirty times, because tails generate support tickets and medians do not. And it prints p95 beside p50, so a comfortable median cannot hide a fifth request taking four times as long.

Change one variable at a time from there. Vary prompt length first, since prefill is usually the largest lever a team controls. Then run it from the region serving production, at the concurrency production sees.

Voice needs time to first audio

Text TTFT is one segment of a voice turn. Cascaded systems add end-of-speech detection, transcription, model inference, speech synthesis, transport, buffering, and playback. Native audio models combine some stages but still carry network, reasoning, synthesis, and playback delay.

Measure end of user speech to first audible agent sample. That is time to first audio, not TTFT.

Track interruption recovery and false end-of-turn decisions beside it, because an aggressive endpoint detector can look fast by cutting the caller off. Speed bought that way shows up later as repeated questions and abandoned calls, and neither appears in a latency chart.

Our voice-agent model guide uses text first-answer rows to build a shortlist, then requires a call replay. That is the correct boundary between a model metric and a product decision.

Fix the stage that is actually slow

If queueing dominates, change capacity, tier, region, concurrency, or provider. If prefill dominates, trim repeated instructions, retrieve fewer better documents, reduce tool schemas, or hold a stable cacheable prefix. If decoding dominates, choose a faster model, request less output, or move long work off the interactive path.

Identify the stage before touching any of them. Model swaps are the most visible lever and frequently the wrong one, because they address stage five of six.

Streaming improves perceived speed only when partial output is safe to show. Tool calls cannot execute half an argument, and structured responses may be invalid until the closing token arrives. For those workloads, total time to a valid result matters more than the first byte.

We would most like to publish one number we cannot yet: first-answer latency measured from more than one region against the same model on the same day. Until that run exists, treat every latency table on the internet as a screen, including ours, and treat the thirty-line script above as the measurement.