Why 'Next-Token Predictor' Is the Wrong Mental Model for LLMs

Calling an LLM a “next-token predictor” isn’t wrong—it’s incomplete. That framing describes the mechanism (autoregressive token generation) but ignores what post-training encodes: simulation of a helpful assistant and knowledge discovered through exploration.

The Training Loops

Pre-training is indeed next-token prediction:

for tokens in training_data:
    for position in range(1, len(tokens)):
        prior_tokens = tokens[:position]
        actual_next_token = tokens[position]
        model.make_more_likely(actual_next_token, after=prior_tokens)

But modern LLMs undergo RLVR (reinforcement learning with verifiable rewards), which looks different:

for task in training_tasks:
    for explored_tokens in model.explore(task):
        reward = evaluate_outcome(task, explored_tokens)
        for position in range(len(explored_tokens)):
            prior_tokens = task + explored_tokens[:position]
            explored_next_token = explored_tokens[position]
            model.make_more_likely(explored_next_token, after=prior_tokens, according_to=reward)

During RLVR, the model generates new sequences and learns from outcomes—never seen in training data. That’s fundamentally different from just imitating existing text.

Chess Analogy

Think of two chess systems. One trained only on grandmaster games predicts the most likely next move—a next-move predictor. Another exhaustively explores all possible games, knows win probability from every position, and picks the move that maximizes winning. Calling the second a “next-move predictor” would be strange—it’s trying to win, not to imitate.

Why It Matters

RLHF also shifts models away from imitation toward helpful-assistant simulation. RLVR goes further, enabling exploration beyond training data. So “next-token predictor” describes the shape but ignores what the loop encodes. A simulation of a helpful assistant and discovered knowledge both fit in the same autoregressive loop—but that’s not all they are.