Large Language Models

A large language model generates text by repeatedly predicting the most probable next token from the tokens already present. A token may be a word, part of a word, punctuation mark or other text unit; repeated next-token prediction builds a longer response one token at a time.

Learning to predict tokens

During pretraining, the model processes large text collections and learns statistical relationships among tokens rather than storing an explicit set of linguistic rules.

• A tokenizer converts text into token identifiers drawn from a fixed vocabulary.
• A transformer uses self-attention to relate each token to relevant earlier tokens and combines this with positional information.
• The training objective compares the predicted next token with the actual token; gradient descent adjusts model parameters to reduce prediction error.
• A causal attention mask prevents the model from using later tokens when learning to predict an earlier next token.

The generation loop

Given a prompt, the trained model calculates a score for every possible next token and converts these scores into a probability distribution.

• A decoding rule selects one token, which is appended to the existing context and fed back into the model.
• Greedy decoding chooses the highest-probability token, while sampling can choose among several plausible tokens.
• Parameters such as temperature alter randomness, while top-k or top-p methods restrict the set from which sampling occurs.
• The loop ends upon reaching a stopping token, a length limit or another system-defined condition.

Why outputs are coherent but fallible

The model's parameters capture recurring patterns of grammar, meaning, style and association, allowing locally predicted tokens to form coherent passages.

• Predictions are conditioned only on tokens within the model's context window.
• The objective rewards plausible continuation, not factual truth; therefore, a fluent answer may contain a hallucination.
• Early token choices influence later predictions, so an initial error can propagate through the generated sequence.
• Tasks requiring exact arithmetic or guaranteed logical procedures may fail because next-token prediction does not itself enforce symbolic correctness.

How UPSC asks this