# Text & chat language models


📂 model-architectures

## MODEL ARCHITECTURES

# Text & chat language models

Generate text one token at a time, usually with a causal decoder trained for next-token prediction and then adapted to follow instructions.

## MENTAL MODEL

A probabilistic continuation engine wrapped in a conversation protocol, not a database, search engine, or deterministic rules engine.

## DATA FLOW

- Messages, documents, or code

- Tokenizer + role/tool markers

- Causal language-model backbone

- Next-token distribution

- Decode until a stop condition

## How it trains

Pretraining commonly minimizes next-token cross-entropy over large token sequences. Instruction tuning, preference optimization, safety training, and tool-use examples then shape interaction behavior; they do not change the basic need for evidence and evaluation.

## How inference runs

The model repeatedly predicts and samples or selects one next token. A KV cache reuses prior attention state; long answers remain serial, and decoding settings change variability rather than factuality.

## Strengths

- •
Open-ended writing, transformation, explanation, code, and conversation

- •
Learns tasks from instructions and examples in context

- •
Can emit structured tool calls or schemas when constrained and validated

## Trade-offs

- •
Can produce fluent unsupported claims

- •
Autoregressive output adds per-token latency

- •
Context, prompting, and sampling choices materially affect behavior

## Use it when

- The task needs flexible language generation or synthesis

- A rubric and representative evaluation set can define acceptable behavior

- Fresh facts can be grounded through retrieval or tools

## Avoid or challenge it when

- A deterministic parser, query, or rules engine solves the task reliably

- Exact current facts are required but no trusted source is connected

- Unreviewed output could directly trigger a high-impact action

## Illustrative published families

- • GPT-style causal language models

- • Instruction-tuned descendants such as the InstructGPT research system

## Commonly combines with

## PRIMARY REFERENCES

## Model Architectures

## Model Architectures