Text & chat language models

📂 model-architectures

MODEL ARCHITECTURES#

Text & chat language models

Generate text one token at a time, usually with a causal decoder trained for next-token prediction and then adapted to follow instructions.

MENTAL MODEL#

A probabilistic continuation engine wrapped in a conversation protocol, not a database, search engine, or deterministic rules engine.

DATA FLOW#

• Messages, documents, or code

• Tokenizer + role/tool markers

• Causal language-model backbone

• Next-token distribution

• Decode until a stop condition

How it trains#

Pretraining commonly minimizes next-token cross-entropy over large token sequences. Instruction tuning, preference optimization, safety training, and tool-use examples then shape interaction behavior; they do not change the basic need for evidence and evaluation.

How inference runs#

The model repeatedly predicts and samples or selects one next token. A KV cache reuses prior attention state; long answers remain serial, and decoding settings change variability rather than factuality.

Strengths#

• • Open-ended writing, transformation, explanation, code, and conversation

• • Learns tasks from instructions and examples in context

• • Can emit structured tool calls or schemas when constrained and validated

Trade-offs#

• • Can produce fluent unsupported claims

• • Autoregressive output adds per-token latency

• • Context, prompting, and sampling choices materially affect behavior

Use it when#

• The task needs flexible language generation or synthesis

• A rubric and representative evaluation set can define acceptable behavior

• Fresh facts can be grounded through retrieval or tools

Avoid or challenge it when#

• A deterministic parser, query, or rules engine solves the task reliably

• Exact current facts are required but no trusted source is connected

• Unreviewed output could directly trigger a high-impact action

Illustrative published families#

• • GPT-style causal language models

• • Instruction-tuned descendants such as the InstructGPT research system

Commonly combines with#

PRIMARY REFERENCES#

Model Architectures#

Model Architectures#