The Transformer and Beyond

The attention mechanism and the Transformer architecture and its variants: tokenizers, positional representations, efficiency improvements, context-window extension, and the designs that followed. Excludes architectures based on state spaces or mixtures of experts, which belong to 5.3, and neural networks as an intellectual school, which belong to 3.3.

• 3.3 Connectionism and Neural Networks
• 5.3 Mamba, State-Space Models, MoE and Emerging Architectures

• 5.1 Foundation Models and General-Purpose AI
• 5.3 Mamba, State-Space Models, MoE and Emerging Architectures
• 5.4 Pre-training and Scaling Laws
• 5.5 Fine-tuning, RLHF, DPO and ORPO
• 5.6 Context Protocols and Tool Use
• 5.7 Multi-Agent Orchestration
• 5.8 Open, Open-Weight and Closed Models
• 5.9 Natively Multimodal Models
• 5.10 Hallucination and Structural Reliability
• 5.11 Sovereign and Localized Models