Pre-training and Scaling Laws

The large-scale pre-training stage: massive corpora, self-supervised objectives, scaling laws, the relation of size, data and compute to performance, and distributed training methods. Excludes post-training adaptation, which belongs to 5.5, data quality and documentation, which belong to 6.4, and the hardware and compute facilities, which belong to 7.2.

• 5.5 Fine-tuning, RLHF, DPO and ORPO
• 6.4 Quality, Documentation and Provenance
• 7.2 Cloud, Data Centers and AI Factories

• 5.1 Foundation Models and General-Purpose AI
• 5.2 The Transformer and Beyond
• 5.3 Mamba, State-Space Models, MoE and Emerging Architectures
• 5.5 Fine-tuning, RLHF, DPO and ORPO
• 5.6 Context Protocols and Tool Use
• 5.7 Multi-Agent Orchestration
• 5.8 Open, Open-Weight and Closed Models
• 5.9 Natively Multimodal Models
• 5.10 Hallucination and Structural Reliability
• 5.11 Sovereign and Localized Models