The engineering discipline surrounding Large Language Models (LLMs) has matured from a scattered collection of experimental scripts into a rigorous, multi-layered software stack. As of late 2025, the ecosystem is defined by a distinct tripartite lifecycle: Pre-training, Post-training (Fine-tuning), and Inference (Deployment). This article provides an exhaustive, expert-level analysis of the frameworks that dominate each phase. We analyze the architectural decisions, hardware optimizations, and trade-offs inherent in tools ranging from NVIDIA’s monolithic Megatron-LM to the agile versatility of ms-swift.
Part I: Pre-training Frameworks
Pre-training is the thermodynamic peak of the LLM lifecycle. It is a phase characterized by massive compute requirements, where the primary engineering objective is maximizing Model FLOPs Utilization (MFU) across thousands of GPUs. The frameworks in this category manage the complex choreography of distributed computing required by massive MoE (Mixture-of-Experts) architectures like Qwen 3 and DeepSeek-V3.
1.1 The Tensor Parallelism Titans: NVIDIA Megatron-LM
Repository: https://github.com/NVIDIA/Megatron-LM
Megatron-LM stands as the foundational reference implementation for the largest models in existence. Developed by NVIDIA, it is less a user-friendly library and more a blueprint for extreme-scale computing. Its architectural philosophy centers on Tensor Parallelism (TP), a technique that splits individual matrix multiplications across GPUs to minimize inter-node communication latency.
The Architecture of Scale
In 2025, Megatron-LM has evolved beyond simple TP. The introduction of Megatron Core (MCore) represents a significant refactoring, modularizing the codebase to separate distributed primitives from model definitions. This shift was necessitated by the increasing complexity of architectures like the dense-MoE hybrids found in Qwen 3. A critical innovation is Context Parallelism (CP). As context windows expanded to 1M+ tokens (driven by Long-Context Qwen variants), standard Sequence Parallelism (SP) became insufficient. CP splits the attention computation itself, allowing the Key-Value (KV) cache and attention scores to be distributed across GPUs without catastrophic communication overhead.
Hardware Symbiosis: Transformer Engine
Megatron-LM’s dominance is reinforced by the Transformer Engine (TE). TE enables native FP8 (8-bit floating point) training on NVIDIA Hopper and Blackwell architectures. By dynamically casting tensors between FP8 and BF16, TE allows Megatron-LM to double the throughput of matrix multiplications.
1.2 The Memory Optimizers: Microsoft DeepSpeed
Repository: https://github.com/microsoft/DeepSpeed
While Megatron-LM focuses on compute parallelization, Microsoft’s DeepSpeed tackles the Memory Wall. Its core contribution, the Zero Redundancy Optimizer (ZeRO), partitions optimizer states, gradients, and parameters across all available GPUs.
The MoE Era: DeepSpeed-MoE
With Qwen 3 and DeepSeek-V3 utilizing heavy Mixture-of-Experts architectures, standard data parallelism is inefficient. The 2025 updates focus on DeepSpeed-MoE, which optimizes the “all-to-all” communication primitives required to route tokens to their respective experts. This significantly reduces the overhead of expert parallelism.
1.3 The Minimalist Challenger: Hugging Face Nanotron
Repository: https://github.com/huggingface/nanotron
As frameworks like Megatron-LM became rigid, Nanotron emerged for researchers who need to iterate on architecture. Nanotron’s philosophy is “Minimalistic 3D Parallelism.” It exposes distributed primitives (TP, PP, DP) via a clean, Pythonic API, avoiding the “config hell” of older frameworks.
1.4 The Native Evolution: PyTorch FSDP2 & Torchtitan
Repository: https://github.com/pytorch/torchtitan
The PyTorch team has closed the gap with Fully Sharded Data Parallel 2 (FSDP2). Unlike the original FSDP, FSDP2 manages sharding at the tensor level (via DTensor), allowing for granular control over memory layout. Torchtitan is the flagship implementation, demonstrating that native PyTorch constructs can scale to thousands of GPUs utilizing Async Checkpointing.
Comparative Technical Analysis: Pre-training
| Feature Set | Megatron-LM | DeepSpeed | Nanotron | PyTorch FSDP2 |
|---|---|---|---|---|
| Primary Parallelism | Tensor (TP) + Pipeline (PP) | Data (ZeRO) + Pipeline | 3D (TP+PP+DP) | Sharded Data (ZeRO-style) |
| State Management | Replicated (TP) / Split (PP) | Partitioned (ZeRO 1/2/3) | Hybrid | Per-Parameter Sharding |
| MoE Support | Native (MCore) | DeepSpeed-MoE (Optimized) | Basic | Native via DTensor |
| Developer Exp. | High Complexity / Rigid | Medium Complexity | High Hackability | Native / Pythonic |
| Best For | Qwen 3 / DeepSeek Pre-training | Heterogeneous Clusters | Research / Prototyping | Native PyTorch Users |
Part II: Post-Training Frameworks
Post-Training (SFT and Preference Alignment) turns a raw predictive engine into a product. The 2025 landscape is dominated by Parameter Efficient Fine-Tuning (PEFT), with a massive shift towards tools that support the specific architectures of Qwen and DeepSeek.
2.1 The Qwen Specialist: ms-swift (ModelScope)
Repository: https://github.com/modelscope/ms-swift
Swift has rapidly become the “Swiss Army Knife” of fine-tuning, particularly for engineers working within the Qwen and DeepSeek ecosystems. Developed by the ModelScope team, it offers the most robust support for the latest architectural quirks found in Qwen 3 (e.g., tie-embedding issues, dynamic resolution in VLMs).
The “Tuners” Abstraction
Swift introduces a high-level “Tuners” abstraction that goes beyond standard LoRA, supporting Res-Tuning (vision-language), NEFTune (noise embeddings), and LoRA+. Swift is unique in its “Push-to-Deploy” workflow, where a model fine-tuned in Swift can be seamlessly exported to a containerized inference engine.
2.2 The Configuration Engine: Axolotl
Repository: https://github.com/axolotl-ai-cloud/axolotl
Axolotl remains the standard-bearer for production pipelines. It acts as a unifying wrapper, exposing the complexity of training via a single YAML configuration file.
- Multipacking: Axolotl’s implementation of Sample Packing is critical for efficiency, concatenating sequences to ensure the GPU computes useful gradients for every token position.
- 2025 Updates: Axolotl fully supports FSDP and DeepSpeed Zero-3 backends for 70B+ parameter models on consumer hardware.
2.3 The Kernel Optimizer: Unsloth
Repository: https://github.com/unslothai/unsloth
Unsloth uses hand-written Triton kernels to replace standard PyTorch implementations of Transformer layers. It manually derives backpropagation steps to reduce VRAM usage by up to 60%. It allows for the fine-tuning of Qwen-3-72B (quantized) on single high-end consumer GPUs, making it the go-to for local fine-tuning.
2.4 The Unified Workflow: LLaMA-Factory
Repository: https://github.com/hiyouga/LLaMA-Factory
LLaMA-Factory is a universal framework supporting over 100 models. It is famous for its WebUI (LLaMA Board), which visualizes training metrics in real-time. It implements advanced adapters like GaLore (Gradient Low-Rank Projection) and DoRA, and unifies SFT, DPO, and PPO into a single graphical pipeline.
Comparative Technical Analysis: Fine-tuning
| Framework | Architecture | Optimization Focus | Best For | Interface |
|---|---|---|---|---|
| ms-swift | Native Wrapper | Qwen/DeepSeek Ecosystem | Multimodal / Asian Models | CLI / Python / UI |
| Axolotl | Wrapper (HF/Peft) | Throughput (Multipack) | Production Pipelines | YAML Config |
| Unsloth | Kernel Replacement | VRAM / Speed | Consumer Hardware | Python Library |
| LLaMA-Factory | Wrapper (Unified) | Algorithm Variety | Experimentation / UI | WebUI / CLI |
| HuggingFace TRL | Native Library | Alignment (DPO/ORPO) | Research / Custom RLHF | Python Library |
Part III: Inference & Deployment Frameworks
The Inference phase introduces a new set of constraints: Time-to-First-Token (TTFT) and Inter-Token Latency (ITL). The 2025 landscape battles the “Memory Wall” created by the KV cache in long-context interactions.
3.1 The Throughput Standard: vLLM
Repository: https://github.com/vllm-project/vllm
vLLM defined modern serving with PagedAttention, which solves KV cache fragmentation.
2025 Features: vLLM has added robust support for FP8 KV Caching, essential for running Qwen 3-72B on limited hardware. It also supports Speculative Decoding and Chunked Prefill, ensuring that processing long system prompts doesn’t stall the generation of other requests.
3.2 The Structured Specialist: SGLang
Repository: https://github.com/sgl-project/sglang
SGLang (Structured Generation Language) optimizes for complex, agentic workflows often seen with DeepSeek-V3. It uses a Radix Tree (RadixAttention) for the KV cache.
- Mechanism: When an agent thinks, plans, and acts, it reuses the history. SGLang caches these prefixes in a tree structure.
- Performance: For “Reason then Answer” loops, SGLang achieves up to 5x higher throughput than vLLM.
3.3 The Enterprise Compiler: NVIDIA TensorRT-LLM
Repository: https://github.com/NVIDIA/TensorRT-LLM
TensorRT-LLM is a compiler. It compiles the model into a binary engine optimized for a specific GPU (e.g., H100). It fuses layers aggressively and is the gold standard for FP8 Inference on Hopper GPUs. However, it requires static compilation, meaning adapters cannot be swapped dynamically.
3.4 The 4-Bit Speedster: LMDeploy
Repository: https://github.com/InternLM/lmdeploy
LMDeploy features the TurboMind engine. It distinguishes itself through extreme optimization of W4A16 (4-bit Weights, 16-bit Activations) inference. It is highly optimized for the Chinese model ecosystem (Qwen, DeepSeek, InternLM) and often outperforms vLLM in single-user latency scenarios.
3.5 The Edge Democratizer: llama.cpp
Repository: https://github.com/ggerganov/llama.cpp
While data centers run on vLLM, llama.cpp has single-handedly democratized high-performance inference on consumer hardware, particularly Apple Silicon. Its “killer app” is the GGUF file format, which enables efficient memory mapping and heterogeneous compute, allowing layers to be split seamlessly between CPU and GPU.
The “K-Quant” Advantage: llama.cpp utilizes sophisticated quantization methods (like Q4_K_M) that preserve higher precision for critical weight matrices while compressing others. This makes it possible to run massive models like DeepSeek-V3 or Qwen-3-Instruct locally on a MacBook Pro with acceptable speed, powering the entire ecosystem of local AI tools like Ollama and LM Studio.
Comparative Technical Analysis: Inference
| Engine | Core Mechanism | Caching Strategy | Best Use Case | Quantization Focus |
|---|---|---|---|---|
| vLLM | PagedAttention | Block Table | General Serving / High QPS | AWQ / GPTQ / FP8 |
| SGLang | RadixAttention | Radix Tree (LRU) | Agents / Reasoning Loops | AWQ / FP8 |
| TensorRT-LLM | Compiler / Fusion | Static Allocation | Enterprise / H100 Clusters | FP8 / INT8 |
| LMDeploy | TurboMind | Persistent Batch | Qwen 4-bit Serving | AWQ / W4A16 |
| llama.cpp | GGUF / Metal | Linear / Mmap | Edge / CPU / Apple Silicon | GGUF (K-Quants) |
Part IV: The Future of the Stack
The trend in late 2025 is the dissolution of boundaries. We are seeing Training Frameworks adding Inference (e.g., DeepSpeed-MII) and Inference Engines adding Training capabilities. The “Holy Grail” currently being pursued is the End-to-End FP8 Pipeline. The goal is to keep the model in FP8 from the first pre-training step of Qwen 3 to the final token generation in vLLM, unifying the stack and drastically reducing the cost of intelligence. For the engineer in 2026, the choice is strategic:
- Foundation: Megatron-LM (Scale) or DeepSpeed (Heterogeneity).
- Refinement: ms-swift (Qwen/Multimodal) or Axolotl (Production Config).
- Deployment: vLLM (General API) or SGLang (Agentic/Reasoning).
The LLM stack of late 2025 has moved beyond the “Cambrian explosion” of tools into a consolidation phase defined by architectural specialization. Success in this domain no longer requires writing custom distributed training loops, but rather mastering the orchestration of these mature frameworks.
The winning engineering strategy lies in selecting the right tool for the specific constraint bottleneck: Megatron-Core for overcoming the compute bottleneck in pre-training, ms-swift/Axolotl for navigating the algorithmic nuance of post-training, and vLLM/SGLang for solving the memory bandwidth bottleneck of inference. As models like Qwen 3 and DeepSeek-V3 push the boundaries of context length and logic, the friction between these layers will decrease, eventually yielding a unified, compiler-driven OS for Artificial Intelligence.