DeepSpeed Distributed LLM Training Checkpoints (global_step*)

Bottom Line: It is safe to delete intermediate DeepSpeed checkpoint folders (global_step*) for completed training runs. DeepSpeed caches model weights, optimizer states, and FP16 gradients per step.

Why Do DeepSpeed Checkpoint Files Exist?

Microsoft DeepSpeed optimizes distributed Large Language Model (LLM) training across multiple GPUs. During fine-tuning, DeepSpeed periodically writes checkpoint states (global_step100, global_step200) containing ZeRO-3 optimizer states and model weight tensors.

• Primary Purpose: Resuming distributed LLM training runs after preemption or hardware failure.
• Storage Growth: Each training checkpoint for a 7B or 13B model can weigh 10 GB to 100 GB+.

What Happens If You Delete DeepSpeed Checkpoints?

• System Safety: ✅ Safe to Delete.
• Completed Runs: If training is complete and final model weights (adapter_model.bin or model.safetensors) are exported, intermediate step checkpoints are no longer needed.
• Reclaimed Storage: Frees 10 GB to 100 GB+ of GPU cluster SSD storage.

How to Delete DeepSpeed Checkpoints

# Delete intermediate step checkpoints, preserving final model output
rm -rf output_dir/global_step*

Frequently Asked Questions (FAQ)

How do I configure DeepSpeed to keep only the latest 2 checkpoints?

In your Hugging Face TrainingArguments or DeepSpeed config, set save_total_limit=2.