v0.3.0 (03/11/2024)

New Features:

• Support mock transformers, see Mock transformers
• Support lm-evaluation-harness for model evaluation
• User Experience Optimization

New Supported Models:

• These models are natively supported by libai

New Mock Models:

• These models are extended and implemented by libai through mocking transformers.

v0.2.0 (07/07/2022)

New Features:

• Support evaluation enabled and set eval_iter
• Support customized sampler in config.py
• Support rdma for pipeline-model-parallel
• Support multi fused kernel
  • fused_scale_mask_softmax_dropout
  • fused_scale_tril_softmax_mask_scale
  • fused_self_attention in branch libai_bench
• User Experience Optimization
• Optimization for training throughput, see benchmark for more details

New Supported Models:

• Support 3D parallel Roberta model
• Support 2D parallel (data parallel + tensor model parallel) SimCSE model
• Support Data parallel MAE model
• Support Data parallel MOCOV3 model

Some major features:

• Data Parallelism
• 1D Tensor Parallelism
• Pipeline Parallelism
• Unified Distributed Layers for single-GPU and multi-GPU
• "LazyConfig" system for more flexible syntax and no predefined structures
• Easy-to-use trainer and engine
• CV & NLP Data Preprocessing
• Mixed Precision Training
• Activation Checkpointing
• Gradient Accumulation
• Gradient Clipping
• Zero Redundancy Optimizer (ZeRO)

Model_Zoo

• Bert (3D Parallelism)
• GPT-2 (3D Parallelism)
• ViT (3D Parallelism)
• Swin-Transformer (data parallelism)
• Add finetune task in projects/
• Add text classification in projects