Function Overview

MindSpore Transformers dynamic graph (PyNative) provides various functions for the entire training process of pre-training/fine-tuning, facilitating configuration-based development and optimization. This section summarizes the functions by general functions and training functions. The inference and deployment capabilities are currently provided by static graphs and are listed in "Static Graph Features."

General Functions

Function	Description
Task startup	One-click startup of single-device, single-node multi-device, and multi-node tasks based on msrun.
Configuration file description	YAML in dataclass style, which centrally manages all configurable items for training.
Logs	Log structure and storage description.

Training Functions

Function	Description
Dataset	Megatron dataset (BlendedMegatronDatasetDataLoader) with preprocessed .bin/.idx, supporting multi-source mixing.
Hyperparameters and optimizers for training	AdamW/Muon optimizer and learning rate strategy with warmup.
Distributed parallel training	DP/FSDP, TP, PP, CP, EP, and SP multi-dimensional hybrid parallelism.
Training memory optimization	Recomputing (full/select), fine-grained SWAP, and CPU offload.
Safetensors weight	Safetensors shard saving and loading, supporting asynchronous saving and redundancy elimination.
Resumable training	Step-level resumable training, reducing the loss caused by interruptions in large-scale training.
Training metric monitoring and profiling	grad/param norm, loss monitoring, and MaxLogits value health monitoring and performance analysis.
Other training features	Gradient accumulation, gradient clipping, and operator fusion.

Static Graph Features

For details about the capabilities that are not covered by dynamic graphs, such as inference and deployment, see Static Graph Features.