mindspore.parallel

mindspore.parallel provides a large number of interfaces for automatic parallelization, including parallel base configuration, model loading and transformation, and functional parallel slicing.

The module import method is as follows:

from mindspore import parallel

Parallel Base Configuration

`mindspore.parallel.auto_parallel.AutoParallel`	Encapsulation of top-level Cells or functions to realize static graph parallelism for a single network.
`mindspore.parallel.distributed.DistributedDataParallel`	DistributedDataParallel wrapper.
`mindspore.parallel.nn.GradAccumulation`	Implementation of parallel gradient accumulation for static graphs.
`mindspore.parallel.nn.MicroBatchInterleaved`	Implement the static graph parallel multi-copy splitting function to enable concurrent computation and communication.
`mindspore.parallel.nn.Pipeline`	Specify the number of micro_batch for pipeline parallelism and the division rules for stage, and specify the segment info.
`mindspore.parallel.nn.PipelineGradReducer`	Functional training scenarios for gradient statute and accumulation of pipeline parallel.

Model Loading and Transformation

`mindspore.parallel.convert_checkpoints`	Convert distributed checkpoint from source sharding strategy to destination sharding strategy for a rank.
`mindspore.parallel.convert_checkpoint_by_rank`	Convert distributed checkpoint from source sharding strategy to destination sharding strategy by rank for a network.
`mindspore.parallel.load_distributed_checkpoint`	Load checkpoint into net for distributed predication.
`mindspore.parallel.load_segmented_checkpoints`	Load checkpoint info from a specified file.
`mindspore.parallel.rank_list_for_convert`	List of original distributed checkpoint rank index for obtaining the target checkpoint of a rank_id during the distributed checkpoint conversion.
`mindspore.parallel.unified_safetensors`	Merge multiple safetensor files into a unified safetensor file.

Functional Parallel Slicing

`mindspore.parallel.function.reshard`	Converting a tensor from one distributed arrangement to another distributed arrangement.
`mindspore.parallel.Layout`	Topological abstraction describing cluster devices for tensor slice placement on the cluster.
`mindspore.parallel.shard`	Specify the input and output slicing strategy for a Cell or function.

Others

`mindspore.parallel.build_searched_strategy`	Extract the sharding strategy for each parameter in the network from the strategy file for distributed inference scenarios.
`mindspore.parallel.merge_pipeline_strategys`	Aggregate the sharding strategy files of all pipeline parallel subgraphs to the destination file.
`mindspore.parallel.parameter_broadcast`	Broadcast parameter to other rank in data parallel dimension.
`mindspore.parallel.restore_group_info_list`	Extract rank list information from communication domain files.
`mindspore.parallel.sync_pipeline_shared_parameters`	Synchronization of shared weights between stages for pipeline parallel inference scenarios.
`mindspore.parallel.strategy.enable_save_strategy_online`	Enable save strategy metadata online.
`mindspore.parallel.strategy.get_strategy_metadata`	Get all params strategy info or specific rank strategy info in this cell.
`mindspore.parallel.strategy.get_current_strategy_metadata`	Get parameters dictionary of cur rank of the network.
`mindspore.parallel.strategy.clear_strategy_metadata`	Clear all saved strategy metadata on the C++ side.