After months of development and contributions from the MindSpore open-source community, the MindSpore 2.10 framework is now available. MindSpore HyperParallel, a hypernode affinity parallelism mode, supports Muon + general computing optimization, improving the performance by 6 times. In addition, the collaborative upgrade of activation value recomputing and swap breaks the graphics memory bottleneck and fully unleashes the Ascend computing power. MindSpore AKG, an Ascend affinity graph mode, integrates the MFusion graph kernel fusion component into Inductor, supporting Ascend affinity and providing higher compilation acceleration benefits. In terms of basic framework evolution, the asynchronous initialization capability of the mccl collective communication is added to improve the startup efficiency of large-scale training. The capabilities of MindSpore Lite are continuously enhanced. The list of cloud-side inference models is expanded, and the multimodal inference acceleration plug-in is added to accelerate end-to-end inference. Now, let's delve into the key features of MindSpore 2.10.

-- MindSpore HyperParallel: Hypernode Affinity Parallelism --

1 Supporting Muon + General Computing Optimization and Improving the Performance by 6 Times

As the scale of LLM training continues to expand, fully sharded data parallel (FSDP) has become an important distributed parallel technology for relieving memory pressure and supporting the training of models with hundreds of billions to trillions of parameters. Concurrently, the Muon optimizer has gradually become a mainstream choice thanks to its efficient convergence. However, the Newton-Schulz computation of the Muon optimizer requires orthogonal updates to the complete parameter matrix, while FSDP typically shards parameters, gradients, and optimizer states and stores them on different devices. This results in a natural conflict between the two in terms of computation granularity and data layout. In large-scale clusters, there are problems such as high communication overheads of full matrix, severe repeated computation in the data parallel domain, and high scheduling overheads of small operators on the host. Based on the MindSpore HyperParallel FSDP2 framework, the FSDP-Muon fusion optimization solution is designed and implemented for the Ascend hypernode training platform. This solution provides multiple performance optimization methods for redundant computation and communication of the FSDP+Muon optimizer in large-scale clusters, including AllGather+DP redundancy removal, high-dimensional parameter communication and computation redundancy removal, Newton-Schulz matrix fusion, and hybrid sharded data parallel (HSDP) grouping redundancy removal. After multiple optimizations, the computing time of the MoE Muon optimizer with hundreds of billions of parameters in the Ascend hypernode 512-die cluster test is reduced from 2700 ms to 450 ms, improving the performance by about 6 times.

1.1 DP Redundancy Removal/HSDP Redundancy Removal

Core conflict: The row-based parameter sharding mechanism of FSDP fundamentally conflicts with the complete matrix required by the Newton-Schulz orthogonalization of the Muon optimizer.

Exploration of the Muon optimizer characteristics: As shown in the preceding figure, the Newton-Schulz process obtains all complete matrices through AllGather and performs Newton-Schulz computation. This results in repeated computation across multiple devices. Optimization idea: Reduce redundant computation, and perform only Newton-Schulz computation of some parameters on each device. Optimization 1: Perform HSDP grouping redundancy removal. Each HSDP group contains a complete matrix copy. The computation parameters are allocated within the group, and the computation workload linearly decreases with the number of copies, as shown in (1) in the figure below. Optimization 2: Perform DP grouping redundancy removal. In (2) of the figure below, each HSDP group contains multiple DP groups, and the matrices in the DP groups are the same. Further redundancy removal can be performed, as shown in (2) of the figure below.

1.2 Ultimate Graphics Memory Reduction

As shown in the preceding figure, the control parameters are updated in batches, so that the intermediate activation memory usage is controllable and the peak memory usage is reduced by 40%. In addition, the general-purpose computing of HSDP redundancy removal is implemented.

1.3 Performance Optimization of Newton-Schulz

After the Muon optimizer solution for DP/HSDP redundancy removal is implemented, the profiling analysis shows that the computing overhead of Newton-Schulz becomes the main bottleneck of the system. The Newton-Schulz computing is further optimized. (1) Redundancy removal optimization for high-dimensional parameter communication and computation

As shown in the figure above, global communication aggregation across devices is not required. Instead, Muon computation can be independently and concurrently executed based on the batch dimension of local shards. This achieves communication-free general-computation redundancy removal, and the absolute time consumed by the expert matrix is reduced from 4.9 ms to 1.1 ms, a decrease of about 77.5%. (2) Matrix fusion optimization

As shown in the preceding figure, a unified batch dimension is constructed on the host to reorganize the original scattered small matrices into a continuous high-dimensional tensor B, M, N. This triggers the efficient batch matrix multiplication (BMM) operator at the underlying hardware layer, significantly reducing the execution latency. For example, if there are 74 parameters with the shape of 24, 10240, the total time consumed by executing Newton-Schulz iterations in batches is reduced to 2.4 ms, which is much lower than the theoretical total time consumed by executing each parameter one by one (74 x 1.0 = 74 ms) before the optimization. Reference: https://gitcode.com/mindspore/hyper-parallel/tree/master/hyper_parallel/core/optimizer

2 Collaborative Upgrade of Activation Value Recomputation and Swap, Breaking Memory Bottlenecks and Fully Unleashing Ascend Computing Power

As the scale of LLM, sequence length, and number of micro-batches continue to increase, the activation value that needs to be retained for backward propagation during training increases rapidly. The activation graphics memory becomes a key factor that limits the model scale and training throughput. Traditional activation value recomputation reduces graphics memory by trading computation for memory. However, it is fixed to trigger recomputation only in the backward phase, making it difficult to fully utilize idle windows in the pipeline. In complex training scenarios, a single memory optimization method cannot balance graphics memory and execution efficiency. Based on the PyNative mode of MindSpore 2.10, MindSpore HyperParallel 1.0 upgrades the activation memory optimization capability in a unified manner. It incorporates the functions of activation value retention, recomputation, and swap-out into the same declarative policy, and supports independent scheduling of recomputation tasks. Users can flexibly combine policies based on the operator computation volume, activation size, and scheduling idle windows to reduce the device graphics memory and hide the overheads of recomputation and host-device data transfer.

2.1 Swap Policy Added for Dynamic Graph Recomputation, Balancing Performance and Graphics Memory

In MindSpore HyperParallel 1.0, recomputation can be selectively activated in PyNative mode, and the swap is used as a policy parallel to saving and recomputation. Users can use the unified policy interface to retain the results of operators with high computation costs, perform recomputation on operators with low computation costs, and asynchronously swap out large activations to the host. This allows multiple activation memory policies to be used together in the same recomputation region without relying on the entire graph capture or compilation. This capability does not depend on the unified planning of the entire forward and backward graphs. It can directly adapt to the native execution mode of Python dynamic control flow and dynamic graphs. The policy granularity is more flexible and it is easier to integrate with existing model code. The following figure shows the resource trade-offs of the three policies.

Reference: https://gitcode.com/mindspore/hyper-parallel/blob/r1.0.0/docs/guide/activation_checkpoint.md

2.2 Function- and Module-Level Suppression of Recomputation, Simplifying Configuration and Reducing Host Overhead

The operator-level policy is suitable for fine-grained control of activation, saving, recomputation, and swapping out. However, when users want to exclude the entire function or module from recomputation, it is inconvenient to identify and configure internal operators one by one. In MindSpore HyperParallel 1.0, checkpoint_exclude_wrapper is provided. Users can directly wrap the functions or modules that do not require recomputation from the perspective of code organization, without the need to be aware of the operators contained in the functions or modules. This capability suppresses recomputation at the function or module boundary, and does not depend on the operator-level dispatch mechanism. During runtime, operators in the region do not need to be verified one by one, reducing the host check overhead caused by the accumulation of operators. The following figure shows the differences between the two configuration modes.

2.3 Asynchronous Swap of Recomputation Inputs Further Reducing Graphics Memory Usage

Even if intermediate activations are released through recomputation, the inputs of the recomputation region must be retained until the backward phase. In MindSpore HyperParallel 1.0, the inputs of recomputation can be asynchronously swapped out to the host and prefetched back to the device before being used in the backward phase. Data transfer is performed using an independent copy stream, which can overlap with subsequent computation, further reducing the time that large inputs stay on the device. For models with long sequences, large hidden size, or large inputs at the recomputation boundary, this capability can further reduce the active graphics memory while retaining the benefits of recomputation. The following figure shows the execution sequence of asynchronous swap-out and prefetching of recomputation inputs.

2.4 Recomputation Supports Independent Scheduling and Early Triggering, Releasing Overlapping Space for PP General-Purpose Computing

In addition to further reducing the graphics memory, MindSpore HyperParallel 1.0 also supports independent scheduling of the recomputation timing. In traditional recomputation, the backward propagation is triggered on demand, and the recomputation latency directly enters the key backward path. In MindSpore 2.10, the dynamic graph supports independent scheduling of the recomputation. The recomputation tasks can be collected in the forward phase, and the recomputation can be triggered in advance in a proper scheduling window. The recomputation result is then reused in the subsequent backward phase. This capability is especially applicable to the PP overlap_b_f scenario. The scheduler can complete the recomputation of the corresponding micro batch before the paired forward computation is started, avoiding resource contention between the recomputation and forward computation. This also creates more overlapping space for forward, backward, and parallel communication between experts. The following figure shows the differences between the traditional triggering mode and the independent scheduling mode.

With the dynamic graph capability of MindSpore 2.10, MindSpore HyperParallel 1.0 upgrades the activation graphics memory optimization from a single functionality to a complete, combinable, and schedulable capability through operator-level policy, function- and module-level recomputation suppression, asynchronous input and swap-out, and independent recomputation scheduling. This helps users flexibly balance the memory, computing power, and host bandwidth under different model structures and parallel policies. Reference: https://gitcode.com/mindspore/hyper-parallel

-- Ascend Affinity Graph Mode: MindSpore AKG --

3 MFusion, a Component-based Graph Kernel Fusion Component, Integrating into Inductor to Support Ascend Affinity and Achieve Higher Compilation Acceleration Benefits

As AI models evolve in different frontend frameworks such as MindSpore and PyTorch, if the graph fusion capability is deeply coupled with the internal representation of a single framework, repeated rule construction may occur, and it is difficult to reuse the verified hardware optimization experience. To address this issue, MFusion will accumulate reusable fusion methods and Ascend optimization experience in MindSpore, and consolidate them into an independent MLIR component. This forms a complete capability system that includes component-based access, manual fusion passes with Ascend affinity, and backend-aware automatic fusion, further expanding the cross-framework reuse boundary of MindSpore's fusion capabilities.

3.1 MLIR-based Component-based Architecture

As shown in the following figure, in the PyTorch scenario, MFusion is not a separate torch.compile backend, but is embedded in the post-grad graph optimization phase of the Inductor. The FX graph is sent to MFusion through Torch-MLIR. After fusion and sharding are completed, the graph is returned to the original pipeline in the form of custom operators. The original NPU backend is used for the non-fused regions, and the fused subgraphs are delivered to the target backend to generate Ascend kernels. In this way, the original capabilities of Dynamo, AOTAutograd, and Inductor can be retained, and the fusion rules and code generation backend can be continuously evolved as independent components.

3.2 Manual Fusion Pass with Ascend Affinity

MFusion first locks high-value structures through interpretable and controllable manual passes. By default, the pipeline serializes the fusion patterns on the Torch and MFuse sides, and supports custom DVM region labeling. For Transformer hotspots, the current implementation covers RMSNorm, Add+RMSNorm, LayerNorm, RoPE, GELU, SwiGLU, and Mean+Var. For MatMul/BMM, it provides casting and bias absorption, transpose absorption, 2D BMM degradation, K=1 rewritten as Mul, and MatMul-Reshape-Bias rearrangement. Each pass checks the dtype, shape, and value semantics synchronously. For example, RoPE will proactively retain the original image when there is a risk of value drift in BF16, avoiding the loss of precision due to fusion.

3.3 Automatic Fusion Based on Backend Awareness

After the high-value structures are preferentially locked using the manual pass, MFusion further performs automatic fusion based on backend awareness to cover long-tail operator combinations. The system first filters candidate nodes based on the operator support scope, dtype, shape, and backend limitations. Then, it forms valid regions through dependency graph search, union-find set merging, and loop detection. When there is no unified SSA insertion position for the entire subgraph, dynamic programming is used to remedy the sharding. In addition, the system completes restructuring and outline based on the computation modes such as Elementwise, Broadcast, Reduce, Reshape, and MatMul, and the Ascend heuristic rules. In this way, two optimization paths are formed: deterministic rules are preferred, and automatic fusion is used as a supplement.

As shown in the preceding figure, the actual running results show that in the test environment of NVIDIA A100 Tensor Core GPU and Ascend Atlas A2 training products, 34 TorchBench networks that pass the precision verification on both sides are computed using the formula "E2E acceleration ratio of Ascend Atlas A2 training products/E2E acceleration ratio of NVIDIA A100 Tensor Core GPU." The geometric mean of the obtained ratios is 1.20 times, and the ratios of 23 networks are greater than 1. Among the six typical networks, namely, NLP Transformer, Vision Transformer, speech, multimodal, classic CNN, and generative networks, the geometric mean of the acceleration ratio of BERT_pytorch, timm_vision_transformer, speech_transformer, torch_multimodal_clip, resnet18, and dcgan is 1.84 times, with the acceleration ratio of 4.46 times, 2.10 times, 1.92 times, 1.39 times, 1.33 times, and 1.15 times, respectively. The results show that in the test of multiple types of networks, MFusion can obtain high compilation acceleration gains in the Ascend Atlas A2 training series products. Reference link: https://gitcode.com/mindspore/akg/tree/master/mfusion

-- Continuous Evolution of the Basic Framework --

4 Adding the Asynchronous Initialization Capability of the mccl Collective Communication to Improve the Startup Efficiency of Large-Scale Training

As the scale of the training cluster for LLMs continues to expand, the initialization of CPU collective communication involves address synchronization and topology connection establishment across multiple nodes. In a cluster with thousands of devices, this process can take several minutes. In synchronous blocking mode, the device side can only wait idly, resulting in severe waste of computing power. In MindSpore 2.10, the asynchronous initialization capability of the mccl is added. As shown in the following figure, the mccl communication connection establishment is executed in the background thread. The main process can continue device initialization without waiting, implementing parallel processing of mccl communication connection establishment and device initialization. This improvement effectively reduces the time required for initializing and establishing connections in large-scale cluster scenarios, and improves the parallel utilization of the host and device sides. When the collective communication operation is performed, the framework automatically waits for the background thread to complete the initialization, and provides a built-in timeout protection mechanism to ensure security and reliability.

Reference: https://gitcode.com/mindspore/mindspore/tree/master/mindspore/ccsrc/plugin/cpu/res_manager/collective

-- Enhanced MindSpore Lite Capabilities --

5 Expanding the Cloud-Side Inference Model Support List to Cover 100 SOTA Models in 6 Categories

The model ecosystem is the key to implementing the inference framework in services. The cloud-side inference model support list of MindSpore Lite 2.10 is continuously expanded to cover 100 SOTA models in 6 categories. Among them, 47 models have provided complete conversion configurations and inference examples. A batch of SOTA models, including Kandinsky-5.0, Qwen3-VL Thinking/Instruct, Qwen3.5, Wan2.2, InternVL3_5 Flash, and Qwen3-TTS/Qwen3-ASR, are added. Reference link: https://gitcode.com/mindspore/mindspore-lite#%E4%BA%91%E4%BE%A7%E6%8E%A8%E7%90%86%E6%A8%A1%E5%9E%8B%E6%94%AF%E6%8C%81%E5%88%97%E8%A1%A8 It is worth mentioning that the cloud-side inference model support list of MindSpore Lite 2.10 works in collaboration with the multimodal inference acceleration plug-in. Currently, the plug-in supports multi-device parallel inference acceleration for the Wan2.1 T2V/I2V series. In the future, more open-source models will be supported based on the model registration mechanism. When using models in the cloud-side inference model support list, users can use this plug-in to obtain additional multi-device parallel and sparse attention acceleration capabilities. With the continuously expanded model support list and multimodal inference acceleration plug-in, MindSpore Lite 2.10 provides an E2E solution for Ascend cloud-side inference, featuring wide coverage and strong acceleration. This solution helps efficiently complete the entire process from model selection, conversion, deployment, to acceleration under a unified framework.

6 Enhanced Multimodal Inference Acceleration, Achieving E2E Inference Acceleration

As LLM inference is widely deployed in production environments, service providers often face challenges in obtaining optimal inference performance on the Ascend NPU. Key acceleration capabilities, such as fused operators, sparse attention, and multi-device sequence parallelism, are scattered in different implementations, lacking the unified and easy-to-use entry. To address this pain point, MindSpore Lite 2.10 is designed for the Ascend hardware. It deeply calls the Ascend CANN aclnn interface through the C++ custom operator, and combines the optimized attention and RoPE implementation at the Python layer with hccl multi-device communication to achieve E2E inference acceleration.

6.1 RainFusionAttention: Block-Level Sparse Attention Integrated with Native Ascend

In scenarios such as video generation and long-sequence language models, the standard attention computation volume increases quadratically with the sequence length. The built-in sparse attention operator directly encapsulates the native aclnnRainFusionAttention interface of Ascend CANN, and performs block-level sparsification on attention computation while retaining the precision. The complete process consists of five phases: token rearrangement, block-level pooling, top-k sparse mask generation, aclnnRainFusionAttention calling, and inverse rearrangement. With the collaboration of token rearrangement and sparse mask, the computation speed can be faster than that of dense attention even when sparsity is 0. To balance customization and out-of-the-box usability, two layers of APIs are provided. rain_fusion_attention is the bottom-layer operator, and sparse_attention is the upper-layer encapsulation, providing the complete pre-processing and post-processing logic. Reference: https://gitcode.com/mindspore/mindspore-lite/blob/master/mindspore-lite/lite_boost/docs/ops/RainFusionAttention.md

6.2 NPU Compatibility with FlashAttention and Optimized RoPE

NPU compatibility has been implemented for common attention mechanisms. FlashAttention automatically selects the backend in the following order based on priority: FA3 → FA2 → NPU native npu_prompt_flash_attention backend, and automatically matches the optimal path. RoPE is implemented using float32 real computation and cos/sin table cache. In the video generation model, the time required for RoPE is reduced by about 88%.

6.3 Multi-device Parallelism: USP + VAE Data Parallelism Time Slicing

Multi-device parallelism provides two types of parallelism policies that do not split model parameters. Ulysses Sequence Parallelism (USP) is a sequence parallelism oriented to the self-attention layer of the Transformer. Each device holds the complete model weights and communicates with each other only through all_to_all. The constraint is that the value of num_heads must be exactly divided by that of world_size. USP is suitable for long sequences and large batch scenarios. The VAE data parallelism time sharding is designed to address the cross-frame state dependency of the causal convolution in video VAE. It divides the video into overlapping frame chunks along the time dimension and distributes the chunks to each device for independent processing. Then, the chunks are collected and stitched together through all_gather. The overlapping frames cover the VAE time receptive field to ensure stitching consistency. The plug-in uses the model registration mechanism to support fast extension. You can implement the boost_xxx(model) function and register the class name in SUPPORTED_MODELS. Currently, the Wan2.1 T2V/I2V series are supported. You can use the ParallelManager(model) code to switch to the USP multi-device parallelism mode. Reference: https://www.mindspore.cn/lite/api/en/master/lite_boost/lite_boost.parallel.ParallelManager.html#lite_boost.parallel.ParallelManager MindSpore Lite provides a unified acceleration solution for cloud-side inference based on PyTorch interfaces, featuring native operators, automatic parallelism, and simplified access. With the LiteBoost multimodal inference acceleration plug-in, you can obtain E2E performance benefits of fused operators, sparse attention, and multi-device parallelism on the Ascend NPU without rewriting model scripts.