Semi-automatic Parallel

Semi-automatic parallel supports the automatic mixing of multiple parallel modes, including:

• Operator-level parallel: Operator-level parallel refers to slicing the input tensor and model parameters into multiple devices for computation on an operator basis to improve overall speed.

• Higher-order Operator-level Parallelism: Higher-order operator-level parallelism refers to operator-level parallelism that allows customized device layout with tensor layout for more complex sharding logic.

• Optimizer parallel: Optimizer parallel reduces redundant computations on multiple devices for the same weight updates, spreading the computation over multiple devices.

• Pipeline parallel: Pipeline parallel means that the model is sliced by layer, with each device processing only a certain part of the model.