Environment Variables

The following table lists the environment variables supported by MindSpore Transformers.

Tuning Variables

Variable Name	Default Value	Function	Value Description	Application Scenario
HCCL_DETERMINISTIC	false	Enables or disables deterministic computing of reduction communication operators, including AllReduce, ReduceScatter, and Reduce.	true: Enables the HCCL deterministic computing. false: Disables the HCCL deterministic computing.	Enabling deterministic computing can eliminate the randomness caused by inconsistent computing sequences of multiple devices, but the performance will be lower than that when deterministic computing is disabled. You are advised to enable this function in scenarios where consistency is required.
LCCL_DETERMINISTIC	0	Sets whether to enable the LCCL deterministic operator AllReduce (order-preserving addition).	1: Enables the LCCL deterministic operator. 0: Disables the LCCL deterministic operator.	Enabling deterministic computing can eliminate the randomness caused by inconsistent computing sequences of multiple devices, but the performance will be lower than that when deterministic computing is disabled. You are advised to enable this function in scenarios where consistency is required. This variable takes effect when rankSize is less than or equal to 8.
CUSTOM_MATMUL_SHUFFLE	on	Enables or disables the shuffling operation for custom matrix multiplication.	on: Enables matrix shuffling. off: Disables matrix shuffling.	The shuffling operation optimizes specific matrix sizes and memory access patterns. If the matrix size does not match the size optimized by shuffling, disabling shuffling may yield better performance. Set it based on the actual situation.
ASCEND_LAUNCH_BLOCKING	0	Determines whether to enable the synchronization mode during operator execution in the training or online inference scenario.	1: Forcible synchronous execution 0: Non-forcible synchronous execution	During NPU model training, operators are executed asynchronously by default. As a result, when an error is reported during operator execution, the printed error stack information is not the actual call stack information. When this parameter is set to 1, the operator is forced to run in synchronous mode for proper printing of the call stack information, making it easier to debug and locate problems in the code. Setting this environment variable to 0 can improve the computing efficiency.
TE_PARALLEL_COMPILER	8	Sets the maximum number of parallel compilation build processes. When the value is greater than 1, parallel compilation is enabled.	The value is a positive integer. The maximum value cannot exceed the number of CPU cores multiplied by 80% divided by the number of Ascend AI Processors. The value ranges from 1 to 32. The default value is 8.	Parallel compilation is especially useful when a large network is used. If this environment variable is set to 1, single-thread compilation is performed, which simplifies debugging.
CPU_AFFINITY	0	Enables the CPU affinity function. If this function is enabled, each process or thread is bound to a CPU core to improve performance.	1: The CPU affinity function is enabled. 0: The CPU affinity function is disabled.	To optimize resource utilization and save energy, the CPU affinity function is disabled by default.
MS_MEMORY_STATISTIC	0	Provides memory statistics.	1: The memory statistics function is enabled. 0: The memory statistics function is disabled.	During memory analysis, you can collect statistics on the basic memory usage. For details, see the optimization guide.
MINDSPORE_DUMP_CONFIG		Specifies the path of the configuration file on which the cloud-side dump function or device-side dump function depends.	Set the value to a relative path or an absolute path.
GLOG_v	3	Determines the level of MindSpore logs.	0: DEBUG 1: INFO 2: WARNING 3: ERROR, indicating that if an error is reported during program execution, an error log is generated, but the program may not terminate. 4: CRITICAL, indicating that if an exception occurs during program execution, the program will be terminated.
ASCEND_GLOBAL_LOG_LEVEL	3	Determines the level of CANN logs.	1: INFO 2: WARNING 3: ERROR 4: NULL. No log is output.
ASCEND_SLOG_PRINT_TO_STDOUT	0	Enables or disables log printing. After this function is enabled, logs are not saved in the log file. Instead, the generated logs are printed on the screen.	1: Log printing is enabled. 0: Log printing is disabled.
ASCEND_GLOBAL_EVENT_ENABLE	0	Specifies whether to enable event logging for applications.	1: Event logging is enabled. 0: Event logging is disabled.
HCCL_EXEC_TIMEOUT	1836	Synchronization wait time during task execution between devices. Within this configured time, each device process waits for other devices to perform communication synchronization.	The value range is (0, 17340]. The default value is 1836, in seconds.
HCCL_CONNECT_TIMEOUT	120	Sets the timeout interval for socket connection establishment between devices in distributed training or inference scenarios.	The value of this environment variable must be an integer. The value range is [120, 7200]. The default value is 120s.
MS_NODE_ID	NA	Specifies the rank ID of the current process in the dynamic networking startup scenario.	The value can be a rank ID of the current process, which is unique in the cluster.
MS_ALLOC_CONF	NA	Sets the memory policy.	Configuration item, in the format of key:value. Use commas (,) to separate multiple configuration items, for example, export MS_ALLOC_CONF=enable_vmm:true,memory_tracker:true. enable_vmm: specifies whether to enable the virtual memory. The default value is true. vmm_align_size: specifies the virtual memory alignment size, in MB. The default value is 2. memory_tracker: specifies whether to enable the memory tracker. The default value is false. memory_tracker_path: specifies the path for saving the memory tracker. By default, the memory tracker is disabled and the save path is empty. simple_tracker: specifies whether to enable the simplified tracker mode. In this mode, the tracker_graph.ir file is not saved, and only the last user task is retained. This item is valid only when memory_tracker is enabled. The default value is false. acl_allocator: specifies whether to use the ACL memory allocator. The default value is true. somas_whole_block: specifies whether to use the SOMAS whole-block memory allocation. The default value is false.
MS_INTERNAL_DISABLE_CUSTOM_KERNEL_LIST	PagedAttention	List of custom operators to be enabled. This is an experimental configuration item and does not need to be set in most cases. It will be deleted.	Set the value to a character string. Operator names are separated by commas (,).
OM_MODULES_CACHE	~/.cache/openmind/modules	Cache path of openMind modules.	Set the value to a directory path in string format.

Other Variables

Variable Name	Default Value	Function	Value Description	Application Scenario
RUN_MODE	predict	Sets the running mode.	predict: inference finetune: fine-tuning train: training eval: evaluation
USE_ROPE_SELF_DEFINE	true	Specifies whether to use fusion operators.	true: The ROPE fusion operator is used. false: The ROPE fusion operator is not used.	By default, the ROPE fusion operator is enabled to improve the computing efficiency. In scenarios other than debugging, disable this function as required. Generally, no special setting is required.
MS_ENABLE_INTERNAL_BOOST	on	Specifies whether to enable the internal acceleration function of the MindSpore framework.	on: The internal acceleration function of MindSpore is enabled. off: The internal acceleration function of MindSpore is disabled.	This function is enabled by default to achieve high-performance inference. When debugging or comparing different acceleration policies, you need to disable this parameter to observe the impact on performance.
MF_LOG_SUFFIX	NA	Sets the custom suffix of all log folders.	The value is the suffix of the log folder. By default, no suffix is specified.	A consistent suffix can be added to isolate logs of each task and prevent the logs from being overwritten.
PLOG_REDIRECT_TO_OUTPUT	False	Determines whether to change the plog storage path.	True: The logs are stored in the ./output directory. False: The logs are stored in the default storage location.	This setting helps users query plogs.
MS_ENABLE_FA_FLATTEN	on	Determines whether to support the FlashAttention flatten optimization.	on: The FlashAttention flatten optimization is enabled. off: The FlashAttention flatten optimization is disabled.	A rollback mechanism is provided for models that have not been adapted to the FlashAttention flatten optimization.
EXPERIMENTAL_KERNEL_LAUNCH_GROUP	NA	Determines whether to support batch parallel delivery of operators. If parallel delivery is supported, you can enable it and configure the number of parallel threads.	thread_num: number of concurrent threads. You are advised not to increase the value. The default value is 2. kernel_group_num: total number of operator groups. Each thread has kernel_group_num/thread_num groups. The default value is 8.	This feature will continue to evolve and the subsequent behavior may change. Currently, this feature is supported only in the deepseek inference scenario and can optimize the performance. However, using this feature for other models may cause performance deterioration. Therefore, exercise caution when using this feature. The usage method is as follows: export EXPERIMENTAL_KERNEL_LAUNCH_GROUP="thread_num:2,kernel_group_num:8".
ENFORCE_EAGER	False	Specifies whether to disable the JIT mode.	False: The JIT mode is enabled. True: The JIT mode is disabled.	JIT compiles functions into a MindSpore graph that can be called. Setting ENFORCE_EAGER to False enables the JIT mode, which can improve performance. Currently, only the inference mode is supported.
MS_ENABLE_TFT	NA	Enables the training fault tolerance (TFT) function. Most functions depend on MindIO TFT.	The value is "{TTP:1,UCE:1,HCCE:1,ARF:1,TRE:1,TSP:1,RSC:1}". To use a function, set the corresponding field to 1.	For details, see the high availability feature.
MS_WORKER_NUM	NA	Specifies the number of processes whose role is MS_WORKER.	The value is an integer greater than 0.	Distributed scenario.
RANK_ID	NA	Specifies the logical ID of the NPU to be called.	The value ranges from 0 to 7. When multiple servers are running in parallel, DEVICE_ID of different servers may be the same. In this case, you can use RANK_ID to avoid this problem. (During multi-node parallel running, RANK_ID = SERVER_ID × DEVICE_NUM + DEVICE_ID. DEVICE_ID indicates the sequence number of the Ascend AI Processor on the current server.)
RANK_SIZE	NA	Specifies the number of NPUs to be called.	The value is an integer greater than 1.
LD_PRELOAD	NA	Specifies the shared library to be preloaded.	The value is the path of the shared library.
DEVICE_ID	0	Specifies the ID of the device that calls the NPU.	The value ranges from 0 to the number of NPUs on the server.
MS_SCHED_PORT	NA	Specifies the port number bound to the scheduler.	The value is a port number ranging from 1024 to 65535.
NPU_ASD_ENABLE	0	Specifies whether to enable the feature value detection function.	0: The feature value detection function is disabled. 1: If an exception is detected, only logs are printed, but the detection operator does not throw an exception. 2: If an exception is detected, logs are printed and the detection operator throws an exception. 3: Logs are printed in both normal and abnormal scenarios. (In normal scenarios, logs are printed only when the INFO and DEBUG levels are enabled for CANN.) If an exception is detected, the detection operator throws an exception.
MS_SDC_DETECT_ENABLE	0	Specifies whether to enable the CheckSum to detect silent faults.	0: CheckSum is disabled to detect silent faults. 1: CheckSum is enabled to detect silent faults.
ASCEND_HOME_PATH	NA	Sets an installation path of the Ascend software package.	The value is a specified path.
LOCAL_DEFAULT_PATH	./output	Sets the default log path.	The value is a specified path.
STDOUT_DEVICES	NA	Sets the list of device IDs for standard output.	Set it to a list of numbers. Use commas (,) to separate multiple IDs.
REGISTER_PATH		Sets the directory where the external code to be registered is located.	The value is a specified path.
LOG_MF_PATH	./output/log	Sets the log path of MindSpore Transformers.	The value is a specified path.
DEVICE_NUM_PER_NODE	8	Sets the number of NPUs on the server.	The value is an integer greater than 0.
SHARED_PATHS		Sets the path of the shared storage.	The value is a specified path.
ASCEND_PROCESS_LOG_PATH	NA	Sets the log path of the Ascend process.	The value is a specified path.