# Logs

MindSpore Transformers dynamic graph (PyNative) training is started through `scripts/msrun_launcher.sh`. All logs generated during the running are collected through the worker process logs of `msrun`. Understanding the log directory structure and key fields is the basis for locating training failures, loss exceptions, and single-device problems.

This section describes the purpose and directory structure of logs, configurable items of `msrun`, meanings of key fields in training logs, and the sequence of checking logs during troubleshooting. For details about how to start a task, see [Starting Tasks](./start_task.md).

## Log Overview

During PyNative training, all logs are carried by the `msrun` worker process logs.

- **`msrun` worker process logs**: The startup script `scripts/msrun_launcher.sh` passes the `LOG_DIR` parameter (default value: `output/msrun_log`) to the `--log_dir` of the underlying `msrun`. `msrun` redirects the **standard output (stdout) and standard error (stderr)** of each worker (device) process to a file named `worker_{i}.log` (another process `scheduler.log` is for scheduling). The setting `to_std=True` (default) of the framework `logger` mirrors all logs to standard outputs. Therefore, **all framework logs, including step-by-step training metrics (such as loss, learning rate, and gradient norm), running mode banner, parallel sharding, dataset/weight loading, Python traceback, C++/HCCL bottom-layer errors, and segfault exit information, are recorded in the `worker_{i}.log` file**. It is also mainly used to locate problems such as process breakdown and networking failures.

> **Troubleshooting Entry**
>
> - **If the process startup fails, breaks down during running, or reports an underlying error**, check the `worker_{i}.log` of the corresponding rank (including the complete traceback and HCCL/driver error information).
> - **When the training can run properly, check the loss, learning rate, convergence, and time consumed by a single step**, as well as the `worker_{i}.log` file, which contains all the metrics printed by the framework by step.
> - **If you suspect that a device is abnormal**, compare the `worker_{i}.log` and `scheduler.log` of the rank.

## Directory Structure Example

In the default configuration (`LOG_DIR` of `msrun_launcher.sh` is `output/msrun_log` by default), the log flushing structure is as follows:

```text
output/
└── msrun_log/                  # msrun's --log_dir, which collects worker process logs.
    ├── scheduler.log           # Logs of the scheduling process (networking and process startup/recycling)
    ├── worker_0.log            # stdout/stderr of the rank 0 process.
    ├── worker_1.log            # stdout/stderr of the rank 1 process
    └── ...                     # Each worker has a log file, **worker_{i}.log**.
```

> **File Name and Path Source**
>
> - `worker_{i}.log`/`scheduler.log`: It is generated by `msrun` in the `--log_dir` directory and named after the worker or scheduler. (This is an msrun behavior and is not defined by MindFormers. For details, see [Starting Tasks](./start_task.md).)
> - The log directory is controlled by the `LOG_DIR` parameter of `msrun_launcher.sh`. The default value is `output/msrun_log`.

## msrun Logs

`msrun` generates a `worker_{i}.log` file for each worker process in the directory specified by `--log_dir`, and writes the standard output and standard error of the process into the file. In addition, a `scheduler.log` file is generated to record the networking and process management. You are advised to use `scripts/msrun_launcher.sh` for startup. The `LOG_DIR` parameter specifies the log directory.

### Single-Node Startup

Single node (8 devices)

```bash
bash scripts/msrun_launcher.sh \
  "run_mindformer.py --config /path/to/pretrain_xxx.yaml --mode 1" \
  8 8 8118 output/msrun_log False 300
```

After the startup, files from `worker_0.log` to `worker_7.log` and `scheduler.log` are displayed under the `output/msrun_log/` directory.

### Multi-Node Startup

In the multi-node scenario, `msrun_launcher.sh` is executed on each node, and the worker logs of each node are stored in the `LOG_DIR` directory of the node (the logs are not automatically aggregated to the primary node). The following uses a two-node system with 16 devices (eight devices per node, primary node IP `192.168.1.1`) as an example:

Primary node (node 0):

```bash
bash scripts/msrun_launcher.sh \
  "run_mindformer.py --config /path/to/pretrain_xxx.yaml --mode 1" \
  16 8 192.168.1.1 8118 0 output/msrun_log False 300
```

Secondary node (node 1): The command is basically the same as that for the primary node. You only need to change the node ID parameter from `0` to `1` (the seventh parameter in `msrun_launcher.sh`), and keep other parameters the same. The `worker_{i}.log` of this node falls under the `output/msrun_log/` of node 1.

> **Multi-Node Check Prompt**
>
> To locate a multi-node issue, log in to the corresponding node and check the `LOG_DIR` of the node. For example, if you suspect that global rank 10 (local rank 2 on node 1) is abnormal, go to the `msrun_log` directory on node 1 to check the `worker_10.log`.

For details about `msrun_launcher.sh` parameters (`WORKER_NUM`, `LOCAL_WORKER`, `MASTER_ADDR`, `MASTER_PORT`, `NODE_RANK`, `LOG_DIR`, `JOIN`, and `CLUSTER_TIME_OUT`), see [Starting Tasks](./start_task.md).

## Key Log Fields

In the PyNative training loop, `LossCallback` (`_print_log` of `mindformers/pynative/callback/loss_callback.py`) to print a line of training metrics to the standard output every several steps. Then, `msrun` collects the metrics to `worker_{i}.log`. A typical record is as follows:

```text
[INFO] 2026-06-09 10:20:30 [.../loss_callback.py:231] _print_log: { step:[  100/ 1000], loss:   2.345678, per_step_time:    850ms, lr: 1.000000e-04, grad_norm:   1.234000, throughput:  12.34T }
```

> If pipeline parallelism (PP) is enabled for a training task, the loss is displayed only on the last stage. For example, if PP 2 is enabled for an 8-device training task, **rank_0** to **rank_3** belong to stage 0, and the corresponding log file `worker_0–3.log` does not contain loss information. You need to view the loss information in the log files `worker_4–7.log` of stage 1 (**rank_4** to **rank_7**).

The fields are described as follows.

| Field                    | Example              | Description                                                         |
|------------------------|------------------|-------------------------------------------------------------|
| `step:[ 100/ 1000]`    | Current steps/Total steps| Training step progress (global steps/`training.steps`).                         |
| `loss: 2.345678`       | Loss of the current steps        | Training loss of the current steps (single value), which is used to determine the convergence trend.                                    |
| `per_step_time: 850ms` | Duration per step            | Average training duration of the current steps. Pay attention to the performance and jitter.                                            |
| `lr: 1.000000e-04`     | Current learning rate           | It reflects the warmup and decay scheduling. If the learning rate scheduler cannot obtain the current learning rate in real time, this field is not printed.               |
| `grad_norm: 1.234000`  | Global gradient norm          | Global L2 norm of all parameter gradients. The training stability is determined based on the gradient clipping threshold. If the value is not obtained in this step, `grad_norm: NaN` is printed.|
| `throughput: 12.34T`   | Throughput              | Training throughput of the current steps (unit: `T`).                                            |

> **Supplementary Notes**
>
> Lines are concatenated per step by `LossCallback._print_log` of the dynamic graph. The field name is **`grad_norm`** (not `global_norm`), and the **`Epoch`**, `loss_scale`, and `overflow cond` fields do not exist. `loss` is a single value (not the two-section format of this step/moving average value). When the itemized loss function is enabled for the MoE or MTP model, fields such as `load_balancing_loss` and `mtp_{i}_loss` (for example, `mtp_1_loss`) are added to a line.
>
> - If more detailed metrics such as `local_norm`, `local_loss`, or tokens-per-expert of MoE are required, you can enable them using `monitor`. (In dynamic graph mode, these metrics are output to training logs and are not written into TensorBoard.) For details, see [Training Metric Monitoring and Profiling](./monitor.md).
> - Check whether the training is running in dynamic graph mode. During the startup phase, the banner `Running MindFormers in PYNATIVE_MODE.` (from `run_mindformer.py`, corresponding to `--mode 1`) is printed to confirm that the training has entered the dynamic graph mode. If this banner is not displayed, the training is not started in `--mode 1`.

## Troubleshooting Suggestions

You can check logs in the following sequence to efficiently locate the fault:

1. **Check `scheduler.log`**: `msrun_log/scheduler.log` records the entire cluster networking process (worker registration, topology construction, cluster initialization, and worker deregistration) and timeout or abnormal exit information (such as `Node X is timed out, please check this node's log`) of each worker. It is the entry for troubleshooting networking failures and abnormal process exits.
2. **If the process crashes or an underlying error is reported, check the `worker_{i}.log` of the corresponding rank**: The log contains the complete Python traceback and underlying error and exit information of the HCCL and driver.
3. **If you suspect that a single device is abnormal, check the `worker_{i}.log` of the corresponding rank**: Compare the log of the rank with the logs of other normal ranks to locate the differences.
4. **If there is a communication or networking problem, check the `scheduler.log` and each `worker_{i}.log`**: The information about networking setup, process startup, and process recycling is recorded in `scheduler.log`. HCCL errors are usually scattered in the logs of each worker, and therefore need to be compared across ranks.
5. **If the issue lies in multiple nodes, log in to the corresponding nodes and check the `LOG_DIR` of the localhost**: Logs of each node are not aggregated. You need to check the `worker_{i}.log` of the rank corresponding to the `msrun_log` on the abnormal nodes.

## Related Documents

- Task startup and `msrun` parameters: [Starting Tasks](./start_task.md)
- Finer-grained metrics and profiling: [Training Metric Monitoring and Profiling](./monitor.md)
- Configuration file overview: [Configuration File Description](./configuration.md)
- Parallel dimension and rank division: [Distributed Parallel Training](./parallel_training.md)
- End-to-end training process: [Training](../guide/training.md)