# Using Python Interface to Perform Cloud-side Inference

[![View Source on AtomGit](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.10.0/resource/_static/logo_source_en.svg)](https://atomgit.com/mindspore/docs/blob/r2.10.0/docs/lite/cloud_docs/source_en/mindir/runtime_python.md)

## Overview

This tutorial provides a sample program for MindSpore Lite to perform cloud-side inference, demonstrating the [Python interface](https://mindspore.cn/lite/api/en/r2.10.0/mindspore_lite.html) to perform the basic process of cloud-side inference through file input, inference execution, inference result printing, dynamic weight update and subgraph splitting inference, and enables users to quickly understand the use of MindSpore Lite APIs related to cloud-side inference execution. The related files are put in the directory [mindspore-lite/examples/cloud_infer/quick_start_python](https://atomgit.com/mindspore/mindspore-lite/tree/r2.10/mindspore-lite/examples/cloud_infer/quick_start_python).

MindSpore Lite cloud-side inference is supported for running in Linux environment deployment only. Atlas 200/300/500 inference product, Atlas inference series, Atlas training series and CPU hardware backends are supported.

> When using MindSpore Lite together with other third-party frameworks, please ensure that the import actions of other third-party frameworks are placed before `mindspore_lite.Model.build_from_file` and `mindspore_lite.ModelGroup.cal_max_size_of_workspace`, otherwise it may lead to unexpected issues.
>
> The Python inference interface supports bfloat16 input and output data types, which depends on the `ml_dtypes` library. See the [bfloat16 Inference](#bfloat16-inference) section for details.

The following is an example of how to use the Python Cloud-side Inference Demo on a Linux X86 operating system and a CPU hardware platform, using Ubuntu as an example:

- One-click installation of inference-related model files, MindSpore Lite and its required dependencies. See the [One-click installation](#one-click-installation) section for details.

- Execute the Python Cloud-side Inference Demo. See the [Execute Demo](#executing-demo) section for details.

- For a description of the Python Cloud-side Inference Demo content, see the [Demo Content Description](#demo-content-description) section for details.

- For a description of Weight Update content, see the [Dynamic Weight Update](#dynamic-weight-update) section for details.

- For a description of Subgraph Splitting Inference content, see the [Subgraph Splitting Inference](#subgraph-splitting-inference) section for details.
- For a description of Pre-Inference content, see the [Pre-Inference](#pre-inference) section for details.
- For a description of bfloat16 Inference content, see the [bfloat16 Inference](#bfloat16-inference) section for details.

## One-click Installation

This session introduces the installation of MindSpore Lite for Python version 3.7 via pip on a Linux-x86_64 system with a CPU environment, taking the new Ubuntu as an example.

Go to the [mindspore-lite/examples/cloud_infer/quick_start_python](https://atomgit.com/mindspore/mindspore-lite/tree/r2.10/mindspore-lite/examples/cloud_infer/quick_start_python) directory, and execute the `lite-cpu-pip.sh` script for a one-click installation, taking installation of MindSpore Lite version 2.0.0 as an example. Script installation needs to download the model required for inference and input data files, the dependencies required for MindSpore_Lite installation, and download and install MindSpore Lite.

Note: This command sets the installed version of MindSpore Lite. Since the cloud-side inference Python interface is supported from MindSpore Lite version 2.0.0, the version cannot be set lower than 2.0.0. See the version provided in [Download MindSpore Lite](https://www.mindspore.cn/lite/docs/en/r2.10.0/use/downloads.html) for details on the versions that can be set.

```bash
MINDSPORE_LITE_VERSION=2.0.0 bash ./lite-cpu-pip.sh
```

> If the MobileNetV2 model download fails, please manually download the relevant model file [mobilenetv2.mindir](https://download.mindspore.cn/model_zoo/official/lite/quick_start/mobilenetv2.mindir) and copy it to the `mindspore-lite/examples/cloud_infer/quick_start_python/model` directory.
>
> If the input.bin input data file download fails, please manually download the relevant input data file [input.bin](https://download.mindspore.cn/model_zoo/official/lite/quick_start/input.bin) and copy it to the `mindspore-lite/examples/cloud_infer/quick_start_python/model` directory.
>
> If MindSpore Lite inference framework by using the script download fails, please manually download [MindSpore Lite model cloud-side inference framework](https://www.mindspore.cn/lite/docs/en/r2.10.0/use/downloads.html) corresponding to the CPU hardware platform and operating system of Linux-x86_64 or Linux-aarch64. Users can use the `uname -m` command to query the operating system in the terminal, and copy it to the `mindspore-lite/examples/cloud_infer/quick_start_python` directory.
>
> If you need to use MindSpore Lite corresponding to Python 3.7 or above, please [compile](https://www.mindspore.cn/lite/cloud_docs/en/r2.10.0/use/build.html) locally. Note that the Python API module compilation depends on Python >= 3.7.0, NumPy >= 1.17.0, wheel >= 0.32.0. After successful compilation, copy the Whl installation package generated in the `output/` directory to the `mindspore-lite/examples/cloud_infer/quick_start_python` directory.
>
> If the MindSpore Lite installation package does not exist in the `mindspore-lite/examples/cloud_infer/quick_start_python` directory, the one-click installation script will uninstall the currently installed MindSpore Lite and then download and install MindSpore Lite from the Huawei image. Otherwise, if the MindSpore Lite installation package exists in the directory, it will be installed first.
>
> After manually downloading and placing the files in the specified location, you need to execute the lite-cpu-pip.sh script again to complete the one-click installation.

A successful execution will show the following results. The model files and input data files can be found in the `mindspore-lite/examples/cloud_infer/quick_start_python/model` directory.

```text
Successfully installed mindspore-lite-2.0.0
```

## Executing Demo

After one-click installation, go to the [mindspore-lite/examples/cloud_infer/quick_start_python](https://atomgit.com/mindspore/mindspore-lite/tree/r2.10/mindspore-lite/examples/cloud_infer/quick_start_python) directory and execute the following command to experience MindSpore Lite inference MobileNetV2 models.

```bash
python quick_start_cloud_infer_python.py
```

When the execution is completed, the following results will be obtained, printing the name of the output Tensor, the data size of the output Tensor, the number of elements of the output Tensor and the first 50 pieces of data.

```text
tensor's name is:shape1 data size is:4000 tensor elements num is:1000
output data is: 5.3937547e-05 0.00037763786 0.00034193686 0.00037316754 0.00022436169 9.953917e-05 0.00025308868 0.00032044895 0.00025788433 0.00018915901 0.00079509866 0.003382262 0.0016214572 0.0010760546 0.0023826156 0.0011769629 0.00088481285 0.000534926 0.0006929171 0.0010826243 0.0005747609 0.0014443205 0.0010454883 0.0016276307 0.00034437355 0.0001039985 0.00022641376 0.00035307938 0.00014567627 0.00051178376 0.00016933997 0.00075814105 9.704676e-05 0.00066705025 0.00087511574 0.00034623547 0.00026317223 0.000319407 0.0015627446 0.0004044049 0.0008798965 0.0005202293 0.00044808138 0.0006453716 0.00044969268 0.0003431648 0.0009871059 0.00020436312 7.405098e-05 8.805057e-05
```

## Demo Content Description

Running MindSpore Lite inference framework mainly consists of the following steps:

1. Model reading: Export MindIR model via MindSpore or get MindIR model by [model conversion tool](https://www.mindspore.cn/lite/cloud_docs/en/r2.10.0/mindir/converter_tool.html).
2. Create configuration context: Create a configuration context [Context](https://mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Context.html#mindspore_lite.Context) and save some basic configuration parameters used to guide model compilation and model execution.
3. Model creation and compilation: Before executing inference, you need to call [build_from_file](https://mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Model.html#mindspore_lite.Model.build_from_file) interface of [Model](https://mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Model.html#mindspore_lite.Model) for model loading and model compilation. The model loading phase parses the file cache into a runtime model. The model compilation phase can take more time, so it is recommended that the model be created once, compiled once and performed inference about multiple times.
4. Input data: The input data needs to be padded before the model execution.
5. Execute inference: Use [Predict](https://mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Model.html#mindspore_lite.Model.predict) of [Model](https://mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Model.html#mindspore_lite.Model) method for model inference.

For more advanced usage and examples of Python interfaces, please refer to the [Python API](https://www.mindspore.cn/lite/api/en/r2.10.0/mindspore_lite.html).

![img](../images/lite_runtime.png)

### Creating Configuration Context

Create the configuration context `Context`. Since this tutorial demonstrates a scenario where inference is performed on a CPU device, they need to set Context's target to cpu.

```python
import numpy as np
import mindspore_lite as mslite

# init context, and set target is cpu
context = mslite.Context()
context.target = ["cpu"]
context.cpu.thread_num = 1
context.cpu.thread_affinity_mode=2
```

If the user needs to run inference on Ascend device, they need to set Context's target to ascend.

```python
import numpy as np
import mindspore_lite as mslite

# init context, and set target is ascend.
context = mslite.Context()
context.target = ["ascend"]
context.ascend.device_id = 0
```

If the backend is Ascend deployed on the Elastic Cloud Server, set the `provider` to `ge`.

```python
context.ascend.provider = "ge"
```

### Model Loading and Compilation

Model loading and compilation can be done by calling [build_from_file](https://www.mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Model.html#mindspore_lite.Model.build_from_file) interface of `Model` to load and compile the runtime model directly from the file cache.

```python
# build model from file
MODEL_PATH = "./model/mobilenetv2.mindir"
IN_DATA_PATH = "./model/input.bin"
model = mslite.Model()
model.build_from_file(MODEL_PATH, mslite.ModelType.MINDIR, context)
```

### Inputting the Data

The way that this tutorial sets the input data is importing from a file. For other ways to set the input data, please refer to [predict](https://www.mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Model.html#mindspore_lite.Model.predict) interface of `Model`.

```python
# set model input
inputs = model.get_inputs()
in_data = np.fromfile(IN_DATA_PATH, dtype=np.float32)
inputs[0].set_data_from_numpy(in_data)
```

### Executing Inference

Call [predict](https://www.mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Model.html#mindspore_lite.Model.predict) interface of `Model` to perform inference, and the inference result is output to `output`.

```python
# execute inference
outputs = model.predict(inputs)
```

### Obtaining the Output

Print the output results after performing inference. Iterate through the `outputs` list and print the name, data size, number of elements, and the first 50 data for each output Tensor.

```python
# get output
for output in outputs:
  name = output.name.rstrip()
  data_size = output.data_size
  element_num = output.element_num
  print("tensor's name is:%s data size is:%s tensor elements num is:%s" % (name, data_size, element_num))
  data = output.get_data_to_numpy()
  data = data.flatten()
  print("output data is:", end=" ")
  for i in range(50):
    print(data[i], end=" ")
  print("")
```

## Dynamic Weight Update

MindSpore Lite inference supports dynamic weight updates on the Ascend backend. The usage steps are as follows:

### Creating Config File

Write the weight names, weight shapes, and operator names corresponding to the operators that need to be updated into a text file. Build a model to load the configuration file, set the configuration file, and the content of the configuration file `config.ini` is as follows:

```text
[ascend_context]
variable_weights_file="update_weight_name_list.txt"
```

The content format of update_weight_name_list.txt is as follows:

```text
weight_name:weight_shape;node_name
weight_name1:weight_shape1;node_name1
weight_name2:weight_shape2;node_name2
......
```

For example:

```text
matmul.weight:512,512;/Matmul
matmul2.weight:512,512;/Matmul2
matmul3.weight:512,512;/Matmul3
......
```

### Model Loading and Compilation

```python
import numpy as np
import mindspore_lite as mslite

# init context, and set target is ascend.
context = mslite.Context()
context.target = ["ascend"]
context.ascend.device_id = 0

# build model from file
MODEL_PATH = "./SD1.5/unet.mindir"
model = mslite.Model()
model.build_from_file(MODEL_PATH, mslite.ModelType.MINDIR, context,  "config.ini")
```

### Building A New Weight Tensor

Convert the SaveTensor data structure exported from third-party framework training to Tensor format that MindSpore Lite can support.

### Update Weights

Call the `update_weights` interface provided by MindSpore Lite to update weights, as shown below:

```python
new_weight = mslite.Tensor(data)
new_weights = [new_weight]
model.update_weights([new_weights])
```

## Subgraph Splitting Inference

When performing offline model conversion, if the [SplitGraph] parameter under the split_node_name is configured in the configuration file, the subgraph splitting inference feature must be used to create and infer the model. The purpose of this feature is to split the original model into multiple parts according to the specified configuration during the conversion process. Users can obtain the output of the intermediate layers of the model or provide input to certain layers in the middle of the model in this way, enabling inference on only a part of the model.

### Creating MultiModelRunner

Create a MultiModelRunner object as shown below:

```python
import mindspore_lite as mslite
model_path = "path_to_model"
context = mslite.Context()
context.target = ["ascend"]
context.ascend.device_id = 0
runner = mslite.MultiModelRunner()
runner.build_from_file(model_path, mslite.ModelType.MINDIR, context)
```

### Obtaining ModelExecutor

ModelExecutor can be understood as a ​​subgraph exported during model conversion​​ based on user-specified inputs and outputs. When creating a MultiModelRunner, multiple ModelExecutor instances are simultaneously generated for inference. Obtain the ModelExecutor as follows:

```python
execs = runner.get_model_executor()
```

### Executing ModelExecutor Inference​

When performing inference using ModelExecutor, you must first identify its input and output names. The inputs of a ModelExecutor may originate from either ​​the inputs of the entire graph​​ or ​​outputs from other ModelExecutor instances​​. You can use the following methods to retrieve the inputs and outputs:
ModelExecutor.get_inputs() returns the input names of the current ModelExecutor.
ModelExecutor.get_outputs() returns the output names of the current ModelExecutor.It is important to note that the sliced subgraph inputs may be more specified in the conversion profile than the output, and the number of subgraphs may also be more than specified in the configuration file, because there are some additional inputs from other subgraphs to prevent duplicate nodes in the subgraph when performing subgraph splitting.
Refer to the following code for inference with ModelExecutor:

```python
import numpy as np
dtype_map = {
  mslite.DataType.FLOAT32:np.float32,
  mslite.DataType.INT32:np.int32,
  mslite.DataType.FLOAT16:np.float16,
  mslite.DataType.INT8:np.int8
}
for exec in execs:
  exec_inputs = exec.get_inputs()
  exec_outputs = exec.get_outputs()
  for i,input in enumerate(exec_inputs):
    print("input name:", input.name, " input.shape:", input.shape, " input.dtype:", input.dtype)
    data = np.random.randn(*input.shape).astype(dtype_map[input.dtype])
    input.set_data_from_numpy(data)
  exec.predict(exec_inputs)
```

## Pre-Inference

Pre-inference refers to automatically performing one inference with randomly generated input data immediately after the model is successfully created (build_from_file), which is used to verify whether the model functions properly.

This feature can be enabled through a configuration file by setting `enable_pre_inference=true` in the `[common]` section:

```ini
[common]
enable_pre_inference=true
```

After enabling, when calling the [build_from_file](https://www.mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Model.html#mindspore_lite.Model.build_from_file) interface to load and compile the model, the underlying framework will automatically generate random data to fill the inputs and call [predict](https://www.mindspore.cn/lite/api/en/r2.10.0/mindspore_lite/mindspore_lite.Model.html#mindspore_lite.Model.predict) once. If the inference fails, the `build_from_file` interface will return an error code, indicating that the model may be abnormal.

The usage of the pre-inference feature is consistent with the normal workflow. You only need to add the configuration item to the configuration file without modifying the code:

```python
import mindspore_lite as mslite

context = mslite.Context()
context.target = ["ascend"]
context.ascend.device_id = 0

model = mslite.Model()
# Pre-inference will be automatically performed inside build_from_file
model.build_from_file(MODEL_PATH, mslite.ModelType.MINDIR, context, "config.ini")
```

> The pre-inference feature only takes effect on the Linux platform and under non-Debug compilation mode. For models with dynamic dimensions (shape contains -1) or input size of 0, pre-inference will be automatically skipped.

## bfloat16 Inference

bfloat16 (Brain Floating Point 16-bit) is a 16-bit floating-point format that has the same exponent width (8 bits) as float32, offering smaller memory footprint and faster computation at the cost of minimal precision loss. MindSpore Lite Python inference interface supports bfloat16 input and output data.

### Installing Dependencies

The `ml_dtypes` library (version >= 0.5.4) is required for using bfloat16 data type:

```bash
pip install ml_dtypes
```

### Creating bfloat16 Input Data

Create bfloat16 numpy arrays and set them as model inputs:

```python
import ml_dtypes
import numpy as np
import mindspore_lite as mslite

model = mslite.Model()
model.build_from_file(MODEL_PATH, mslite.ModelType.MINDIR, context)

inputs = model.get_inputs()
# Create a bfloat16 numpy array as input
in_data = np.ones(inputs[0].shape, dtype=ml_dtypes.bfloat16)
inputs[0].set_data_from_numpy(in_data)
```

### Checking Tensor Data Type

The data type of an input or output Tensor can be checked via the `dtype` attribute. The bfloat16 type corresponds to `mslite.DataType.BFLOAT16`:

```python
print(inputs[0].dtype)  # Output: DataType.BFLOAT16
```

### Getting bfloat16 Output

After calling `predict` for inference, the output data obtained via `get_data_to_numpy()` will be automatically converted to a numpy array with `ml_dtypes.bfloat16` dtype:

```python
outputs = model.predict(inputs)
out_data = outputs[0].get_data_to_numpy()  # dtype is ml_dtypes.bfloat16
print(out_data.dtype)  # Output: bfloat16
```

### Complete Example

The following is a complete example of inference with bfloat16 data type on the Ascend backend:

```python
import ml_dtypes
import numpy as np
import mindspore_lite as mslite

# Create context
context = mslite.Context()
context.target = ["ascend"]
context.ascend.device_id = 0

# Load model
model = mslite.Model()
model.build_from_file(model_path="matmul_bf16.mindir", model_type=mslite.ModelType.MINDIR, context=context)

# Prepare bfloat16 input data
x = np.ones((2, 4), dtype=ml_dtypes.bfloat16)
y = np.ones((4, 3), dtype=ml_dtypes.bfloat16)

# Set inputs
inputs = model.get_inputs()
inputs[0].set_data_from_numpy(x)
inputs[1].set_data_from_numpy(y)

# Execute inference
outputs = model.predict(inputs)

# Get bfloat16 output
out = outputs[0].get_data_to_numpy()
print("Output dtype:", out.dtype)  # bfloat16
print("Output data:", out)
```