Building Custom Operators Offline

Overview

Our Conversion Tool is a highly flexible tool. In addition to the basic ability of model converter, we have designed a set of registration mechanism, which allows users to expand, including node-parse extension, model-parse extension and graph-optimization extension. The users can combined them as needed to achieve their own intention.

node-parse extension: The users can define the process to parse a certain node of a model by themselves, which only support ONNX, CAFFE, TF and TFLITE. The related interface is NodeParser, NodeParserRegistry. model-parse extension: The users can define the process to parse a model by themselves, which only support ONNX, CAFFE, TF and TFLITE. The related interface is ModelParser, ModelParserRegistry. graph-optimization extension: After parsing a model, a graph structure defined by MindSpore will show up and then, the users can define the process to optimize the parsed graph. The related interfaces are PassBase, PassPosition, PassRegistry.

The node-parse extension needs to rely on the flatbuffers, protobuf and the serialization files of third-party frameworks, at the same time, the version of flatbuffers and the protobuf needs to be consistent with that of the released package, the serialized files must be compatible with that used by the released package. Note that the flatbuffers, protobuf and the serialization files are not provided in the released package, users need to compile and generate the serialized files by themselves. The users can obtain the basic information about flabuffers, probobuf, ONNX prototype file, CAFFE prototype file, TF prototype file and TFLITE prototype file from the MindSpore WareHouse.

MindSpore Lite alse providers a series of registration macros to facilitate user access. These macros include node-parse registration REG_NODE_PARSER, model-parse registration REG_MODEL_PARSER, graph-optimization registration REG_PASS and graph-optimization scheduled registration REG_SCHEDULED_PASS

The expansion capability of MindSpore Lite conversion tool only support on Linux system currently.

In this chapter, we will show the users a sample of extending Mindspore Lite converter tool, covering the example of expanding node, example of optimizing graph, compiling and linking. The example will help the users understand the extension ability as soon as possible.

Due to that model-parse extension is a modular extension ability, the chapter will not introduce in details. However, we still provide the users with a simplified unit case for inference.

The chapter takes a add.tflite, which only includes an opreator of adding, as an example. We will show the users how to convert the single operator of adding to that of Custom and finally obtain a model which only includs a single operator of custom.

The code related to the example can be obtained from the directory mindspore/lite/examples/converter_extend.

Node Extension

Self-defined node-parse: The users need to inherit the base class NodeParser, and then, choose a interface to override according to model frameworks.
Node-parse Registration: The users can directly call the registration interface REG_NODE_PARSER, so that the self-defined node-parse will be registered in the converter tool of MindSpore Lite.

class AddParserTutorial : public NodeParser {  // inherit the base class
 public:
  AddParserTutorial() = default;
  ~AddParserTutorial() = default;
  ops::PrimitiveC *Parse(const std::unique_ptr<tflite::OperatorT> &tflite_op,            // override interface
                         const std::unique_ptr<tflite::SubGraphT> &tflite_subgraph,
                         const std::unique_ptr<tflite::ModelT> &tflite_model) override;
};

REG_NODE_PARSER(kFmkTypeTflite, ADD, std::make_shared<AddParserTutorial>());     // call the registration macro

For the sample code, please refer to node_parser.

Model Extension

For the sample code, please refer to the unit case ModelParserRegistryTest.

Optimization Extension

Self-defined Pass: The users need to inherit the base class PassBase, and override the interface function Execute.
Pass Registration: The users can directly call the registration interface REG_PASS, so that the self-defined pass can be registered in the converter tool of MindSpore Lite.

class PassTutorial : public registry::PassBase {  // inherit the base class
 public:
  PassTutorial() : PassBase("PassTutorial") {}

  ~PassTutorial() = default;

  bool Execute(const api::FuncGraphPtr &func_graph) override;     // override interface

 private:
  AnfNodePtr CreateCustomOp(const api::FuncGraphPtr func_graph, const CNodePtr &cnode);
};

using mindspore::registry::POSITION_BEGIN;            // choose a scheduling position
REG_PASS(PassTutorial, opt::PassTutorial)             // register PassBase's subclass
REG_SCHEDULED_PASS(POSITION_BEGIN, {"PassTutorial"})  // register scheduling logic

For the sample code, please refer to pass.

In the offline phase of conversion, we will infer the basic information of output tensors of each node of the model, including the format, data type and shape. So, in this phase, users need to provide the inferring process of self-defined operator. Here, users can refer to Operator Infershape Extension.

Example

Compile

Environment Requirements
- System environment: Linux x86_64; Recommend Ubuntu 18.04.02LTS
- compilation dependencies:
  - CMake >= 3.18.3
  - GCC >= 7.3.0
Compilation preparation

The release package of MindSpore Lite doesn’t provide serialized files of other frameworks, therefore, users need to compile and obtain by yourselves. Here, please refer to Overview.

The case is a tflite model, users need to compile flatbuffers and combine the TFLITE Proto File to generate the serialized file.

After generating, users need to create a directory schema under the directory of mindspore/lite/examples/converter_extend and then place the serialized file in it.
Compilation and Build

Execute the script build.sh in the directory of mindspore/lite/examples/converter_extend. And then, the released package of Mindspore Lite will be downloaded and the demo will be compiled automatically.
```
bash build.sh
```
If the automatic download is failed, users can download the specified package manually, of which the hardware platform is CPU and the system is Ubuntu-x64 mindspore-lite-{version}-linux-x64.tar.gz, After unzipping, please copy the directory of tools/converter/lib and tools/converter/include to the directory of mindspore/lite/examples/converter_extend.

After manually downloading and storing the specified file, users need to execute the build.sh script to complete the compilation and build process.
Compilation Result

The dynamic library libconverter_extend_tutorial.so will be generated in the directory of mindspore/lite/examples/converter_extend/build.

Executing Program

Copy library

Copy the dynamic library libconverter_extend_tutorial.so to the directory of tools/converter/lib of the released package.
Enter the conversion directory of the released package.
```
cd ${PACKAGE_ROOT_PATH}/tools/converter/converter
```

Create extension configuration file(converter.cfg), the content is as follows:

[registry]
plugin_path=libconverter_extend_tutorial.so      # users need to configure the correct path of the dynamic library

Add the required dynamic library to the environment variable LD_LIBRARY_PATH
```
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/tools/converter/lib
```

Execute the script

./converter_lite --fmk=TFLITE --modelFile=add.tflite --configFile=converter.cfg --outputFile=add_extend

The model file add_extend.ms will be generated, the place of which is up to the parameter outputFile.