{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 中低阶API实现深度学习\n",
    "\n",
    "`Ascend` `GPU` `CPU` `进阶` `全流程`\n",
    "\n",
    "[![在线运行](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9yMS42L3R1dG9yaWFscy96aF9jbi9pbnRlcm1lZGlhdGUvbWluZHNwb3JlX21pZF9sb3dfbGV2ZWxfYXBpLmlweW5i&imageid=59a6e9f5-93c0-44dd-85b0-82f390c5d53b) [![下载Notebook](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_notebook.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.6/tutorials/zh_cn/intermediate/mindspore_mid_low_level_api.ipynb) [![下载样例代码](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_download_code.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.6/tutorials/zh_cn/intermediate/mindspore_mid_low_level_api.py) [![查看源文件](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.6/tutorials/source_zh_cn/intermediate/mid_low_level_api.ipynb)\n",
    "\n",
    "为方便用户控制整网的执行流程，MindSpore提供了高阶的训练和推理接口`mindspore.Model`，通过指定要训练的神经网络模型和常见的训练设置，调用`train`和`eval`方法对网络进行训练和推理。同时，用户如果想要对特定模块进行个性化设置，也可以调用对应的中低阶接口自行定义，本文介绍了如何使用中低阶API定义各个模块。\n",
    "\n",
    "> 你可以在这里下载完整的样例代码：<https://gitee.com/mindspore/docs/blob/r1.6/docs/sample_code/mid_low_level_api.py>。\n",
    "\n",
    "## 定义数据集\n",
    "\n",
    "MindSpore的`mindspore.dataset`模块集成了常见的数据处理功能：用户既可以调用此模块的相关接口来[加载常见的数据集](https://www.mindspore.cn/tutorials/zh-CN/r1.6/dataset.html#%E5%8A%A0%E8%BD%BD%E6%95%B0%E6%8D%AE%E9%9B%86)，也可以构造数据集类并配合使用`GeneratorDataset`接口，实现自定义数据集和数据加载。使用`GeneratorDataset`生成具有多项式关系的样本和对应的结果，代码样例如下：\n",
    "\n",
    "导入所需的包："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:20.862099Z",
     "start_time": "2021-12-22T06:58:18.832643Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import mindspore as ms\n",
    "from mindspore import ops, nn\n",
    "from mindspore import dtype as mstype\n",
    "import mindspore.dataset as ds\n",
    "import mindspore.common.initializer as init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:20.872190Z",
     "start_time": "2021-12-22T06:58:20.862099Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_data(data_num, data_size):\n",
    "    for _ in range(data_num):\n",
    "        data = np.random.randn(data_size)\n",
    "        p = np.array([1, 0, -3, 5])\n",
    "        label = np.polyval(p, data).sum()\n",
    "        yield data.astype(np.float32), np.array([label]).astype(np.float32)\n",
    "\n",
    "def create_dataset(data_num, data_size, batch_size=32, repeat_size=1):\n",
    "    \"\"\"定义数据集\"\"\"\n",
    "    input_data = ds.GeneratorDataset(list(get_data(data_num, data_size)), column_names=['data', 'label'])\n",
    "    input_data = input_data.batch(batch_size)\n",
    "    input_data = input_data.repeat(repeat_size)\n",
    "    return input_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义网络\n",
    "\n",
    "MindSpore的Cell类是构建所有网络的基类，也是网络的基本单元。当用户需要自定义网络时，可以继承Cell类，并重写`__init__`方法和`construct`方法。MindSpore的`ops`模块提供了基础算子的实现，`nn`模块实现了对基础算子的进一步封装，用户可以根据需要，灵活使用不同的算子。\n",
    "\n",
    "使用常用的`nn`算子构建一个简单的全连接网络："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:20.890664Z",
     "start_time": "2021-12-22T06:58:20.874805Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyNet(nn.Cell):\n",
    "    \"\"\"定义网络\"\"\"\n",
    "    def __init__(self, input_size=32):\n",
    "        super(MyNet, self).__init__()\n",
    "        self.fc1 = nn.Dense(input_size, 120, weight_init=init.Normal(0.02))\n",
    "        self.fc2 = nn.Dense(120, 84, weight_init=init.Normal(0.02))\n",
    "        self.fc3 = nn.Dense(84, 1, weight_init=init.Normal(0.02))\n",
    "        self.relu = nn.ReLU()\n",
    "\n",
    "    def construct(self, x):\n",
    "        x = self.relu(self.fc1(x))\n",
    "        x = self.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义损失函数\n",
    "\n",
    "损失函数用于衡量预测值与真实值差异的程度。深度学习中，模型训练就是通过不停地迭代来缩小损失函数值的过程。因此在模型训练过程中损失函数的选择非常重要，定义一个好的损失函数可以帮助loss更快收敛，达到更好的精度。\n",
    "\n",
    "MindSpore提供了许多通用损失函数供用户选择，也支持用户根据需要自定义损失函数。定义损失函数类时，既可以继承网络的基类`nn.Cell`，也可以继承损失函数的基类`nn.LossBase`。\n",
    "\n",
    "### 继承Cell定义损失函数\n",
    "\n",
    "`Cell`是MindSpore的基本网络单元，可以用于构建网络，损失函数也可以通过`Cell`来定义。使用`Cell`定义损失函数的方法与定义一个普通的网络的差别在于，其执行逻辑用于计算前向网络输出与真实值之间的误差。\n",
    "\n",
    "以平均绝对误差为例，损失函数的定义方法如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:20.904946Z",
     "start_time": "2021-12-22T06:58:20.893198Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyL1Loss(nn.Cell):\n",
    "    \"\"\"定义损失\"\"\"\n",
    "    def __init__(self):\n",
    "        super(MyL1Loss, self).__init__()\n",
    "        self.abs = ops.Abs()\n",
    "        self.reduce_mean = ops.ReduceMean()\n",
    "\n",
    "    def construct(self, predict, target):\n",
    "        x = self.abs(predict - target)\n",
    "        return self.reduce_mean(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 继承基类定义损失函数\n",
    "\n",
    "在定义损失函数时还可以继承损失函数的基类`nn.LossBase`。损失函数的基类在`Cell`的基础上，提供了`get_loss`方法，用于对损失值求和或求均值，输出一个标量。MyL1Loss使用`LossBase`作为基类的定义如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:20.931378Z",
     "start_time": "2021-12-22T06:58:20.906995Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyL1Loss(nn.LossBase):\n",
    "    \"\"\"定义损失\"\"\"\n",
    "    def __init__(self, reduction=\"mean\"):\n",
    "        super(MyL1Loss, self).__init__(reduction)\n",
    "        self.abs = ops.Abs()\n",
    "\n",
    "    def construct(self, base, target):\n",
    "        x = self.abs(base - target)\n",
    "        return self.get_loss(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义优化器\n",
    "\n",
    "优化器在模型训练过程中，用于计算和更新网络参数，合适的优化器可以有效减少训练时间，提高最终模型性能。\n",
    "\n",
    "MindSpore提供了许多通用的优化器供用户选择，同时也支持用户根据需要自定义优化器。自定义优化器时可以继承优化器基类`optimizer`，重写`construct`函数实现参数的更新。\n",
    "\n",
    "使用基础的运算算子自定义优化器："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:20.992077Z",
     "start_time": "2021-12-22T06:58:20.933414Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyMomentum(nn.Optimizer):\n",
    "    \"\"\"定义优化器\"\"\"\n",
    "    def __init__(self, params, learning_rate, momentum=0.9, use_nesterov=False):\n",
    "        super(MyMomentum, self).__init__(learning_rate, params)\n",
    "        self.momentum = ms.Parameter(ms.Tensor(momentum, mstype.float32), name=\"momentum\")\n",
    "        self.use_nesterov = use_nesterov\n",
    "        self.moments = self.parameters.clone(prefix=\"moments\", init=\"zeros\")\n",
    "        self.assign = ops.Assign()\n",
    "\n",
    "    def construct(self, gradients):\n",
    "        lr = self.get_lr()\n",
    "        params = self.parameters\n",
    "        for i in range(len(params)):\n",
    "            self.assign(self.moments[i], self.moments[i] * self.momentum + gradients[i])\n",
    "            if self.use_nesterov:\n",
    "                update = params[i] - (self.moments[i] * self.momentum + gradients[i]) * lr\n",
    "            else:\n",
    "                update = params[i] - self.moments[i] * lr\n",
    "            self.assign(params[i], update)\n",
    "        return params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MindSpore也封装了`ApplyMomentum`算子供用户使用，使用`ApplyMomentum`算子自定义优化器："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:21.005876Z",
     "start_time": "2021-12-22T06:58:20.994115Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyMomentum(nn.Optimizer):\n",
    "    \"\"\"定义优化器\"\"\"\n",
    "    def __init__(self, params, learning_rate, momentum=0.9, use_nesterov=False):\n",
    "        super(MyMomentum, self).__init__(learning_rate, params)\n",
    "        self.moments = self.parameters.clone(prefix=\"moments\", init=\"zeros\")\n",
    "        self.momentum = momentum\n",
    "        self.opt = ops.ApplyMomentum(use_nesterov=use_nesterov)\n",
    "\n",
    "    def construct(self, gradients):\n",
    "        params = self.parameters\n",
    "        success = None\n",
    "        for param, mom, grad in zip(params, self.moments, gradients):\n",
    "            success = self.opt(param, mom, self.learning_rate, grad, self.momentum)\n",
    "        return success"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义训练流程\n",
    "\n",
    "MindSpore的nn模块提供了部分封装接口方便用户在训练过程中使用，例如，`nn.WithLossCell`接口可以将前向网络与损失函数连接起来；`nn.TrainOneStepCell`封装了损失网络和优化器，在执行训练时通过`GradOperation`算子来进行梯度的获取，通过优化器来实现权重的更新。\n",
    "\n",
    "定义损失网络："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:21.023142Z",
     "start_time": "2021-12-22T06:58:21.008460Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyWithLossCell(nn.Cell):\n",
    "    \"\"\"定义损失网络\"\"\"\n",
    "    def __init__(self, backbone, loss_fn):\n",
    "        super(MyWithLossCell, self).__init__(auto_prefix=False)\n",
    "        self.backbone = backbone\n",
    "        self.loss_fn = loss_fn\n",
    "\n",
    "    def construct(self, data, label):\n",
    "        out = self.backbone(data)\n",
    "        return self.loss_fn(out, label)\n",
    "\n",
    "    def backbone_network(self):\n",
    "        return self.backbone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MindSpore的nn模块提供了训练网络封装函数`nn.TrainOneStepCell`，如无特殊需求，用户可以继承`nn.TrainOneStepCell`来定义自己的训练流程。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:21.040020Z",
     "start_time": "2021-12-22T06:58:21.025185Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyTrainStep(nn.TrainOneStepCell):\n",
    "    \"\"\"定义训练流程\"\"\"\n",
    "    def __init__(self, network, optimizer):\n",
    "        \"\"\"参数初始化\"\"\"\n",
    "        super(MyTrainStep, self).__init__(network, optimizer)\n",
    "        self.grad = ops.GradOperation(get_by_list=True)\n",
    "\n",
    "    def construct(self, data, label):\n",
    "        \"\"\"构建训练过程\"\"\"\n",
    "        weights = self.weights\n",
    "        loss = self.network(data, label)\n",
    "        grads = self.grad(self.network, weights)(data, label)\n",
    "        return loss, self.optimizer(grads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "执行训练："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:22.681175Z",
     "start_time": "2021-12-22T06:58:21.041577Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "108.16456\n",
      "18.994844\n",
      "10.551674\n",
      "16.937834\n",
      "15.274784\n"
     ]
    }
   ],
   "source": [
    "# 生成多项式分布的训练数据\n",
    "dataset_size = 32\n",
    "ds_train = create_dataset(2048, dataset_size)\n",
    "# 网络\n",
    "net = MyNet()\n",
    "# 损失函数\n",
    "loss_func = MyL1Loss()\n",
    "# 优化器\n",
    "opt = MyMomentum(net.trainable_params(), 0.01)\n",
    "# 构建损失网络\n",
    "net_with_criterion = MyWithLossCell(net, loss_func)\n",
    "# 构建训练网络\n",
    "train_net = MyTrainStep(net_with_criterion, opt)\n",
    "# 执行训练，每个epoch打印一次损失值\n",
    "epochs = 5\n",
    "for epoch in range(epochs):\n",
    "    for train_x, train_y in ds_train:\n",
    "        train_net(train_x, train_y)\n",
    "        loss_val = net_with_criterion(train_x, train_y)\n",
    "    print(loss_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义metric\n",
    "\n",
    "当训练任务结束，常常需要评估函数来评估模型的好坏，MindSpore的`metric`模块提供了常见的评估函数，用户也可以根据需要自行定义评估指标："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:22.707013Z",
     "start_time": "2021-12-22T06:58:22.683370Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyMAE(nn.Metric):\n",
    "    \"\"\"定义metric\"\"\"\n",
    "    def __init__(self):\n",
    "        super(MyMAE, self).__init__()\n",
    "        self.clear()\n",
    "\n",
    "    def clear(self):\n",
    "        self.abs_error_sum = 0\n",
    "        self.samples_num = 0\n",
    "\n",
    "    def update(self, *inputs):\n",
    "        y_pred = inputs[0].asnumpy()\n",
    "        y = inputs[1].asnumpy()\n",
    "        error_abs = np.abs(y.reshape(y_pred.shape) - y_pred)\n",
    "        self.abs_error_sum += error_abs.sum()\n",
    "        self.samples_num += y.shape[0]\n",
    "\n",
    "    def eval(self):\n",
    "        return self.abs_error_sum / self.samples_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义验证流程\n",
    "\n",
    "MindSpore的nn模块提供了评估网络包装函数`nn.WithEvalCell`，用户也可以自己定义评估网络包装函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:22.726360Z",
     "start_time": "2021-12-22T06:58:22.709549Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyWithEvalCell(nn.Cell):\n",
    "    \"\"\"定义验证流程\"\"\"\n",
    "    def __init__(self, network):\n",
    "        super(MyWithEvalCell, self).__init__(auto_prefix=False)\n",
    "        self.network = network\n",
    "\n",
    "    def construct(self, data, label):\n",
    "        outputs = self.network(data)\n",
    "        return outputs, label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "执行推理并评估：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:22.935251Z",
     "start_time": "2021-12-22T06:58:22.729350Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mae:  15.619880557060242\n"
     ]
    }
   ],
   "source": [
    "# 获取验证数据\n",
    "ds_eval = create_dataset(128, dataset_size, 1)\n",
    "# 定义评估网络\n",
    "eval_net = MyWithEvalCell(net)\n",
    "eval_net.set_train(False)\n",
    "# 定义评估指标\n",
    "mae = MyMAE()\n",
    "# 执行推理过程\n",
    "for eval_x, eval_y in ds_eval:\n",
    "    output, eval_y = eval_net(eval_x, eval_y)\n",
    "    mae.update(output, eval_y)\n",
    "\n",
    "mae_result = mae.eval()\n",
    "print(\"mae: \", mae_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输出评估误差，MAE与模型在训练集上效果大致相同。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型\n",
    "\n",
    "使用MindSpore提供的`save_checkpoint`保存模型，传入网络和保存路径："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-22T06:58:22.943387Z",
     "start_time": "2021-12-22T06:58:22.936263Z"
    }
   },
   "outputs": [],
   "source": [
    "ms.save_checkpoint(net, \"./MyNet.ckpt\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore",
   "language": "python",
   "name": "mindspore"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}