{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 训练和评估\n", "\n", "[![在线运行](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.7/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9vYnMuZHVhbHN0YWNrLmNuLW5vcnRoLTQubXlodWF3ZWljbG91ZC5jb20vbWluZHNwb3JlLXdlYnNpdGUvbm90ZWJvb2svcjEuNy90dXRvcmlhbHMvemhfY24vYWR2YW5jZWQvdHJhaW4vbWluZHNwb3JlX3RyYWluX2V2YWwuaXB5bmI=&imageid=9d63f4d1-dc09-4873-b669-3483cea777c0) [![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.7/resource/_static/logo_notebook.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.7/tutorials/zh_cn/advanced/train/mindspore_train_eval.ipynb) [![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.7/resource/_static/logo_download_code.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.7/tutorials/zh_cn/advanced/train/mindspore_train_eval.py) [![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.7/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.7/tutorials/source_zh_cn/advanced/train/train_eval.ipynb)\n", "\n", "前面章节讲解了MindSpore构建网络所使用的基本元素，如MindSpore的网络基本单元、损失函数、优化器和评价函数等。\n", "\n", "本章重点介绍如何使用这些元素自定义训练和评估网络。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 构建训练和评估\n", "\n", "构建训练网络首先需要构建前向网络，然后在前向网络的基础上叠加损失函数、反向传播和优化器。\n", "\n", "### 定义数据集\n", "\n", "如下示例定义`get_data`函数生成样本数据及对应的标签，定义`create_dataset`函数加载自定义数据集。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import mindspore.dataset as ds\n", "import numpy as np\n", "\n", "def get_data(num, w=2.0, b=3.0):\n", " \"\"\"生成样本数据及对应的标签\"\"\"\n", " for _ in range(num):\n", " x = np.random.uniform(-10.0, 10.0)\n", " noise = np.random.normal(0, 1)\n", " y = x * w + b + noise\n", " yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32)\n", "\n", "def create_dataset(num_data, batch_size=16):\n", " \"\"\"生成数据集\"\"\"\n", " dataset = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label'])\n", " dataset = dataset.batch(batch_size)\n", " return dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 构建前向网络\n", "\n", "使用`nn.Cell`构建前向网络，如下示例定义一个简单的线性回归网络`LinearNet`：" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import mindspore.nn as nn\n", "from mindspore.common.initializer import Normal\n", "\n", "class LinearNet(nn.Cell):\n", " def __init__(self):\n", " super().__init__()\n", " self.fc = nn.Dense(1, 1, Normal(0.02), Normal(0.02))\n", "\n", " def construct(self, x):\n", " return self.fc(x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 构建训练流程\n", "\n", "MindSpore的`nn`模块提供了训练网络封装函数`TrainOneStepCell`，用来封装网络和优化器。其参数如下：\n", "\n", "- `network`：训练网络，只支持单输出网络。\n", "- `optimizer`： 用于更新网络参数的优化器。\n", "- `sens`：反向传播的输入，缩放系数，默认值为1.0。\n", "\n", "如下示例使用`nn.TrainOneStepCell`将上述定义的线性回归网络封装成一个训练网络，并执行训练，打印损失值。\n", "\n", "示例代码中使用`set_train`通过`mode`参数指定模型是否为训练模式，其中`mode`参数默认为True，即默认情况下为训练模式，若`mode`为False，则为评估或推理模式。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2021-12-31T07:16:16.275751Z", "start_time": "2021-12-31T07:16:16.256615Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch: [0 / 2], step: [0 / 10], loss: 139.95065\n", "Epoch: [0 / 2], step: [1 / 10], loss: 77.1288\n", "Epoch: [0 / 2], step: [2 / 10], loss: 23.511435\n", "Epoch: [0 / 2], step: [3 / 10], loss: 15.275428\n", "Epoch: [0 / 2], step: [4 / 10], loss: 80.57905\n", "Epoch: [0 / 2], step: [5 / 10], loss: 86.396\n", "Epoch: [0 / 2], step: [6 / 10], loss: 78.92796\n", "Epoch: [0 / 2], step: [7 / 10], loss: 16.025606\n", "Epoch: [0 / 2], step: [8 / 10], loss: 2.996492\n", "Epoch: [0 / 2], step: [9 / 10], loss: 9.382026\n", "Epoch: [1 / 2], step: [10 / 10], loss: 46.85878\n", "Epoch: [1 / 2], step: [11 / 10], loss: 78.591515\n", "Epoch: [1 / 2], step: [12 / 10], loss: 39.523586\n", "Epoch: [1 / 2], step: [13 / 10], loss: 3.0048246\n", "Epoch: [1 / 2], step: [14 / 10], loss: 7.835808\n", "Epoch: [1 / 2], step: [15 / 10], loss: 27.37307\n", "Epoch: [1 / 2], step: [16 / 10], loss: 34.076313\n", "Epoch: [1 / 2], step: [17 / 10], loss: 54.53374\n", "Epoch: [1 / 2], step: [18 / 10], loss: 19.80341\n", "Epoch: [1 / 2], step: [19 / 10], loss: 1.8542566\n" ] } ], "source": [ "# 生成训练数据集\n", "train_dataset = create_dataset(num_data=160, batch_size=16)\n", "\n", "net = LinearNet()\n", "loss = nn.MSELoss()\n", "\n", "# 连接前向网络与损失函数\n", "net_with_loss = nn.WithLossCell(net, loss)\n", "opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9)\n", "\n", "# 定义训练网络，封装网络和优化器\n", "train_net = nn.TrainOneStepCell(net_with_loss, opt)\n", "# 设置网络为训练模式\n", "train_net.set_train()\n", "\n", "# 真正训练迭代过程\n", "step = 0\n", "epochs = 2\n", "steps = train_dataset.get_dataset_size()\n", "\n", "for epoch in range(epochs):\n", " for d in train_dataset.create_dict_iterator():\n", " result = train_net(d[\"data\"], d[\"label\"])\n", " print(f\"Epoch: [{epoch} / {epochs}], \"\n", " f\"step: [{step} / {steps}], \"\n", " f\"loss: {result}\")\n", " step = step + 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 构建评估流程\n", "\n", "MindSpore的`nn`模块提供了评估网络封装函数`WithEvalCell`，用来在验证集上评估模型训练的效果。其参数如下：\n", "\n", "- `network`：前向网络。\n", "- `loss_fn`：损失函数。\n", "- `add_cast_fp32`：是否将数据类型调整为float32。\n", "\n", "`nn.WithEvalCell`只接受两个输入，分别为数据`data`及其对应的标签`label`，用前面定义的前向网络和损失函数构建一个评估网络，示例如下：" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "ExecuteTime": { "end_time": "2021-12-31T07:16:16.821621Z", "start_time": "2021-12-31T07:16:16.792286Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "mae: 2.9597126245498657\n", "loss: 11.539738941192628\n" ] } ], "source": [ "eval_dataset = create_dataset(num_data=160, batch_size=16)\n", "\n", "# 构建评估网络\n", "eval_net = nn.WithEvalCell(net, loss)\n", "eval_net.set_train(False)\n", "loss = nn.Loss()\n", "mae = nn.MAE()\n", "\n", "mae.clear()\n", "loss.clear()\n", "\n", "# 真正验证迭代过程\n", "for data in eval_dataset.create_dict_iterator():\n", " outputs = eval_net(data[\"data\"], data[\"label\"])\n", " mae.update(outputs[1], outputs[2])\n", " loss.update(outputs[0])\n", "\n", "# 评估结果\n", "mae_result = mae.eval()\n", "loss_result = loss.eval()\n", "\n", "print(\"mae: \", mae_result)\n", "print(\"loss: \", loss_result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 自定义训练和评估\n", "\n", "### 自定义训练网络\n", "\n", "[自定义损失函数章节](https://www.mindspore.cn/tutorials/zh-CN/r1.7/advanced/network/loss.html#自定义损失函数)已经介绍了使用`nn.WithLossCell`将前向网络与损失函数连接起来，本节将介绍如何自定义训练网络。\n", "\n", "如下示例定义`CustomTrainOneStepCell`函数来封装网络和优化器。" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch: [0 / 2], step: [0 / 10], loss: 70.774574\n", "Epoch: [0 / 2], step: [1 / 10], loss: 71.33737\n", "Epoch: [0 / 2], step: [2 / 10], loss: 63.126896\n", "Epoch: [0 / 2], step: [3 / 10], loss: 8.946123\n", "Epoch: [0 / 2], step: [4 / 10], loss: 32.131054\n", "Epoch: [0 / 2], step: [5 / 10], loss: 38.90644\n", "Epoch: [0 / 2], step: [6 / 10], loss: 126.410255\n", "Epoch: [0 / 2], step: [7 / 10], loss: 41.496185\n", "Epoch: [0 / 2], step: [8 / 10], loss: 5.7309575\n", "Epoch: [0 / 2], step: [9 / 10], loss: 16.104172\n", "Epoch: [1 / 2], step: [10 / 10], loss: 26.39038\n", "Epoch: [1 / 2], step: [11 / 10], loss: 52.73621\n", "Epoch: [1 / 2], step: [12 / 10], loss: 38.053413\n", "Epoch: [1 / 2], step: [13 / 10], loss: 4.555399\n", "Epoch: [1 / 2], step: [14 / 10], loss: 1.8704597\n", "Epoch: [1 / 2], step: [15 / 10], loss: 11.614007\n", "Epoch: [1 / 2], step: [16 / 10], loss: 25.868422\n", "Epoch: [1 / 2], step: [17 / 10], loss: 26.153322\n", "Epoch: [1 / 2], step: [18 / 10], loss: 9.847598\n", "Epoch: [1 / 2], step: [19 / 10], loss: 2.0711172\n" ] } ], "source": [ "import mindspore.ops as ops\n", "\n", "class CustomTrainOneStepCell(nn.Cell):\n", " \"\"\"自定义训练网络\"\"\"\n", "\n", " def __init__(self, network, optimizer):\n", " \"\"\"入参有两个：训练网络，优化器\"\"\"\n", " super(CustomTrainOneStepCell, self).__init__(auto_prefix=False)\n", " self.network = network # 定义前向网络\n", " self.network.set_grad() # 构建反向网络\n", " self.optimizer = optimizer # 定义优化器\n", " self.weights = self.optimizer.parameters # 待更新参数\n", " self.grad = ops.GradOperation(get_by_list=True) # 反向传播获取梯度\n", "\n", " def construct(self, *inputs):\n", " loss = self.network(*inputs) # 计算当前输入的损失函数值\n", " grads = self.grad(self.network, self.weights)(*inputs) # 进行反向传播，计算梯度\n", " self.optimizer(grads) # 使用优化器更新权重参数\n", " return loss\n", "\n", "net1 = LinearNet() # 定义前向网络\n", "loss = nn.MSELoss() # 损失函数\n", "\n", "\n", "# 连接前向网络与损失函数\n", "net_with_loss = nn.WithLossCell(net1, loss)\n", "opt = nn.Momentum(net1.trainable_params(), learning_rate=0.005, momentum=0.9)\n", "\n", "# 定义训练网络，封装网络和优化器\n", "train_net = CustomTrainOneStepCell(net_with_loss, opt)\n", "# 设置网络为训练模式\n", "train_net.set_train()\n", "\n", "# 真正训练迭代过程\n", "step = 0\n", "epochs = 2\n", "steps = train_dataset.get_dataset_size()\n", "for epoch in range(epochs):\n", " for d in train_dataset.create_dict_iterator():\n", " result = train_net(d[\"data\"], d[\"label\"])\n", " print(f\"Epoch: [{epoch} / {epochs}], \"\n", " f\"step: [{step} / {steps}], \"\n", " f\"loss: {result}\")\n", " step = step + 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 自定义评估网络\n", "\n", "由于`nn.WithEvalCell`只有两个输入`data`和`label`，对于多个数据或多个标签的场景显然不适用，此时如果想要构建评估网络就需要自定义评估网络。\n", "\n", "在自定义时，如不需要损失函数作为评价指标，则无需定义`loss_fn`。当输入为多数据或多标签时，可参考如下示例来自定义评估网络。" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "CustomWithEvalCell<\n", " (network): LinearNet<\n", " (fc): Dense\n", " >\n", " >" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class CustomWithEvalCell(nn.Cell):\n", "\n", " def __init__(self, network):\n", " super(CustomWithEvalCell, self).__init__(auto_prefix=False)\n", " self.network = network\n", "\n", " def construct(self, data, label1, label2):\n", " \"\"\"输入数据为三个：一个数据及其对应的两个标签\"\"\"\n", " outputs = self.network(data)\n", " return outputs, label1, label2\n", "\n", "custom_eval_net = CustomWithEvalCell(net)\n", "custom_eval_net.set_train(False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 网络的权重共享\n", "\n", "通过前面的介绍可以看出，前向网络、训练网络和评估网络具有不同的逻辑，因此在需要时我们会构建三张网络。若使用训练好的模型进行评估和推理，需要推理和评估网络中的权重值与训练网络中相同。\n", "\n", "使用模型保存和加载接口，将训练好的模型保存下来，再加载到评估和推理网络中，可以确保权重值相同。在训练平台上完成模型训练，但在推理平台进行推理时，模型保存与加载是必不可少的。\n", "\n", "在网络调测过程中，或使用边训练边推理方式进行模型调优时，往往在同一Python脚本中完成模型训练，评估或推理，此时MindSpore的权重共享机制可确保不同网络间的权重一致性。\n", "\n", "使用MindSpore构建不同网络结构时，只要这些网络结构是在一个实例的基础上封装的，那这个实例中的所有权重便是共享的，一个网络中的权重发生变化，意味着其他网络中的权重同步发生了变化。\n", "\n", "如下示例中，定义训练和评估网络时便使用了权重共享机制：" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "ExecuteTime": { "end_time": "2021-12-31T07:16:17.017589Z", "start_time": "2021-12-31T07:16:16.991264Z" } }, "outputs": [ { "data": { "text/plain": [ "WithEvalCell<\n", " (_network): LinearNet<\n", " (fc): Dense\n", " >\n", " (_loss_fn): MSELoss<>\n", " >" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 实例化前向网络\n", "net = LinearNet()\n", "# 设定损失函数并连接前向网络与损失函数\n", "loss = nn.MSELoss()\n", "net_with_loss = nn.WithLossCell(net, loss)\n", "# 设定优化器\n", "opt = nn.Adam(params=net.trainable_params())\n", "\n", "# 定义训练网络\n", "train_net = nn.TrainOneStepCell(net_with_loss, opt)\n", "train_net.set_train()\n", "\n", "# 构建评估网络\n", "eval_net = nn.WithEvalCell(net, loss)\n", "eval_net.set_train(False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`train_net`和`eval_net`均在`net`实例的基础上封装，因此在进行模型评估时，不需要加载`train_net`的权重。\n", "\n", "若在构建`eval_net`时重新的定义前向网络，那`train_net`和`eval_net`之间便没有共享权重，如下：" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "ExecuteTime": { "end_time": "2021-12-31T07:16:17.046932Z", "start_time": "2021-12-31T07:16:17.018611Z" } }, "outputs": [ { "data": { "text/plain": [ "WithEvalCell<\n", " (_network): LinearNet<\n", " (fc): Dense\n", " >\n", " (_loss_fn): MSELoss<>\n", " >" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 定义训练网络\n", "train_net = nn.TrainOneStepCell(net_with_loss, opt)\n", "train_net.set_train()\n", "\n", "# 再次实例化前向网络\n", "net2 = LinearNet()\n", "# 构建评估网络\n", "eval_net = nn.WithEvalCell(net2, loss)\n", "eval_net.set_train(False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "此时，若要在模型训练后进行评估，就需要将`train_net`中的权重加载到`eval_net`中。在同一脚本中进行模型训练、评估和推理时，利用好权重共享机制不失为一种更简便的方式。\n", "\n", "在使用Model进行训练时，对于简单的场景，`Model`内部使用`nn.WithLossCell`、`nn.TrainOneStepCell`和`nn.WithEvalCell`在前向`network`实例的基础上构建训练和评估网络，`Model`本身确保了推理、训练、评估网络之间权重共享。\n", "\n", "但对于自定义使用Model的场景，用户需要注意前向网络仅实例化一次。如果构建训练网络和评估网络时分别实例化前向网络，那在使用`eval`进行模型评估时，便需要手动加载训练网络中的权重，否则模型评估使用的将是初始的权重值。" ] } ], "metadata": { "kernelspec": { "display_name": "MindSpore", "language": "python", "name": "mindspore" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 4 }