{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_notebook.svg)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/master/tutorials/zh_cn/advanced/modules/mindspore_loss.ipynb) [![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_download_code.svg)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/master/tutorials/zh_cn/advanced/modules/mindspore_loss.py) [![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_source.svg)](https://gitee.com/mindspore/docs/blob/master/tutorials/source_zh_cn/advanced/modules/loss.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 损失函数\n",
    "\n",
    "损失函数，亦称目标函数，用于衡量预测值与真实值差异的程度。\n",
    "\n",
    "在深度学习中，模型训练就是通过不断迭代来缩小损失函数值的过程。因此，在模型训练过程中损失函数的选择非常重要，一个好的损失函数能有效提升模型的性能。\n",
    "\n",
    "`mindspore.nn`模块中提供了许多[通用损失函数](https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.nn.html#损失函数)，但这些通用损失函数无法满足所有需求，很多情况需要用户自定义所需的损失函数。因此，本教程介绍如何自定义损失函数。\n",
    "\n",
    "![lossfun.png](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_zh_cn/advanced/modules/images/loss_function.png)\n",
    "\n",
    "## 内置损失函数\n",
    "\n",
    "首先介绍`mindspore.nn`模块中内置的[损失函数](https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.nn.html#损失函数)。\n",
    "\n",
    "下例以`nn.L1Loss`为例，计算预测值和目标值之间的平均绝对误差：\n",
    "\n",
    "$$\\ell(x, y) = L = \\{l_1,\\dots,l_N\\}^\\top, \\quad \\text{with } l_n = \\left| x_n - y_n \\right|$$\n",
    "\n",
    "其中N为数据集中的`batch_size`值。\n",
    "\n",
    "$$\\ell(x, y) =\n",
    "        \\begin{cases}\n",
    "            \\operatorname{mean}(L), & \\text{if reduction} = \\text{'mean';}\\\\\n",
    "            \\operatorname{sum}(L),  & \\text{if reduction} = \\text{'sum'.}\n",
    "        \\end{cases}$$\n",
    "\n",
    "`nn.L1Loss`中的参数`reduction`取值可为`mean`，`sum`，或`none`。若`reduction` 为`mean`或`sum`，则输出一个经过均值或求和后的标量Tensor（降维）；若`reduction`为`none`，则所输出Tensor的shape为广播后的shape。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T03:42:22.717822Z",
     "start_time": "2021-12-29T03:42:20.636585Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 1.5\n",
      "loss_sum: 9.0\n",
      "loss_none: [[1. 0. 2.]\n",
      " [1. 2. 3.]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from mindspore import nn\n",
    "from mindspore import Tensor\n",
    "\n",
    "# 输出loss均值\n",
    "loss = nn.L1Loss()\n",
    "# 输出loss和\n",
    "loss_sum = nn.L1Loss(reduction='sum')\n",
    "# 输出loss原值\n",
    "loss_none = nn.L1Loss(reduction='none')\n",
    "\n",
    "input_data = Tensor(np.array([[1, 2, 3], [2, 3, 4]]).astype(np.float32))\n",
    "target_data = Tensor(np.array([[0, 2, 5], [3, 1, 1]]).astype(np.float32))\n",
    "\n",
    "print(\"loss:\", loss(input_data, target_data))\n",
    "print(\"loss_sum:\", loss_sum(input_data, target_data))\n",
    "print(\"loss_none:\", loss_none(input_data, target_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义损失函数\n",
    "\n",
    "自定义损失函数的方法有两种：一是基于`nn.Cell`来定义损失函数；二是`nn.LossBase`来定义损失函数。`nn.LossBase`继承自`nn.Cell`，额外提供了`get_loss`方法，利用`reduction`参数对损失值求和或求均值，输出一个标量。\n",
    "\n",
    "下面将分别使用继承`Cell`和继承`LossBase`的方法，来定义平均绝对误差损失函数(Mean Absolute Error，MAE)，MAE算法的公式如下所示：\n",
    "\n",
    "$$ loss= \\frac{1}{m}\\sum_{i=1}^m\\lvert y_i-f(x_i) \\rvert$$\n",
    "\n",
    "上式中$f(x)$为预测值，$y$为样本真实值，$loss$为预测值与真实值之间距离的平均值。\n",
    "\n",
    "### 基于`nn.Cell`构造损失函数\n",
    "\n",
    "`nn.Cell`是MindSpore的基类，不但可用于构建网络，还可用于定义损失函数。使用`nn.Cell`定义损失函数的过程与定义一个普通的网络相似，差别在于，其执行逻辑部分要计算的是前向网络输出与真实值之间的误差。\n",
    "\n",
    "下面展示怎样基于`nn.Cell`自定义损失函数`MAELoss`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T03:42:22.729232Z",
     "start_time": "2021-12-29T03:42:22.723517Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.033333335\n"
     ]
    }
   ],
   "source": [
    "from mindspore import ops\n",
    "import mindspore as ms\n",
    "\n",
    "class MAELoss(nn.Cell):\n",
    "    \"\"\"自定义损失函数MAELoss\"\"\"\n",
    "    def construct(self, base, target):\n",
    "        return ops.abs(base - target).mean()\n",
    "\n",
    "loss = MAELoss()\n",
    "\n",
    "input_data = Tensor(np.array([0.1, 0.2, 0.3]).astype(np.float32))  # 生成预测值\n",
    "target_data = Tensor(np.array([0.1, 0.2, 0.2]).astype(np.float32)) # 生成真实值\n",
    "\n",
    "output = loss(input_data, target_data)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 基于`nn.LossBase`构造损失函数\n",
    "\n",
    "基于[nn.LossBase](https://www.mindspore.cn/docs/zh-CN/master/api_python/nn/mindspore.nn.LossBase.html#mindspore.nn.LossBase)构造损失函数`MAELoss`与基于`nn.Cell`构造损失函数的过程类似，都要重写`__init__`方法和`construct`方法。\n",
    "\n",
    "`nn.LossBase`可使用方法`get_loss`将`reduction`应用于损失计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T03:42:22.766767Z",
     "start_time": "2021-12-29T03:42:22.759510Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.033333335\n"
     ]
    }
   ],
   "source": [
    "class MAELoss(nn.LossBase):\n",
    "    \"\"\"自定义损失函数MAELoss\"\"\"\n",
    "    def construct(self, base, target):\n",
    "        x = ops.abs(base - target)\n",
    "        return self.get_loss(x)  # 返回loss均值\n",
    "\n",
    "loss = MAELoss()\n",
    "\n",
    "input_data = Tensor(np.array([0.1, 0.2, 0.3]).astype(np.float32))  # 生成预测值\n",
    "target_data = Tensor(np.array([0.1, 0.2, 0.2]).astype(np.float32))  # 生成真实值\n",
    "\n",
    "output = loss(input_data, target_data)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 损失函数与模型训练\n",
    "\n",
    "损失函数`MAELoss`自定义完成后，可使用MindSpore的接口[Model](https://www.mindspore.cn/docs/zh-CN/master/api_python/train/mindspore.train.Model.html#mindspore.train.Model)中`train`接口进行模型训练，构造`Model`时需传入前向网络、损失函数和优化器，`Model`会在内部将它们关联起来，生成一个可用于训练的网络模型。\n",
    "\n",
    "在`Model`中，前向网络和损失函数通过[nn.WithLossCell](https://www.mindspore.cn/docs/zh-CN/master/api_python/nn/mindspore.nn.WithLossCell.html#mindspore.nn.WithLossCell)关联起来，`nn.WithLossCell`支持两个输入，分别为`data`和`label`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T03:42:23.488075Z",
     "start_time": "2021-12-29T03:42:23.312491Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1 step: 10, loss is 6.525373935699463\n",
      "epoch: 2 step: 10, loss is 4.005467414855957\n",
      "epoch: 3 step: 10, loss is 2.1115174293518066\n",
      "epoch: 4 step: 10, loss is 2.7334954738616943\n",
      "epoch: 5 step: 10, loss is 1.7042752504348755\n",
      "epoch: 6 step: 10, loss is 1.6317998170852661\n",
      "epoch: 7 step: 10, loss is 1.035435438156128\n",
      "epoch: 8 step: 10, loss is 0.6060740351676941\n",
      "epoch: 9 step: 10, loss is 1.0374044179916382\n",
      "epoch: 10 step: 10, loss is 0.736151397228241\n"
     ]
    }
   ],
   "source": [
    "from mindspore.train import Model, LossMonitor\n",
    "from mindspore.dataset import GeneratorDataset\n",
    "\n",
    "def get_data(num, w=2.0, b=3.0):\n",
    "    \"\"\"生成数据及对应标签\"\"\"\n",
    "    for _ in range(num):\n",
    "        x = np.random.uniform(-10.0, 10.0)\n",
    "        noise = np.random.normal(0, 1)\n",
    "        y = x * w + b + noise\n",
    "        yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32)\n",
    "\n",
    "def create_dataset(num_data, batch_size=16):\n",
    "    \"\"\"加载数据集\"\"\"\n",
    "    dataset = GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label'])\n",
    "    dataset = dataset.batch(batch_size)\n",
    "    return dataset\n",
    "\n",
    "train_dataset = create_dataset(num_data=160)\n",
    "network = nn.Dense(1, 1)\n",
    "loss_fn = MAELoss()\n",
    "optimizer = nn.Momentum(network.trainable_params(), learning_rate=0.005, momentum=0.9)\n",
    "\n",
    "# 使用model接口将网络、损失函数和优化器关联起来\n",
    "model = Model(network, loss_fn, optimizer)\n",
    "model.train(10, train_dataset, callbacks=[LossMonitor(10)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多标签损失函数与模型训练\n",
    "\n",
    "上述定义了一个简单的平均绝对误差损失函数`MAELoss`，但许多深度学习应用的数据集较复杂，如目标检测网络Faster R-CNN的数据中就包含多个标签，而非简单的一条数据对应一个标签，这时损失函数的定义和使用略有不同。\n",
    "\n",
    "本节介绍在多标签数据集场景下，如何定义多标签损失函数（Multi label loss function），并使用Model进行模型训练。\n",
    "\n",
    "### 多标签数据集\n",
    "\n",
    "下例通过`get_multilabel_data`函数拟合两组线性数据$y1$和$y2$，拟合的目标函数为：\n",
    "\n",
    "$$f(x)=2x+3$$\n",
    "\n",
    "最终数据集应随机分布于函数周边，这里按以下公式的方式生成，其中`noise`为服从标准正态分布的随机值。`get_multilabel_data`函数返回数据$x$、$y1$和$y2$：\n",
    "\n",
    "$$f(x)=2x+3+noise$$\n",
    "\n",
    "通过`create_multilabel_dataset`生成多标签数据集，并将`GeneratorDataset`中的`column_names`参数设置为['data', 'label1', 'label2']，最终返回的数据集格式为一条数据`data`对应两个标签`label1`和`label2`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_multilabel_data(num, w=2.0, b=3.0):\n",
    "    for _ in range(num):\n",
    "        x = np.random.uniform(-10.0, 10.0)\n",
    "        noise1 = np.random.normal(0, 1)\n",
    "        noise2 = np.random.normal(-1, 1)\n",
    "        y1 = x * w + b + noise1\n",
    "        y2 = x * w + b + noise2\n",
    "        yield np.array([x]).astype(np.float32), np.array([y1]).astype(np.float32), np.array([y2]).astype(np.float32)\n",
    "\n",
    "def create_multilabel_dataset(num_data, batch_size=16):\n",
    "    dataset = GeneratorDataset(list(get_multilabel_data(num_data)), column_names=['data', 'label1', 'label2'])\n",
    "    dataset = dataset.batch(batch_size)  # 每个batch有16个数据\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 多标签损失函数\n",
    "\n",
    "针对上一步创建的多标签数据集，定义多标签损失函数`MAELossForMultiLabel`。\n",
    "\n",
    "$$ loss1= \\frac{1}{m}\\sum_{i=1}^m\\lvert y1_i-f(x_i) \\rvert$$\n",
    "\n",
    "$$ loss2= \\frac{1}{m}\\sum_{i=1}^m\\lvert y2_i-f(x_i) \\rvert$$\n",
    "\n",
    "$$ loss = \\frac{(loss1 + loss2)}{2}$$\n",
    "\n",
    "上式中，$f(x)$ 为样例标签的预测值，$y1$ 和 $y2$ 为样例标签的真实值，$loss1$ 为预测值与真实值 $y1$ 之间距离的平均值，$loss2$ 为预测值与真实值 $y2$ 之间距离的平均值 ，$loss$ 为损失值 $loss1$ 与损失值 $loss2$ 平均值。\n",
    "\n",
    "在`MAELossForMultiLabel`中的`construct`方法的输入有三个，预测值`base`，真实值`target1`和`target2`，在`construct`中分别计算预测值与真实值`target1`、预测值与真实值`target2`之间的误差，将两误差取平均后作为最终的损失函数值。\n",
    "\n",
    "示例代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MAELossForMultiLabel(nn.LossBase):\n",
    "\n",
    "    def construct(self, base, target1, target2):\n",
    "        x1 = ops.abs(base - target1)\n",
    "        x2 = ops.abs(base - target2)\n",
    "        return (self.get_loss(x1) + self.get_loss(x2)) / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 多标签模型训练\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用Model连接前向网络、多标签损失函数和优化器时，`Model`的网络`network`指定为自定义的损失网络`loss_net`，损失函数`loss_fn`不指定，优化器仍使用`Momentum`。\n",
    "\n",
    "在未指定`loss_fn`时，`Model`会默认`network`内部已实现损失函数的逻辑，不再在内部使用`nn.WithLossCell`关联前向网络和损失函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T03:42:24.079033Z",
     "start_time": "2021-12-29T03:42:23.851418Z"
    }
   },
   "outputs": [],
   "source": [
    "train_dataset = create_multilabel_dataset(num_data=160)\n",
    "\n",
    "# 定义多标签损失函数\n",
    "loss_fn = MAELossForMultiLabel()\n",
    "# 定义优化器\n",
    "opt = nn.Momentum(network.trainable_params(), learning_rate=0.005, momentum=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_fn(data, label1, label2):\n",
    "    output = network(data)\n",
    "    return loss_fn(output, label1, label2)\n",
    "\n",
    "grad_fn = ms.value_and_grad(forward_fn, None, opt.parameters)\n",
    "\n",
    "def train_step(data, label1, label2):\n",
    "    loss, grads = grad_fn(data, label1, label2)\n",
    "    opt(grads)\n",
    "    return loss\n",
    "\n",
    "def train(model, dataset):\n",
    "    size = dataset.get_dataset_size()\n",
    "    model.set_train()\n",
    "    for batch, (data, label1, label2) in enumerate(dataset.create_tuple_iterator()):\n",
    "        loss = train_step(data, label1, label2)\n",
    "\n",
    "        if batch % 2 == 0:\n",
    "            loss, current = loss.asnumpy(), batch\n",
    "            print(f\"loss: {loss:>7f}  [{current:>3d}/{size:>3d}]\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1\n",
      "-------------------------------\n",
      "loss: 0.739832  [  0/ 10]\n",
      "loss: 0.949316  [  2/ 10]\n",
      "loss: 1.052085  [  4/ 10]\n",
      "loss: 0.982260  [  6/ 10]\n",
      "loss: 0.784400  [  8/ 10]\n",
      "Epoch 2\n",
      "-------------------------------\n",
      "loss: 0.963160  [  0/ 10]\n",
      "loss: 0.899232  [  2/ 10]\n",
      "loss: 0.934914  [  4/ 10]\n",
      "loss: 0.757601  [  6/ 10]\n",
      "loss: 0.965961  [  8/ 10]\n",
      "Epoch 3\n",
      "-------------------------------\n",
      "loss: 0.815042  [  0/ 10]\n",
      "loss: 0.999898  [  2/ 10]\n",
      "loss: 1.008266  [  4/ 10]\n",
      "loss: 1.024307  [  6/ 10]\n",
      "loss: 0.798073  [  8/ 10]\n",
      "Epoch 4\n",
      "-------------------------------\n",
      "loss: 0.844747  [  0/ 10]\n",
      "loss: 0.958094  [  2/ 10]\n",
      "loss: 0.898447  [  4/ 10]\n",
      "loss: 0.879910  [  6/ 10]\n",
      "loss: 0.969592  [  8/ 10]\n",
      "Epoch 5\n",
      "-------------------------------\n",
      "loss: 0.917983  [  0/ 10]\n",
      "loss: 0.862990  [  2/ 10]\n",
      "loss: 0.947069  [  4/ 10]\n",
      "loss: 0.854086  [  6/ 10]\n",
      "loss: 0.910622  [  8/ 10]\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "epochs = 5\n",
    "for t in range(epochs):\n",
    "    print(f\"Epoch {t+1}\\n-------------------------------\")\n",
    "    train(network, train_dataset)\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本章节简单讲解了多标签数据集场景下，如何定义损失函数并进行模型训练。在很多其他场景中，也可采用此类方法进行模型训练。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore",
   "language": "python",
   "name": "mindspore"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5 (default, Oct 25 2019, 15:51:11) \n[GCC 7.3.0]"
  },
  "vscode": {
   "interpreter": {
    "hash": "8c9da313289c39257cb28b126d2dadd33153d4da4d524f730c81a4aaccbd2ca7"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}