{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_notebook.svg)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/master/tutorials/zh_cn/advanced/model/mindspore_metric.ipynb) [![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_download_code.svg)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/master/tutorials/zh_cn/advanced/model/mindspore_metric.py) [![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_source.svg)](https://gitee.com/mindspore/docs/blob/master/tutorials/source_zh_cn/advanced/model/metric.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 评价指标 Metrics\n",
    "\n",
    "当训练任务结束，常常需要评价函数（Metrics）来评估模型的好坏。不同的训练任务往往需要不同的Metrics函数。例如，对于二分类问题，常用的评价指标有precision（准确率）、recall（召回率）等，而对于多分类任务，可使用宏平均（Macro）和微平均（Micro）来评估。\n",
    "\n",
    "MindSpore提供了大部分常见任务的评价函数，如`Accuracy`、`Precision`、`MAE`和`MSE`等，由于MindSpore提供的评价函数无法满足所有任务的需求，很多情况下用户需要针对具体的任务自定义Metrics来评估训练的模型。\n",
    "\n",
    "本章主要介绍如何自定义Metrics以及如何在`mindspore.train.Model`中使用Metrics。\n",
    "\n",
    "> 详情可参考[评价指标](https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.train.html#评价指标)。\n",
    "\n",
    "## 自定义Metrics\n",
    "\n",
    "自定义Metrics函数需要继承`mindspore.train.Metric`父类，并重新实现父类中的`clear`方法、`update`方法和`eval`方法。\n",
    "\n",
    "- `clear`：初始化相关的内部参数。\n",
    "- `update`：接收网络预测输出和标签，计算误差，每次step后并更新内部评估结果。\n",
    "- `eval`：计算最终评估结果，在每次epoch结束后计算最终的评估结果。\n",
    "\n",
    "平均绝对误差（MAE）算法如式(1)所示：\n",
    "\n",
    "$$ MAE=\\frac{1}{n}\\sum_{i=1}^n\\lvert ypred_i - y_i \\rvert \\tag{1}$$\n",
    "\n",
    "下面以简单的MAE算法为例，介绍`clear`、`update`和`eval`三个函数及其使用方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-04T11:04:40.488396Z",
     "start_time": "2022-01-04T11:04:38.090036Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1499999612569809\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import mindspore as ms\n",
    "\n",
    "class MyMAE(ms.train.Metric):\n",
    "    def __init__(self):\n",
    "        super(MyMAE, self).__init__()\n",
    "        self.clear()\n",
    "\n",
    "    def clear(self):\n",
    "        \"\"\"初始化变量_abs_error_sum和_samples_num\"\"\"\n",
    "        self._abs_error_sum = 0  # 保存误差和\n",
    "        self._samples_num = 0    # 累计数据量\n",
    "\n",
    "    def update(self, *inputs):\n",
    "        \"\"\"更新_abs_error_sum和_samples_num\"\"\"\n",
    "        y_pred = inputs[0].asnumpy()\n",
    "        y = inputs[1].asnumpy()\n",
    "\n",
    "        # 计算预测值与真实值的绝对误差\n",
    "        abs_error_sum = np.abs(y - y_pred)\n",
    "        self._abs_error_sum += abs_error_sum.sum()\n",
    "\n",
    "        # 样本的总数\n",
    "        self._samples_num += y.shape[0]\n",
    "\n",
    "    def eval(self):\n",
    "        \"\"\"计算最终评估结果\"\"\"\n",
    "        return self._abs_error_sum / self._samples_num\n",
    "\n",
    "# 网络有两个输出\n",
    "y_pred = ms.Tensor(np.array([[0.1, 0.2, 0.6, 0.9], [0.1, 0.2, 0.6, 0.9]]), ms.float32)\n",
    "y = ms.Tensor(np.array([[0.1, 0.25, 0.7, 0.9], [0.1, 0.25, 0.7, 0.9]]), ms.float32)\n",
    "\n",
    "error = MyMAE()\n",
    "error.clear()\n",
    "error.update(y_pred, y)\n",
    "result = error.eval()\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型训练中使用Metrics\n",
    "\n",
    "[mindspore.train.Model](https://www.mindspore.cn/docs/zh-CN/master/api_python/train/mindspore.train.Model.html#mindspore.train.Model)是用于训练和评估的高层API，可以将自定义或MindSpore已有的Metrics作为参数传入，Model能够自动调用传入的Metrics进行评估。\n",
    "\n",
    "在网络模型训练后，需要使用评价指标，来评估网络模型的训练效果，因此在演示具体代码之前首先简单拟定数据集，对数据集进行加载和定义一个简单的线性回归网络模型：\n",
    "\n",
    "$$f(x)=w*x+b \\tag{2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from mindspore import dataset as ds\n",
    "\n",
    "def get_data(num, w=2.0, b=3.0):\n",
    "    \"\"\"生成数据及对应标签\"\"\"\n",
    "    for _ in range(num):\n",
    "        x = np.random.uniform(-10.0, 10.0)\n",
    "        noise = np.random.normal(0, 1)\n",
    "        y = x * w + b + noise\n",
    "        yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32)\n",
    "\n",
    "def create_dataset(num_data, batch_size=16):\n",
    "    \"\"\"加载数据集\"\"\"\n",
    "    dataset = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label'])\n",
    "    dataset = dataset.batch(batch_size)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用内置评价指标\n",
    "\n",
    "使用MindSpore内置的Metrics作为参数传入Model时，Metrics可以定义为一个字典类型，字典的key值为字符串类型，字典的value值为MindSpore内置的[评价指标](https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.train.html#评价指标)，如下示例使用`train.Accuracy`计算分类的准确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1 step: 10, loss is 5.908090114593506\n",
      "Eval result: epoch 1, metrics: {'MAE': 5.1329233884811405}\n",
      "epoch: 2 step: 10, loss is 3.9280264377593994\n",
      "Eval result: epoch 2, metrics: {'MAE': 3.0886757612228393}\n",
      "epoch: 3 step: 10, loss is 2.9104671478271484\n",
      "Eval result: epoch 3, metrics: {'MAE': 2.461756193637848}\n",
      "epoch: 4 step: 10, loss is 1.8725224733352661\n",
      "Eval result: epoch 4, metrics: {'MAE': 2.11311993598938}\n",
      "epoch: 5 step: 10, loss is 2.1637942790985107\n",
      "Eval result: epoch 5, metrics: {'MAE': 1.6749439239501953}\n",
      "epoch: 6 step: 10, loss is 1.3848766088485718\n",
      "Eval result: epoch 6, metrics: {'MAE': 1.317658966779709}\n",
      "epoch: 7 step: 10, loss is 1.052016258239746\n",
      "Eval result: epoch 7, metrics: {'MAE': 1.043285644054413}\n",
      "epoch: 8 step: 10, loss is 1.1781564950942993\n",
      "Eval result: epoch 8, metrics: {'MAE': 0.8706761479377747}\n",
      "epoch: 9 step: 10, loss is 0.8200418949127197\n",
      "Eval result: epoch 9, metrics: {'MAE': 0.7817940771579742}\n",
      "epoch: 10 step: 10, loss is 0.7065591812133789\n",
      "Eval result: epoch 10, metrics: {'MAE': 0.7885207533836365}\n"
     ]
    }
   ],
   "source": [
    "import mindspore.nn as nn\n",
    "from mindspore.train import Model, MAE, LossMonitor\n",
    "\n",
    "net = nn.Dense(1, 1)\n",
    "loss_fn = nn.L1Loss()\n",
    "optimizer = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9)\n",
    "\n",
    "# 定义model，使用内置的Accuracy函数\n",
    "model = Model(net, loss_fn, optimizer, metrics={\"MAE\": MAE()})\n",
    "\n",
    "train_dataset = create_dataset(num_data=160)\n",
    "eval_dataset = create_dataset(num_data=160)\n",
    "train_dataset_size = train_dataset.get_dataset_size()\n",
    "\n",
    "model.fit(10, train_dataset, eval_dataset, callbacks=LossMonitor(train_dataset_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用自定义评价指标\n",
    "\n",
    "如下示例在`Model`中传入上述自定义的评估指标`MAE()`，将验证数据集传入`model.fit()`接口边训练边验证。\n",
    "\n",
    "验证结果为一个字典类型，验证结果的key值与`metrics`的key值相同，验证结果的value值为预测值与实际值的平均绝对误差。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1 step: 10, loss is 0.7992362380027771\n",
      "Eval result: epoch 1, metrics: {'MAE': 0.8640150725841522}\n",
      "epoch: 2 step: 10, loss is 0.8377518653869629\n",
      "Eval result: epoch 2, metrics: {'MAE': 0.9286439001560212}\n",
      "epoch: 3 step: 10, loss is 0.894376277923584\n",
      "Eval result: epoch 3, metrics: {'MAE': 0.8669328391551971}\n",
      "epoch: 4 step: 10, loss is 0.8098692893981934\n",
      "Eval result: epoch 4, metrics: {'MAE': 0.9018074989318847}\n",
      "epoch: 5 step: 10, loss is 0.8556416630744934\n",
      "Eval result: epoch 5, metrics: {'MAE': 0.8721640467643738}\n",
      "epoch: 6 step: 10, loss is 0.8508825302124023\n",
      "Eval result: epoch 6, metrics: {'MAE': 0.8601282179355622}\n",
      "epoch: 7 step: 10, loss is 0.7443522810935974\n",
      "Eval result: epoch 7, metrics: {'MAE': 0.9004024684429168}\n",
      "epoch: 8 step: 10, loss is 0.7394096851348877\n",
      "Eval result: epoch 8, metrics: {'MAE': 0.9380556881427765}\n",
      "epoch: 9 step: 10, loss is 0.7989674210548401\n",
      "Eval result: epoch 9, metrics: {'MAE': 0.8629323005676269}\n",
      "epoch: 10 step: 10, loss is 0.6581473350524902\n",
      "Eval result: epoch 10, metrics: {'MAE': 0.9144346475601196}\n"
     ]
    }
   ],
   "source": [
    "train_dataset = create_dataset(num_data=160)\n",
    "eval_dataset = create_dataset(num_data=160)\n",
    "\n",
    "model = Model(net, loss_fn, optimizer, metrics={\"MAE\": MyMAE()})\n",
    "\n",
    "# 定义model，将自定义metrics函数MAE传入Model中\n",
    "model.fit(10, train_dataset, eval_dataset, callbacks=LossMonitor(train_dataset_size))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore",
   "language": "python",
   "name": "mindspore"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5 (default, Oct 25 2019, 15:51:11) \n[GCC 7.3.0]"
  },
  "vscode": {
   "interpreter": {
    "hash": "8c9da313289c39257cb28b126d2dadd33153d4da4d524f730c81a4aaccbd2ca7"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}