{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# 动态图与静态图\n",
    "\n",
    "`Ascend` `GPU` `CPU` `进阶` `模型运行`\n",
    "\n",
    "[![在线运行](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9tb2RlbGFydHMvcXVpY2tfc3RhcnQvbWluZHNwb3JlX3B5bmF0aXZlX21vZGVfYW5kX2dyYXBoX21vZGUuaXB5bmI=&imageid=59a6e9f5-93c0-44dd-85b0-82f390c5d53b) [![下载Notebook](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_notebook.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.6/tutorials/zh_cn/intermediate/mindspore_pynative_mode_and_graph_mode.ipynb) [![下载样例代码](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_download_code.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r1.6/tutorials/zh_cn/intermediate/mindspore_pynative_mode_and_graph_mode.py)\n [![查看源文件](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.6/tutorials/source_zh_cn/intermediate/pynative_mode_and_graph_mode.ipynb)\n",
    "\n",
    "## 概述\n",
    "\n",
    "MindSpore支持两种运行模式：\n",
    "\n",
    "- 静态图模式：又称Graph模式，将神经网络模型编译成一整张图，然后下发执行。该模式利用图优化、计算图整图下沉等技术提高运行性能，同时有助于规模部署和跨平台运行。\n",
    "- 动态图模式：又称PyNative模式，将神经网络中的各个算子逐一下发执行，方便用户编写和调试神经网络模型。\n",
    "\n",
    "Graph和PyNative两种模式的区别主要有：\n",
    "\n",
    "- 使用场景：Graph模式需要一开始就构建好网络结构，然后框架做整图优化和执行，比较适合网络固定没有变化，且需要高性能的场景。而PyNative模式逐行执行算子，支持单独求梯度。\n",
    "\n",
    "- 网络执行：Graph模式和PyNative模式在执行相同的网络和算子时，精度效果是一致的。由于Graph模式运用了图优化、计算图整图下沉等技术，Graph模式执行网络的性能和效率更高。\n",
    "\n",
    "- 代码调试：在脚本开发和网络流程调试中，推荐使用PyNative模式进行调试。在PyNative模式下，可以方便地设置断点，获取网络执行的中间结果，也可以通过pdb的方式对网络进行调试。而Graph模式无法设置断点，只能先指定算子进行打印，然后在网络执行完成后查看输出结果。\n",
    "\n",
    "默认情况下，MindSpore处于Graph模式，可以通过`context.set_context(mode=context.PYNATIVE_MODE)`切换为PyNative模式；同样地，MindSpore处于PyNative模式时，可以通过 `context.set_context(mode=context.GRAPH_MODE)`切换为Graph模式。\n",
    "\n",
    "下面以Graph模式为例，演示MindSpore单算子、普通函数、模型的执行方式，并进一步说明如何在Graph模式和PyNative模式下进行性能优化及梯度求取。\n",
    "\n",
    "## 执行方式\n",
    "\n",
    "这里演示在Graph模式和PyNative模式下，单算子、普通函数、模型的执行方式。\n",
    "\n",
    "> 在本案例的实际执行中，采取了MindSpore的默认方式`GRAPH_MODE`，用户也可以将其变更为`PYNATIVE_MODE`进行尝试。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import numpy as np\n",
    "import mindspore.nn as nn\n",
    "from mindspore import context, Tensor, ParameterTuple, ms_function\n",
    "import mindspore.ops as ops\n",
    "from mindspore.common.initializer import Normal\n",
    "from mindspore.nn import WithLossCell, Momentum\n",
    "\n",
    "# 设定为Graph模式，也可替换为PYNATIVE_MODE\n",
    "context.set_context(mode=context.GRAPH_MODE, device_target=\"Ascend\")"
   ],
   "outputs": [],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 执行单算子"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "# 打印Conv2d算子的输出\n",
    "conv = nn.Conv2d(3, 4, 3, bias_init='zeros')\n",
    "input_data = Tensor(np.ones([1, 3, 5, 5]).astype(np.float32))\n",
    "output = conv(input_data)\n",
    "print(output.asnumpy())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[[[[-0.02593261  0.01615404  0.01615404  0.01615404  0.01196378]\n",
      "   [-0.01535788  0.05602208  0.05602208  0.05602208  0.04094065]\n",
      "   [-0.01535788  0.05602208  0.05602208  0.05602208  0.04094065]\n",
      "   [-0.01535788  0.05602208  0.05602208  0.05602208  0.04094065]\n",
      "   [-0.01409336  0.04544117  0.04544117  0.04544117  0.0373004 ]]\n",
      "\n",
      "  [[ 0.03874376  0.02201786  0.02201786  0.02201786  0.02687691]\n",
      "   [ 0.05751193  0.02690699  0.02690699  0.02690699  0.03515062]\n",
      "   [ 0.05751193  0.02690699  0.02690699  0.02690699  0.03515062]\n",
      "   [ 0.05751193  0.02690699  0.02690699  0.02690699  0.03515062]\n",
      "   [ 0.02599058  0.01130002  0.01130002  0.01130002  0.02304572]]\n",
      "\n",
      "  [[-0.00022919  0.02640852  0.02640852  0.02640852  0.04932421]\n",
      "   [ 0.01657246  0.0705748   0.0705748   0.0705748   0.0874946 ]\n",
      "   [ 0.01657246  0.0705748   0.0705748   0.0705748   0.0874946 ]\n",
      "   [ 0.01657246  0.0705748   0.0705748   0.0705748   0.0874946 ]\n",
      "   [ 0.03821789  0.09614976  0.09614976  0.09614976  0.10491695]]\n",
      "\n",
      "  [[ 0.0190958   0.02602289  0.02602289  0.02602289  0.01660084]\n",
      "   [ 0.03556763  0.06862713  0.06862713  0.06862713  0.02653556]\n",
      "   [ 0.03556763  0.06862713  0.06862713  0.06862713  0.02653556]\n",
      "   [ 0.03556763  0.06862713  0.06862713  0.06862713  0.02653556]\n",
      "   [ 0.00727296  0.04514674  0.04514674  0.04514674  0.01423099]]]]\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 执行普通函数\n",
    "\n",
    "将若干算子组合成一个函数，然后直接通过函数调用的方式执行这些算子，并打印相关结果，如下例所示。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "def add_func(x, y):\n",
    "    z = ops.add(x, y)\n",
    "    z = ops.add(z, x)\n",
    "    return z\n",
    "\n",
    "x = Tensor(np.ones([3, 3], dtype=np.float32))\n",
    "y = Tensor(np.ones([3, 3], dtype=np.float32))\n",
    "output = add_func(x, y)\n",
    "print(output.asnumpy())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[[3. 3. 3.]\n",
      " [3. 3. 3.]\n",
      " [3. 3. 3.]]\n"
     ]
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 执行网络"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "import numpy as np\n",
    "import mindspore.nn as nn\n",
    "import mindspore.ops as ops\n",
    "from mindspore import context, Tensor\n",
    "\n",
    "class Net(nn.Cell):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.mul = ops.Mul()\n",
    "\n",
    "    def construct(self, x, y):\n",
    "        return self.mul(x, y)\n",
    "\n",
    "x = Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
    "y = Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n",
    "\n",
    "net = Net()\n",
    "print(net(x, y))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[ 4. 10. 18.]\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Graph模式说明\n",
    "\n",
    "### 性能优化\n",
    "\n",
    "在Graph模式下，MindSpore通过源码转换的方式，将Python的源码转换成IR，再在此基础上进行相关的图优化，最终在硬件设备上执行优化后的图。MindSpore使用的是一种基于图表示的函数式IR，即MindIR，采用了接近于ANF函数式的语义。Graph模式是基于MindIR进行编译优化的，编译器可以利用图优化、计算图整图下沉等技术对执行图进行更大程度的优化，从而获得更好的执行性能。"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "使用Graph模式时，需要使用`nn.Cell`类并且在`construct`函数中编写执行代码， 或者调用`@ms_function`装饰器。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 梯度求取\n",
    "\n",
    "Graph模式中，定义了[GradOperation](https://www.mindspore.cn/docs/api/zh-CN/r1.6/api_python/ops/mindspore.ops.GradOperation.html#mindspore.ops.GradOperation)计算神经网络的梯度，使用反向自动微分模式，即从正向网络的输出开始计算梯度。关于自动微分的详细信息，请参考[自动微分](https://www.mindspore.cn/tutorials/zh-CN/r1.6/autograd.html)。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## PyNative模式说明\n",
    "\n",
    "### 性能优化\n",
    "\n",
    "正如文章开头所说，Graph模式适合高性能的场景，但PyNative模式中也提供了性能优化的手段。MindSpore提供了Staging功能，该功能可以在PyNative模式下将Python函数或者Python类的方法编译成计算图，通过图优化等技术提高运行速度，是一种混合运行机制。Staging功能的使用通过`ms_function`装饰器达成，该装饰器会将模块编译成计算图，在给定输入之后，以图的形式下发执行。如下例所示："
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "# 导入ms_function\n",
    "from mindspore import ms_function\n",
    "\n",
    "# 仍设定为PyNative模式\n",
    "context.set_context(mode=context.PYNATIVE_MODE, device_target=\"Ascend\")\n",
    "\n",
    "add = ops.Add()\n",
    "\n",
    "# 使用装饰器编译计算图\n",
    "@ms_function\n",
    "def add_fn(x, y):\n",
    "    res = add(x, y)\n",
    "    return res\n",
    "\n",
    "x = Tensor(np.ones([4, 4]).astype(np.float32))\n",
    "y = Tensor(np.ones([4, 4]).astype(np.float32))\n",
    "z = add_fn(x, y)\n",
    "print(z.asnumpy())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[[2. 2. 2. 2.]\n",
      " [2. 2. 2. 2.]\n",
      " [2. 2. 2. 2.]\n",
      " [2. 2. 2. 2.]]\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "在加装了`ms_function`装饰器的函数中，如果包含不需要进行参数训练的算子（如`pooling`、`add`等算子），则这些算子可以在被装饰的函数中直接调用，如上例所示。如果被装饰的函数中包含了需要进行参数训练的算子（如`Convolution`、`BatchNorm`等算子），则这些算子必须在被装饰的函数之外完成实例化操作。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "# Conv2d实例化操作\n",
    "conv_obj = nn.Conv2d(in_channels=3, out_channels=4, kernel_size=3, stride=2, padding=0)\n",
    "conv_obj.init_parameters_data()\n",
    "\n",
    "@ms_function\n",
    "def conv_fn(x):\n",
    "    res = conv_obj(x)\n",
    "    return res\n",
    "\n",
    "input_data = np.random.randn(2, 3, 6, 6).astype(np.float32)\n",
    "z = conv_fn(Tensor(input_data))\n",
    "print(z.asnumpy())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[[[[ 0.00994964  0.01850731 -0.05146599]\n",
      "   [ 0.02427048 -0.09082688 -0.00945184]\n",
      "   [ 0.02710651 -0.07322617  0.02594434]]\n",
      "\n",
      "  [[ 0.00056772 -0.05043615 -0.03873939]\n",
      "   [-0.00445028  0.03694705 -0.03555503]\n",
      "   [ 0.07329068 -0.02026664  0.01922888]]\n",
      "\n",
      "  [[ 0.02257145 -0.04093865 -0.00493869]\n",
      "   [ 0.01740007  0.02478302  0.02072578]\n",
      "   [ 0.05831327 -0.03933404  0.01767443]]\n",
      "\n",
      "  [[-0.03954437  0.02160874 -0.00700614]\n",
      "   [ 0.03856367 -0.04015685  0.02508826]\n",
      "   [-0.0229507  -0.03803677  0.02813173]]]\n",
      "\n",
      "\n",
      " [[[ 0.01678797 -0.02227589 -0.04470547]\n",
      "   [-0.05720481 -0.15464461  0.00911596]\n",
      "   [ 0.02566019 -0.04340314  0.03164666]]\n",
      "\n",
      "  [[ 0.03300299 -0.05849815  0.05841954]\n",
      "   [-0.11595733 -0.01524522  0.02947116]\n",
      "   [ 0.05930116  0.00831041 -0.0466827 ]]\n",
      "\n",
      "  [[-0.0797728   0.02910854  0.00766015]\n",
      "   [-0.01380327 -0.03338642  0.02625138]\n",
      "   [ 0.02279372 -0.00952736  0.02026749]]\n",
      "\n",
      "  [[ 0.04039776 -0.05340278 -0.0083563 ]\n",
      "   [ 0.04991922 -0.05205034 -0.0058607 ]\n",
      "   [ 0.00686666  0.00064385  0.00301326]]]]\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 梯度求取\n",
    "\n",
    "PyNative模式中支持单独的梯度求取操作，下面演示如何利用这一特性调试网络模型。具体操作可通过`GradOperation`求该函数或者网络所有的输入梯度。需要注意，输入类型仅支持Tensor。\n",
    "\n",
    "构建网络如下。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "class LeNet5(nn.Cell):\n",
    "    \"\"\"\n",
    "    Lenet网络结构\n",
    "    \"\"\"\n",
    "    def __init__(self, num_class=10, num_channel=1):\n",
    "        super(LeNet5, self).__init__()\n",
    "        # 定义所需要的运算\n",
    "        self.conv1 = nn.Conv2d(num_channel, 6, 5, pad_mode='valid')\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5, pad_mode='valid')\n",
    "        self.fc1 = nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02))\n",
    "        self.fc2 = nn.Dense(120, 84, weight_init=Normal(0.02))\n",
    "        self.fc3 = nn.Dense(84, num_class, weight_init=Normal(0.02))\n",
    "        self.relu = nn.ReLU()\n",
    "        self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "        self.flatten = nn.Flatten()\n",
    "\n",
    "    def construct(self, x):\n",
    "        # 使用定义好的运算构建前向网络\n",
    "        x = self.conv1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.max_pool2d(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.max_pool2d(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "\n",
    "# 实例化网络\n",
    "net = LeNet5()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "如上文所说，利用`GradOperation`求函数输入的梯度。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "class GradWrap(nn.Cell):\n",
    "    \"\"\"求函数输入梯度\"\"\"\n",
    "    def __init__(self, network):\n",
    "        super(GradWrap, self).__init__(auto_prefix=False)\n",
    "        self.network = network\n",
    "        # 用Tuple的形式包装weight\n",
    "        self.weights = ParameterTuple(filter(lambda x: x.requires_grad, network.get_parameters()))\n",
    "\n",
    "    def construct(self, x, label):\n",
    "        weights = self.weights\n",
    "        # 返回值为梯度\n",
    "        return ops.GradOperation(get_by_list=True)(self.network, weights)(x, label)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "在PyNative模式中进行网络训练。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "# 设定优化器、损失函数\n",
    "optimizer = Momentum(filter(lambda x: x.requires_grad, net.get_parameters()), 0.1, 0.9)\n",
    "criterion = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')\n",
    "\n",
    "# 通过WithLossCell获取Loss值\n",
    "net_with_criterion = WithLossCell(net, criterion)\n",
    "\n",
    "# 调用GradWrap\n",
    "train_network = GradWrap(net_with_criterion)\n",
    "train_network.set_train()\n",
    "\n",
    "# 产生输入数据\n",
    "input_data = Tensor(np.ones([32, 1, 32, 32]).astype(np.float32) * 0.01)\n",
    "label = Tensor(np.ones([32]).astype(np.int32))\n",
    "output = net(Tensor(input_data))\n",
    "\n",
    "# 利用前向网络计算loss\n",
    "loss_output = criterion(output, label)\n",
    "\n",
    "# 求得梯度\n",
    "grads = train_network(input_data, label)\n",
    "\n",
    "# 输出loss\n",
    "success = optimizer(grads)\n",
    "loss = loss_output.asnumpy()\n",
    "print(loss)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "2.3025854\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 动静统一\n",
    "\n",
    "MindSpore支持动态图和静态图两种模式，动态图通过解释执行，具有动态语法亲和性，表达灵活；静态图使用JIT编译优化执行，偏静态语法，在语法上有较多限制。由于动态图和静态图的编译流程不一致，两者的语法约束是不一致的。"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 动态图和静态图互相转换\n",
    "\n",
    "在MindSpore中，我们可以通过控制模式输入参数来切换执行使用动态图还是静态图，通过`context.set_context(mode=context.GRAPH_MODE)`可以设置静态图模式，通过`context.set_context(mode=context.PYNATIVE_MODE)`可以设置成动态图模式。由于在静态图下，对于Python语法有所限制，因此从动态图切换成静态图时，需要符合静态图的语法限制，才能正确使用静态图来进行执行。MindSpore静态图的语法限制可以参考[静态图语法限制](https://www.mindspore.cn/docs/note/zh-CN/r1.6/static_graph_syntax_support.html)。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 动静结合\n",
    "\n",
    "MindSpore支持在动态图下使用静态编译的方式来进行混合执行，通过使用ms_function修饰需要用静态图来执行的函数对象，即可实现动态图和静态图的混合执行，更多ms_function的使用可参考[ms_function文档](https://www.mindspore.cn/docs/programming_guide/zh-CN/r1.6/ms_function.html)。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### JIT Fallback\n",
    "\n",
    "JIT Fallback是从静态图的角度出发考虑静态图和动态图的统一，希望静态图模式能够尽量多地支持动态图模式的语法，其借鉴了传统JIT编译的Fallback的思路。MindSpore默认使用静态图模式即Graph模式，不是所有的Python语法都能支持，用户在编写程序时容易遇到语法约束限制。通过JIT Fallback，用户可以灵活地进行静态图和动态图的切换。\n",
    "\n",
    "当前JIT Fallback有条件地支持Graph模式的部分常量场景，包括调用第三方库、创建及使用Tensor、调用Python的print打印等。\n",
    "\n",
    "代码用例如下："
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "import numpy as np\n",
    "import mindspore.nn as nn\n",
    "from mindspore import context, Tensor\n",
    "\n",
    "context.set_context(mode=context.GRAPH_MODE, device_target=\"GPU\")\n",
    "\n",
    "class Net(nn.Cell):\n",
    "    def construct(self):\n",
    "        x = np.array([1, 2, 3])\n",
    "        y = Tensor(x)\n",
    "        return y\n",
    "\n",
    "net = Net()\n",
    "print(net())\n"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[1 2 3]\n"
     ]
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore",
   "language": "python",
   "name": "mindspore"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}