{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 流程控制语句\n", "\n", "[![在线运行](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9vYnMuZHVhbHN0YWNrLmNuLW5vcnRoLTQubXlodWF3ZWljbG91ZC5jb20vbWluZHNwb3JlLXdlYnNpdGUvbm90ZWJvb2svcjIuMC4wLWFscGhhL3R1dG9yaWFscy9leHBlcnRzL3poX2NuL25ldHdvcmsvbWluZHNwb3JlX2NvbnRyb2xfZmxvdy5pcHluYg==&imageid=77ef960a-bd26-4de4-9695-5b85a786fb90) [![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/resource/_static/logo_notebook.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r2.0.0-alpha/tutorials/experts/zh_cn/network/mindspore_control_flow.ipynb) \n", "[![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/resource/_static/logo_download_code.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r2.0.0-alpha/tutorials/experts/zh_cn/network/mindspore_control_flow.py) \n", "[![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r2.0.0-alpha/tutorials/experts/source_zh_cn/network/control_flow.ipynb)\n", "\n", "目前主流的深度学习框架的执行模式有两种，分别为静态图模式`GRAPH_MODE`和动态图`PYNATIVE_MODE`模式。\n", "\n", "在`PYNATIVE_MODE`模式下，MindSpore完全支持Python原生语法的流程控制语句。`GRAPH_MODE`模式下，MindSpore在编译时做了性能优化，因此，在定义网络时使用流程控制语句时会有部分特殊约束，其他部分仍和Python原生语法保持一致。\n", "\n", "运行模式从动态图切换到静态图时，请留意[静态图语法支持](https://www.mindspore.cn/docs/zh-CN/r2.0.0-alpha/note/static_graph_syntax_support.html#静态图语法支持)。下面我们详细介绍在`GRAPH_MODE`模式下定义网络时流程控制语句的使用方式。\n", "\n", "## 常量与变量条件\n", "\n", "在`GRAPH_MODE`模式下定义网络，MindSpore将流程控制语句中的条件表达式分为两类：即常量条件和变量条件。在图编译时可以确定结果为True或False的条件表达式为常量条件，在图编译时不能确定结果为True或False的条件表达式为变量条件。**只有当条件表达式为变量条件时，MindSpore才会在网络中生成控制流算子**。\n", "\n", "需要注意的是，当网络中存在控制流算子时，网络会被切分成多个执行子图，子图间进行流程跳转和数据传递会产生一定的性能损耗。\n", "\n", "### 常量条件\n", "\n", "判断方式：\n", "\n", "- 条件表达式中不存在Tensor类型，且也不存在元素为Tensor类型的List、Tuple、Dict。\n", "- 条件表达式中存在Tensor类型，或者元素为Tensor类型的List、Tuple、Dict，但是表达式结果不受Tensor的值影响。\n", "\n", "举例：\n", "\n", "- `for i in range(0,10)`，`i`为标量：潜在的条件表达式`i < 10`在图编译时可以确定结果，因此为常量条件；\n", "\n", "- `self.flag`，`self.flag`为标量：此处`self.flag`为一个bool类型标量，其值在构建Cell对象时已确定；\n", "\n", "- `x + 1 < 10`，`x`为标量：此处`x + 1`的值在构建Cell对象时是不确定的，但是在图编译时MindSpore会计算所有标量表达式的结果，因此该表达式的值也是在编译期确定的。\n", "\n", "- `len(my_list) < 10`，`my_list`为元素是Tensor类型的List对象：该条件表达式包含Tensor，但是表达式结果不受Tensor的值影响，只与`my_list`中Tensor的数量有关；\n", "\n", "### 变量条件\n", "\n", "判断方式：\n", "\n", "- 条件表达式中存在Tensor类型或者元素为Tensor类型的List、Tuple、Dict，并且条件表达式的结果受Tensor的值影响。\n", "\n", "举例：\n", "\n", "- `x < y`，`x`和`y`为算子输出。\n", "\n", "- `x in list`，`x`为算子输出。\n", "\n", "由于算子输出是图在各个step执行时才能确定，因此上面两个都属于变量条件。\n", "\n", "## if语句\n", "\n", "在`GRAPH_MODE`模式下定义网络时，使用`if`语句需要注意：**在条件表达式为变量条件时，在不同分支的同一变量应被赋予相同的数据类型，例如Tensor类型变量要求shape和type一致。**Tensor变量的shape一致性约束详见[ShapeJoin规则](#shapejoin规则)。\n", "\n", "### 变量条件的if语句\n", "\n", "在下面代码中，在`if`和`else`分支中，变量`out`在`if`语句不同分支被赋予的Tensor的Shape分别是()和(2,)。网络最终返回的Tensor的shape由条件`x < y`决定，而在图编译时期无法确定`x < y`的结果，因此图编译时期无法确定`out`的Shape是()还是(2,)，MindSpore最终因类型推导失败而抛出异常。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "import numpy as np\n", "import mindspore as ms\n", "from mindspore import nn\n", "\n", "class SingleIfNet(nn.Cell):\n", "\n", " def construct(self, x, y, z):\n", " # 构造条件表达式为变量条件的if语句\n", " if x < y:\n", " out = x\n", " else:\n", " out = z\n", " out = out + 1\n", " return out\n", "\n", "forward_net = SingleIfNet()\n", "\n", "x = ms.Tensor(np.array(0), dtype=ms.int32)\n", "y = ms.Tensor(np.array(1), dtype=ms.int32)\n", "z = ms.Tensor(np.array([1, 2]), dtype=ms.int32)\n", "\n", "output = forward_net(x, y, z)\n", "```\n", "\n", "执行上面的代码，报错信息如下：\n", "\n", "```text\n", "ValueError: mindspore/ccsrc/pipeline/jit/static_analysis/static_analysis.cc:800 ProcessEvalResults] Cannot join the return values of different branches, perhaps you need to make them equal.\n", "Shape Join Failed: shape1 = (), shape2 = (2).\n", "```\n", "\n", "### 常量条件的if语句\n", "\n", "当`if`语句中的条件表达式为常量条件时，其使用方式与Python原生语法保持一致，并无额外的约束。如下代码中的`if`语句条件表达式`x < y + 1`为常量条件(因为x和y都是标量常量类型)，在图编译时期可确定变量`out`的类型为标量`int`类型，网络可正常编译和执行，输出正确结果`1`。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2022-01-07T03:35:40.804471Z", "start_time": "2022-01-07T03:35:39.226569Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "output: 1\n" ] } ], "source": [ "import numpy as np\n", "import mindspore as ms\n", "from mindspore import nn\n", "\n", "class SingleIfNet(nn.Cell):\n", "\n", " def construct(self, z):\n", " x = 0\n", " y = 1\n", "\n", " # 构造条件表达式为常量条件的if语句\n", " if x < y + 1:\n", " out = x\n", " else:\n", " out = z\n", " out = out + 1\n", "\n", " return out\n", "\n", "z = ms.Tensor(np.array([0, 1]), dtype=ms.int32)\n", "forward_net = SingleIfNet()\n", "\n", "output = forward_net(z)\n", "print(\"output:\", output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## for语句\n", "\n", "`for`语句会将循环体展开，因此使用`for`语句的网络的子图数量、算子数量取决于`for`语句的循环次数，算子数量过多或者子图过多会消耗更多的硬件资源。\n", "\n", "下面的示例代码中，`for`语句中的循环体会被执行3次，输出结果为`5`。" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2022-01-07T03:35:40.922574Z", "start_time": "2022-01-07T03:35:40.806485Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "output: 5\n" ] } ], "source": [ "import numpy as np\n", "from mindspore import nn\n", "import mindspore as ms\n", "\n", "class IfInForNet(nn.Cell):\n", "\n", " def construct(self, x, y):\n", " out = 0\n", "\n", " # 构造条件表达式为常量条件的for语句\n", " for i in range(0, 3):\n", " # 构造条件表达式为变量条件的if语句\n", " if x + i < y:\n", " out = out + x\n", " else:\n", " out = out + y\n", " out = out + 1\n", "\n", " return out\n", "\n", "forward_net = IfInForNet()\n", "\n", "x = ms.Tensor(np.array(0), dtype=ms.int32)\n", "y = ms.Tensor(np.array(1), dtype=ms.int32)\n", "\n", "output = forward_net(x, y)\n", "print(\"output:\", output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "由于`for`语句会展开循环体内容，所以上面的代码和下面的代码等价：" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2022-01-07T03:35:40.985953Z", "start_time": "2022-01-07T03:35:40.923594Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "output: 5\n" ] } ], "source": [ "import numpy as np\n", "from mindspore import nn\n", "import mindspore as ms\n", "\n", "class IfInForNet(nn.Cell):\n", " def construct(self, x, y):\n", " out = 0\n", "\n", " # 循环： 0\n", " if x + 0 < y:\n", " out = out + x\n", " else:\n", " out = out + y\n", " out = out + 1\n", " # 循环： 1\n", " if x + 1 < y:\n", " out = out + x\n", " else:\n", " out = out + y\n", " out = out + 1\n", " # 循环： 2\n", " if x + 2 < y:\n", " out = out + x\n", " else:\n", " out = out + y\n", " out = out + 1\n", "\n", " return out\n", "\n", "forward_net = IfInForNet()\n", "\n", "x = ms.Tensor(np.array(0), dtype=ms.int32)\n", "y = ms.Tensor(np.array(1), dtype=ms.int32)\n", "\n", "output = forward_net(x, y)\n", "print(\"output:\", output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "从上面两段示例代码我们可以看出，在部分场景下，使用`for`语句会导致出现子图过多的问题时。为了节约硬件资源开销，提升网络编译性能，可尝试将`for`语句等价转换为条件表达式是变量条件的`while`语句。\n", "\n", "## while语句\n", "\n", "`while`语句相比`for`语句更为灵活。当`while`的条件为常量时，`while`对循环体的处理和`for`类似，会展开循环体内容。\n", "\n", "当`while`的条件表达式是变量条件时，`while`语句则不会展开循环体内容，而是在执行图中产生控制流算子，因此可以避免`for`循环带来的子图过多的问题。\n", "\n", "### 常量条件的while语句\n", "\n", "下面的示例代码中，`for`语句中的循环体会被执行3次，输出结果为`5`，和上面介绍`for`语句中的示例代码本质上是一样的。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2022-01-07T03:35:41.051756Z", "start_time": "2022-01-07T03:35:40.988065Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "output: 5\n" ] } ], "source": [ "import numpy as np\n", "from mindspore import nn\n", "import mindspore as ms\n", "\n", "class IfInWhileNet(nn.Cell):\n", "\n", " def construct(self, x, y):\n", " i = 0\n", " out = x\n", " # 构造条件表达式为常量条件的while语句\n", " while i < 3:\n", " # 构造条件表达式为变量条件的if语句\n", " if x + i < y:\n", " out = out + x\n", " else:\n", " out = out + y\n", " out = out + 1\n", " i = i + 1\n", " return out\n", "\n", "forward_net = IfInWhileNet()\n", "x = ms.Tensor(np.array(0), dtype=ms.int32)\n", "y = ms.Tensor(np.array(1), dtype=ms.int32)\n", "\n", "output = forward_net(x, y)\n", "print(\"output:\", output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 变量条件的while语句\n", "\n", "1. 约束一：**当while语句中的条件表达式是变量条件时，while循环体内部不能出现标量、List、Tuple等非Tensor类型的计算操作**。\n", "\n", "为了避免产生过多的控制流算子，我们可以尝试使用条件表达式为变量条件的`while`语句重写上面的代码：" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "output: 5\n" ] } ], "source": [ "import numpy as np\n", "from mindspore import nn\n", "import mindspore as ms\n", "\n", "class IfInWhileNet(nn.Cell):\n", "\n", " def construct(self, x, y, i):\n", " out = x\n", " # 构造条件表达式为变量条件的while语句\n", " while i < 3:\n", " # 构造条件表达式为变量条件的if语句\n", " if x + i < y:\n", " out = out + x\n", " else:\n", " out = out + y\n", " out = out + 1\n", " i = i + 1\n", " return out\n", "\n", "forward_net = IfInWhileNet()\n", "i = ms.Tensor(np.array(0), dtype=ms.int32)\n", "x = ms.Tensor(np.array(0), dtype=ms.int32)\n", "y = ms.Tensor(np.array(1), dtype=ms.int32)\n", "\n", "output = forward_net(x, y, i)\n", "print(\"output:\", output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "需要注意的是，在上面的代码中，`while`语句的条件表达式为变量条件，`while`循环体不会被展开，`while`循环体内的表达式都是在各个step运行时计算，同时也产生了如下约束：\n", "\n", "> 当`while`语句中的条件表达式是变量条件时，`while`循环体内部不能出现标量、List、Tuple等非Tensor类型的计算操作。\n", "\n", "因为这些类型的计算操作是在图编译时期完成的，这与`while`循环体在执行期进行计算的机制是矛盾的。下面我们通过示例代码说明：\n", "\n", "```Python\n", "import numpy as np\n", "from mindspore import nn\n", "import mindspore as ms\n", "class IfInWhileNet(nn.Cell):\n", "\n", " def __init__(self):\n", " super().__init__()\n", " self.nums = [1, 2, 3]\n", "\n", " def construct(self, x, y, i):\n", " j = 0\n", " out = x\n", "\n", " # 构造条件表达式为变量条件的while语句\n", " while i < 3:\n", " if x + i < y:\n", " out = out + x\n", " else:\n", " out = out + y\n", " out = out + self.nums[j]\n", " i = i + 1\n", " # 在条件表达式为变量条件的while语句循环体内构造标量计算\n", " j = j + 1\n", "\n", " return out\n", "\n", "forward_net = IfInWhileNet()\n", "i = ms.Tensor(np.array(0), dtype=ms.int32)\n", "x = ms.Tensor(np.array(0), dtype=ms.int32)\n", "y = ms.Tensor(np.array(1), dtype=ms.int32)\n", "\n", "output = forward_net(x, y, i)\n", "```\n", "\n", "上面的代码中，条件表达式`i < 3`为变量条件的`while`循环体内部存在标量计算`j = j + 1`，因此会导致图编译出错。代码在执行时报错信息如下：\n", "\n", "```text\n", "IndexError: mindspore/core/abstract/prim_structures.cc:127 InferTupleOrListGetItem] list_getitem evaluator index should be in range[-3, 3), but got 3.\n", "```\n", "\n", "2. 约束二：**当while语句中的条件表达式是变量条件时，循环体内部不能更改算子的输入shape，并且循环体内与循环体外相同名称变量数据类型应该一致，例如Tensor类型变量要求shape和type一致。**Tensor变量的shape一致性约束详见[ShapeJoin规则](#shapejoin规则)。\n", "\n", "MindSpore要求网络的同一个算子的输入shape在图编译时是确定的，而在`while`的循环体内部改变算子输入shape的操作是在图执行时生效，两者是矛盾的。\n", "\n", "下面我们通过示例代码来说明：\n", "\n", "```Python\n", "import numpy as np\n", "from mindspore import nn\n", "import mindspore as ms\n", "from mindspore import ops\n", "\n", "class IfInWhileNet(nn.Cell):\n", "\n", " def __init__(self):\n", " super().__init__()\n", " self.expand_dims = ops.ExpandDims()\n", "\n", " def construct(self, x, y, i):\n", " out = x\n", " # 构造条件表达式为变量条件的while语句\n", " while i < 3:\n", " if x + i < y:\n", " out = out + x\n", " else:\n", " out = out + y\n", " out = out + 1\n", " # 更改算子的输入shape\n", " out = self.expand_dims(out, -1)\n", " i = i + 1\n", " return out\n", "\n", "forward_net = IfInWhileNet()\n", "i = ms.Tensor(np.array(0), dtype=ms.int32)\n", "x = ms.Tensor(np.array(0), dtype=ms.int32)\n", "y = ms.Tensor(np.array(1), dtype=ms.int32)\n", "\n", "output = forward_net(x, y, i)\n", "```\n", "\n", "上面的代码中，条件表达式`i < 3`为变量条件的`while`循环体内部的`ExpandDims`算子会改变表达式`out = out + 1`在下一轮循环的输入shape，因此会导致图编译出错。代码在执行时报错信息如下:\n", "\n", "```text\n", "ValueError: mindspore/ccsrc/pipeline/jit/static_analysis/static_analysis.cc:800 ProcessEvalResults] Cannot join the return values of different branches, perhaps you need to make them equal.\n", "Shape Join Failed: shape1 = (1), shape2 = (1, 1).\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ShapeJoin规则\n", "\n", "`unknow_shape`表示动态shape场景下，对应维度的长度是动态的，`unknown_rank`表示动态rank场景下shape的维度是动态的，`shape1`与`shape2`分别表示进行Join的两个分支的shape。当满足下面任一规则时，Shape Join会成功，否则会出现`Shape Join Failed`异常。\n", "\n", "- 规则1：\n", "\n", " shape1和shape2维度均固定且两者维度相等，且shape1[i]等于shape2[i]。\n", "- 规则2：\n", "\n", " shape1和shape2维度均固定且两者维度相等，且shape1[i]或shape2[i]至少有一个是unknown_shape。\n", "- 规则3：\n", "\n", " shape1和shape2维度至少有一个是动态的，即shape1或者shape2是动态rank。\n", "- 规则4：\n", "\n", " shape1和shape2维度固定且两者维度不相等，较小的维度是m, 较大的维度是n。\n", "\n", " 在0~m-1维范围内满足：\n", "\n", " 1. shape1[i]或shape2[i]相等。\n", "\n", " 2. 或shape1[i]与shape2[i]均是unknown_shape。\n", "\n", " 在m~n-1维范围内满足：维度较大者的shape[i]是unknown_shape。\n", "\n", "下述列表是Shape Join的规则示例。\n", "\n", "| shape1 | shape2 | Join结果|\n", "| :----- | :----- | :------- |\n", "| (3, 4)| (3, 4)| (3, 4) |\n", "| (3, 5)| (3, 4)| Join Fail |\n", "| (3, 4)| (3, 4, 1)| Join Fail |\n", "| (3, unknown_shape) | (3, 4)| (3, unknown_shape) |\n", "| unknown_rank | (3, 4)| unknown_rank |\n", "| (3, unknown_shape)| (3, unknown_shape, unknown_shape)| unknown_rank |\n", "| (3, unknown_shape)| (4, unknown_shape, unknown_shape)| Join Fail |\n", "| (3, unknown_shape)| (3, 4, unknown_shape)| Join Fail |\n" ] } ], "metadata": { "kernelspec": { "display_name": "MindSpore", "language": "python", "name": "mindspore" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 4 }