{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 加载文本数据集\n",
    "\n",
    "`Linux` `Ascend` `GPU` `CPU` `数据准备` `初级` `中级` `高级`\n",
    "\n",
    "[![](https://gitee.com/mindspore/docs/raw/r1.2/tutorials/training/source_zh_cn/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.2/tutorials/training/source_zh_cn/use/load_dataset_text.ipynb) \n",
    "[![](https://gitee.com/mindspore/docs/raw/r1.2/tutorials/training/source_zh_cn/_static/logo_notebook.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.2/mindspore_load_dataset_text.ipynb) \n",
    "[![](https://gitee.com/mindspore/docs/raw/r1.2/tutorials/training/source_zh_cn/_static/logo_modelarts.png)](https://console.huaweicloud.com/modelarts/?region=cn-north-4#/notebook/loading?share-url-b64=aHR0cHM6Ly9vYnMuZHVhbHN0YWNrLmNuLW5vcnRoLTQubXlodWF3ZWljbG91ZC5jb20vbWluZHNwb3JlLXdlYnNpdGUvbm90ZWJvb2svbW9kZWxhcnRzL21pbmRzcG9yZV9sb2FkX2RhdGFzZXRfdGV4dC5pcHluYg==&image_id=65f636a0-56cf-49df-b941-7d2a07ba8c8c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 概述\n",
    "\n",
    "MindSpore提供的`mindspore.dataset`模块可以帮助用户构建数据集对象，分批次地读取文本数据。同时，在各个数据集类中还内置了数据处理和数据分词算子，使得数据在训练过程中能够像经过pipeline管道的水一样源源不断地流向训练系统，提升数据训练效果。\n",
    "\n",
    "此外，MindSpore还支持分布式场景数据加载，用户可以在加载数据集时指定分片数目，具体用法参见[数据并行模式加载数据集](https://www.mindspore.cn/tutorial/training/zh-CN/r1.2/advanced_use/distributed_training_ascend.html#id6)。\n",
    "\n",
    "下面，本教程将简要演示如何使用MindSpore加载和处理文本数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备环节"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 准备文本数据，内容如下：\n",
    "\n",
    "    ```text\n",
    "    Welcome to Beijing\n",
    "    北京欢迎您！\n",
    "    我喜欢English!\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 创建`tokenizer.txt`文件并复制文本数据到该文件中，将该文件存放在`./datasets`路径下。执行如下代码完成本步骤。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./datasets\n",
      "└── tokenizer.txt\n",
      "\n",
      "0 directories, 1 file\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "if not os.path.exists('./datasets'):\n",
    "    os.mkdir('./datasets')\n",
    "file_handle=open('./datasets/tokenizer.txt',mode='w')\n",
    "file_handle.write('Welcome to Beijing \\n北京欢迎您！ \\n我喜欢English! \\n')\n",
    "file_handle.close()\n",
    "! tree ./datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. 导入`mindspore.dataset`和`mindspore.dataset.text`模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mindspore.dataset as ds\n",
    "import mindspore.dataset.text as text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 加载数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MindSpore目前支持加载文本领域常用的经典数据集和多种数据存储格式下的数据集，用户也可以通过构建自定义数据集类实现自定义方式的数据加载。各种数据集的详细加载方法，可参考编程指南中[数据集加载](https://www.mindspore.cn/doc/programming_guide/zh-CN/r1.2/dataset_loading.html)章节。\n",
    "\n",
    "下面演示使用`MindSpore.dataset`模块中的`TextFileDataset`类加载数据集。\n",
    "\n",
    "1. 配置数据集目录，创建数据集对象。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_FILE = './datasets/tokenizer.txt'\n",
    "dataset = ds.TextFileDataset(DATA_FILE, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 创建字典迭代器，通过迭代器获取数据，可以获得分词前的数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Welcome to Beijing \n",
      "北京欢迎您！ \n",
      "我喜欢English! \n"
     ]
    }
   ],
   "source": [
    "for data in dataset.create_dict_iterator(output_numpy=True):\n",
    "    print(text.to_str(data['text']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MindSpore目前支持的数据处理算子及其详细使用方法，可参考编程指南中[数据处理](https://www.mindspore.cn/doc/programming_guide/zh-CN/r1.2/pipeline.html)章节。\n",
    "\n",
    "下面演示构建pipeline，对文本数据集进行混洗和文本替换操作。\n",
    "\n",
    "1. 对数据集进行混洗。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我喜欢English! \n",
      "Welcome to Beijing \n",
      "北京欢迎您！ \n"
     ]
    }
   ],
   "source": [
    "ds.config.set_seed(58)\n",
    "dataset = dataset.shuffle(buffer_size=3)\n",
    "\n",
    "for data in dataset.create_dict_iterator(output_numpy=True):\n",
    "    print(text.to_str(data['text']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 对数据集进行文本替换。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我喜欢English! \n",
      "Welcome to Shanghai \n",
      "上海欢迎您！ \n"
     ]
    }
   ],
   "source": [
    "replace_op1 = text.RegexReplace(\"Beijing\", \"Shanghai\")\n",
    "replace_op2 = text.RegexReplace(\"北京\", \"上海\")\n",
    "\n",
    "dataset = dataset.map(operations=replace_op1)\n",
    "dataset = dataset.map(operations=replace_op2)\n",
    "\n",
    "for data in dataset.create_dict_iterator(output_numpy=True):\n",
    "    print(text.to_str(data['text']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据分词"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MindSpore目前支持的数据分词算子及其详细使用方法，可参考编程指南中[分词器](https://www.mindspore.cn/doc/programming_guide/zh-CN/r1.2/tokenizer.html)章节。\n",
    "\n",
    "下面演示使用`WhitespaceTokenizer`分词器来分词，该分词是按照空格来进行分词。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 创建`tokenizer`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = text.WhitespaceTokenizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 执行操作`tokenizer`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.map(operations=tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. 创建字典迭代器，通过迭代器获取数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['我喜欢English!']\n",
      "['Welcome', 'to', 'Shanghai']\n",
      "['上海欢迎您！']\n"
     ]
    }
   ],
   "source": [
    "for data in dataset.create_dict_iterator(num_epochs=1,output_numpy=True):\n",
    "    print(text.to_str(data['text']).tolist())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore-1.1.1",
   "language": "python",
   "name": "mindspore-1.1.1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}