{ "cells": [ { "cell_type": "markdown", "source": [ "# 数据集加载总览\n", "\n", "`Ascend` `GPU` `CPU` `数据准备`\n", "\n", "[![在线运行](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9tb2RlbGFydHMvcHJvZ3JhbW1pbmdfZ3VpZGUvbWluZHNwb3JlX2RhdGFzZXRfbG9hZGluZy5pcHluYg==&imageid=65f636a0-56cf-49df-b941-7d2a07ba8c8c) [![下载Notebook](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_notebook.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.6/programming_guide/zh_cn/mindspore_dataset_loading.ipynb) [![下载样例代码](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_download_code.png)](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/r1.6/programming_guide/zh_cn/mindspore_dataset_loading.py) [![查看源文件](https://gitee.com/mindspore/docs/raw/r1.6/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r1.6/docs/mindspore/programming_guide/source_zh_cn/dataset_loading.ipynb)" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 概述\n", "\n", "MindSpore支持加载图像领域常用的数据集，用户可以直接使用`mindspore.dataset`中对应的类实现数据集的加载。支持的常用数据集及对应的数据集类举例如下，更多数据集支持情况请参考API文档。\n", "\n", "| 图像数据集 | 数据集类 | 数据集简介 |\n", "| :--------- | :-------------- | :----------------------------------------------------------------------------------------------------------------------------------- |\n", "| MNIST | MnistDataset | MNIST是一个大型手写数字图像数据集，拥有60,000张训练图像和10,000张测试图像，常用于训练各种图像处理系统。 |\n", "| CIFAR-10 | Cifar10Dataset | CIFAR-10是一个微小图像数据集，包含10种类别下的60,000张32x32大小彩色图像，平均每种类别6,000张，其中5,000张为训练集，1,000张为测试集。 |\n", "| CIFAR-100 | Cifar100Dataset | CIFAR-100与CIFAR-10类似，但拥有100种类别，平均每种类别600张，其中500张为训练集，100张为测试集。 |\n", "| CelebA | CelebADataset | CelebA是一个大型人脸图像数据集，包含超过200,000张名人人脸图像，每张图像拥有40个特征标记。 |\n", "| PASCAL-VOC | VOCDataset | PASCAL-VOC是一个常用图像数据集，被广泛用于目标检测、图像分割等计算机视觉领域。 |\n", "| COCO | CocoDataset | COCO是一个大型目标检测、图像分割、姿态估计数据集。 |\n", "| CLUE | CLUEDataset | CLUE是一个大型中文语义理解数据集。 |\n", "| Manifest | ManifestDataset | Manifest是华为ModelArts支持的一种数据格式，描述了原始文件和标注信息，可用于标注、训练、推理场景。 |\n", "\n", "MindSpore支持加载文本领域常用的数据集，用户可以直接使用`mindspore.dataset`中对应的类实现数据集的加载。支持的常用数据集及对应的数据集类举例如下，更多数据集支持情况请参考API文档。\n", "\n", "| 文本数据集 | 数据集类 | 数据集简介 |\n", "| :--------- | :-------------- | :----------------------------------------------------------------------------------------------------------------------------------- |\n", "| IMDB | IMDBDataset | IMDB 数据集包含来自互联网电影数据库（IMDB）的 50,000 条严重两极分化的评论，25,000 条用于训练，25000条用于测试。 |\n", "| Wiki Text | WikiTextDataset | WikiText 英语词库数据是一个包含1亿个词汇的英文词库数据，这些词汇是从Wikipedia的优质文章和标杆文章中提取得到。 |\n", "| Yahoo Answers | YahooAnswersDataset | 数据集的 10 个主要分类数据。每个类 别分别包含 140,000 个训练样本和 5,000 个测试样本。 |\n", "| Text File | TextFileDataset | 文本文件数据集，其中每行文本是一个样本。 |\n", "\n", "MindSpore支持加载音频领域常用的数据集，用户可以直接使用`mindspore.dataset`中对应的类实现数据集的加载。支持的常用数据集及对应的数据集类举例如下，更多数据集支持情况请参考API文档。\n", "\n", "| 音频数据集 | 数据集类 | 数据集简介 |\n", "| :--------- | :-------------- | :----------------------------------------------------------------------------------------------------------------------------------- |\n", "| LJSpeech | LJSpeechDataset | 这是一个公共领域数据集语音数据集，由 13,100 个短音频剪辑组成，单个发言者阅读 7 本非小说类书籍段落。 |\n", "| Speech Commands | SpeechCommandsDataset | 是一个有声单词的音频数据集，旨在帮助训练和评估关键字识别系统。 |\n", "| Ted-Lium | TedliumDataset | TED-LIUM语料库是英语TED演讲，带有转录，采样频率为 16kHZ，它包含大约 118 个小时的演讲时间。 |\n", "\n", "MindSpore还支持加载多种数据存储格式下的数据集，用户可以直接使用`mindspore.dataset`中对应的类加载磁盘中的数据文件。目前支持的数据格式及对应加载方式如下表所示。\n", "\n", "| 数据格式 | 数据集类 | 数据格式简介 |\n", "| :--------- | :----------------- | :------------------------------------------------------------------------------------------------ |\n", "| MindRecord | MindDataset | MindRecord是MindSpore的自研数据格式，具有读写高效、易于分布式处理等优势。 |\n", "| TFRecord | TFRecordDataset | TFRecord是TensorFlow定义的一种二进制数据文件格式。 |\n", "| CSV File | CSVDataset | CSV指逗号分隔值，其文件以纯文本形式存储表格数据。 |\n", "\n", "MindSpore也同样支持使用`GeneratorDataset`自定义数据集的加载方式，用户可以根据需要实现自己的数据集类。\n", "\n", "| 自定义数据集类 | 数据格式简介 |\n", "| :----------------- | :------------------------------------ |\n", "| GeneratorDataset | 用户自定义的数据集读取、处理的方式。 |\n", "| NumpySlicesDataset | 用户自定义的由NumPy构建数据集的方式。 |\n", "\n", "> 更多详细的数据集加载接口说明，参见[API文档](https://www.mindspore.cn/docs/api/zh-CN/r1.6/api_python/mindspore.dataset.html)。" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 常用数据集加载\n", "\n", "下面将介绍几种常用数据集的加载方式。\n", "\n", "### CIFAR-10/100数据集\n", "\n", "下载[CIFAR-10数据集](https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz)并解压到指定位置，以下示例代码将数据集下载并解压到指定位置。" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "import os\n", "import requests\n", "import tarfile\n", "import zipfile\n", "import shutil\n", "\n", "requests.packages.urllib3.disable_warnings()\n", "\n", "def download_dataset(url, target_path):\n", " \"\"\"download and decompress dataset\"\"\"\n", " if not os.path.exists(target_path):\n", " os.makedirs(target_path)\n", " download_file = url.split(\"/\")[-1]\n", " if not os.path.exists(download_file):\n", " res = requests.get(url, stream=True, verify=False)\n", " if download_file.split(\".\")[-1] not in [\"tgz\", \"zip\", \"tar\", \"gz\"]:\n", " download_file = os.path.join(target_path, download_file)\n", " with open(download_file, \"wb\") as f:\n", " for chunk in res.iter_content(chunk_size=512):\n", " if chunk:\n", " f.write(chunk)\n", " if download_file.endswith(\"zip\"):\n", " z = zipfile.ZipFile(download_file, \"r\")\n", " z.extractall(path=target_path)\n", " z.close()\n", " if download_file.endswith(\".tar.gz\") or download_file.endswith(\".tar\") or download_file.endswith(\".tgz\"):\n", " t = tarfile.open(download_file)\n", " names = t.getnames()\n", " for name in names:\n", " t.extract(name, target_path)\n", " t.close()\n", " print(\"The {} file is downloaded and saved in the path {} after processing\".format(os.path.basename(url), target_path))\n", "\n", "download_dataset(\"https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/cifar-10-binary.tar.gz\", \"./datasets\")\n", "test_path = \"./datasets/cifar-10-batches-bin/test\"\n", "train_path = \"./datasets/cifar-10-batches-bin/train\"\n", "os.makedirs(test_path, exist_ok=True)\n", "os.makedirs(train_path, exist_ok=True)\n", "if not os.path.exists(os.path.join(test_path, \"test_batch.bin\")):\n", " shutil.move(\"./datasets/cifar-10-batches-bin/test_batch.bin\", test_path)\n", "[shutil.move(\"./datasets/cifar-10-batches-bin/\"+i, train_path) for i in os.listdir(\"./datasets/cifar-10-batches-bin/\") if os.path.isfile(\"./datasets/cifar-10-batches-bin/\"+i) and not i.endswith(\".html\") and not os.path.exists(os.path.join(train_path, i))]" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "解压后数据集文件的目录结构如下：\n", "\n", "```text\n", "./datasets/cifar-10-batches-bin\n", "├── readme.html\n", "├── test\n", "│ └── test_batch.bin\n", "└── train\n", " ├── batches.meta.txt\n", " ├── data_batch_1.bin\n", " ├── data_batch_2.bin\n", " ├── data_batch_3.bin\n", " ├── data_batch_4.bin\n", " └── data_batch_5.bin\n", "```" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "下面的样例通过`Cifar10Dataset`接口加载CIFAR-10数据集，使用顺序采样器获取其中5个样本，然后展示了对应图片的形状和标签。\n", "\n", "CIFAR-100数据集和MNIST数据集的加载方式也与之类似。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 2, "source": [ "import mindspore.dataset as ds\n", "\n", "DATA_DIR = \"./datasets/cifar-10-batches-bin/train/\"\n", "\n", "sampler = ds.SequentialSampler(num_samples=5)\n", "dataset = ds.Cifar10Dataset(DATA_DIR, sampler=sampler)\n", "\n", "for data in dataset.create_dict_iterator():\n", " print(\"Image shape:\", data['image'].shape, \", Label:\", data['label'])" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Image shape: (32, 32, 3) , Label: 6\n", "Image shape: (32, 32, 3) , Label: 9\n", "Image shape: (32, 32, 3) , Label: 9\n", "Image shape: (32, 32, 3) , Label: 4\n", "Image shape: (32, 32, 3) , Label: 1\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### VOC数据集\n", "\n", "VOC数据集有多个版本，此处以VOC2012为例。下载[VOC2012数据集](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar)并解压，如果点击下载不成功，请尝试复制链接地址后下载。目录结构如下。\n", "\n", "```text\n", "└─ VOCtrainval_11-May-2012\n", "   └── VOCdevkit\n", "    └── VOC2012\n", " ├── Annotations\n", " ├── ImageSets\n", " ├── JPEGImages\n", " ├── SegmentationClass\n", " └── SegmentationObject\n", "```\n", "\n", "下面的样例通过`VOCDataset`接口加载VOC2012数据集，分别演示了将任务指定为分割（Segmentation）和检测（Detection）时的原始图像形状和目标形状。\n", "\n", "```python\n", "import mindspore.dataset as ds\n", "\n", "DATA_DIR = \"VOCtrainval_11-May-2012/VOCdevkit/VOC2012/\"\n", "\n", "dataset = ds.VOCDataset(DATA_DIR, task=\"Segmentation\", usage=\"train\", num_samples=2, decode=True, shuffle=False)\n", "\n", "print(\"[Segmentation]:\")\n", "for data in dataset.create_dict_iterator():\n", " print(\"image shape:\", data[\"image\"].shape)\n", " print(\"target shape:\", data[\"target\"].shape)\n", "\n", "dataset = ds.VOCDataset(DATA_DIR, task=\"Detection\", usage=\"train\", num_samples=1, decode=True, shuffle=False)\n", "\n", "print(\"[Detection]:\")\n", "for data in dataset.create_dict_iterator():\n", " print(\"image shape:\", data[\"image\"].shape)\n", " print(\"bbox shape:\", data[\"bbox\"].shape)\n", "```\n", "\n", "输出结果：\n", "\n", "```text\n", "[Segmentation]:\n", "image shape: (281, 500, 3)\n", "target shape: (281, 500, 3)\n", "image shape: (375, 500, 3)\n", "target shape: (375, 500, 3)\n", "[Detection]:\n", "image shape: (442, 500, 3)\n", "bbox shape: (2, 4)\n", "```" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### COCO数据集\n", "\n", "COCO数据集有多个版本，此处以COCO2017的验证数据集为例。下载COCO2017的[验证集](http://images.cocodataset.org/zips/val2017.zip)、[检测任务标注](http://images.cocodataset.org/annotations/annotations_trainval2017.zip)和[全景分割任务标注](http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip)并解压，如果点击下载不成功，请尝试复制链接地址后下载。只取其中的验证集部分，按以下目录结构存放。\n", "\n", "```text\n", "└─ COCO\n", " ├── val2017\n", "   └── annotations\n", " ├── instances_val2017.json\n", " ├── panoptic_val2017.json\n", "    └── person_keypoints_val2017.json\n", "```\n", "\n", "下面的样例通过`CocoDataset`接口加载COCO2017数据集，分别演示了将任务指定为目标检测（Detection）、背景分割（Stuff）、关键点检测（Keypoint）和全景分割（Panoptic）时获取到的不同数据。\n", "\n", "```python\n", "import mindspore.dataset as ds\n", "\n", "DATA_DIR = \"COCO/val2017/\"\n", "ANNOTATION_FILE = \"COCO/annotations/instances_val2017.json\"\n", "KEYPOINT_FILE = \"COCO/annotations/person_keypoints_val2017.json\"\n", "PANOPTIC_FILE = \"COCO/annotations/panoptic_val2017.json\"\n", "\n", "dataset = ds.CocoDataset(DATA_DIR, annotation_file=ANNOTATION_FILE, task=\"Detection\", num_samples=1)\n", "for data in dataset.create_dict_iterator():\n", " print(\"Detection:\", data.keys())\n", "\n", "dataset = ds.CocoDataset(DATA_DIR, annotation_file=ANNOTATION_FILE, task=\"Stuff\", num_samples=1)\n", "for data in dataset.create_dict_iterator():\n", " print(\"Stuff:\", data.keys())\n", "\n", "dataset = ds.CocoDataset(DATA_DIR, annotation_file=KEYPOINT_FILE, task=\"Keypoint\", num_samples=1)\n", "for data in dataset.create_dict_iterator():\n", " print(\"Keypoint:\", data.keys())\n", "\n", "dataset = ds.CocoDataset(DATA_DIR, annotation_file=PANOPTIC_FILE, task=\"Panoptic\", num_samples=1)\n", "for data in dataset.create_dict_iterator():\n", " print(\"Panoptic:\", data.keys())\n", "```\n", "\n", "输出结果：\n", "\n", "```text\n", "Detection: dict_keys(['image', 'bbox', 'category_id', 'iscrowd'])\n", "Stuff: dict_keys(['image', 'segmentation', 'iscrowd'])\n", "Keypoint: dict_keys(['image', 'keypoints', 'num_keypoints'])\n", "Panoptic: dict_keys(['image', 'bbox', 'category_id', 'iscrowd', 'area'])\n", "```" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Manifest数据格式\n", "\n", "Manifest是华为ModelArts支持的数据格式文件，详细说明请参见[Manifest文档](https://support.huaweicloud.com/engineers-modelarts/modelarts_23_0009.html)。\n", "\n", "本次示例需下载测试数据`test_manifest.zip`并将其解压到指定位置，执行如下命令：" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "download_dataset(\"https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/datasets/test_manifest.zip\", \"./datasets/mindspore_dataset_loading/test_manifest/\")" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "解压后数据集文件的目录结构如下：\n", "\n", "```text\n", "./datasets/mindspore_dataset_loading/test_manifest/\n", "├── eval\n", "│ ├── 1.JPEG\n", "│ └── 2.JPEG\n", "├── test_manifest.json\n", "└── train\n", " ├── 1.JPEG\n", " └── 2.JPEG\n", "```" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "下面的样例通过`ManifestDataset`接口加载Manifest文件`test_manifest.json`，并展示已加载数据的标签。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 6, "source": [ "import mindspore.dataset as ds\n", "\n", "DATA_FILE = \"./datasets/mindspore_dataset_loading/test_manifest/test_manifest.json\"\n", "manifest_dataset = ds.ManifestDataset(DATA_FILE)\n", "\n", "for data in manifest_dataset.create_dict_iterator():\n", " print(data[\"label\"])" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "0\n", "1\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 特定格式数据集加载\n", "\n", "下面将介绍几种特定格式数据集文件的加载方式。\n", "\n", "### MindRecord数据格式\n", "\n", "MindRecord是MindSpore定义的一种数据格式，使用MindRecord能够获得更好的性能提升。\n", "\n", "> 阅读[数据格式转换](https://www.mindspore.cn/docs/programming_guide/zh-CN/r1.6/dataset_conversion.html)章节，了解如何将数据集转化为MindSpore数据格式。\n", "\n", "执行本例之前需下载对应的测试数据`test_mindrecord.zip`并解压到指定位置，执行如下命令：" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "download_dataset(\"https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/datasets/test_mindrecord.zip\", \"./datasets/mindspore_dataset_loading/\")" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "下载的数据集文件的目录结构如下：\n", "\n", "```text\n", "./datasets/mindspore_dataset_loading/\n", "├── test.mindrecord\n", "└── test.mindrecord.db\n", "```" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "下面的样例通过`MindDataset`接口加载MindRecord文件，并展示已加载数据的标签。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 4, "source": [ "import mindspore.dataset as ds\n", "\n", "DATA_FILE = [\"./datasets/mindspore_dataset_loading/test.mindrecord\"]\n", "mindrecord_dataset = ds.MindDataset(DATA_FILE)\n", "\n", "for data in mindrecord_dataset.create_dict_iterator(output_numpy=True):\n", " print(data.keys())" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "dict_keys(['chinese', 'english'])\n", "dict_keys(['chinese', 'english'])\n", "dict_keys(['chinese', 'english'])\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### TFRecord数据格式\n", "\n", "TFRecord是TensorFlow定义的一种二进制数据文件格式。\n", "\n", "下面的样例通过`TFRecordDataset`接口加载TFRecord文件，并介绍了两种不同的数据集格式设定方案。" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "下载`tfrecord`测试数据`test_tftext.zip`并解压到指定位置，执行如下命令：" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "download_dataset(\"https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/datasets/test_tftext.zip\", \"./datasets/mindspore_dataset_loading/test_tfrecord/\")" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "解压后数据集文件的目录结构如下：\n", "\n", "```text\n", "./datasets/mindspore_dataset_loading/test_tfrecord/\n", "└── test_tftext.tfrecord\n", "```" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "1. 传入数据集路径或TFRecord文件列表，本例使用`test_tftext.tfrecord`，创建`TFRecordDataset`对象。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 8, "source": [ "import mindspore.dataset as ds\n", "\n", "DATA_FILE = \"./datasets/mindspore_dataset_loading/test_tfrecord/test_tftext.tfrecord\"\n", "tfrecord_dataset = ds.TFRecordDataset(DATA_FILE)\n", "\n", "for tf_data in tfrecord_dataset.create_dict_iterator():\n", " print(tf_data.keys())" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "dict_keys(['chinese', 'line', 'words'])\n", "dict_keys(['chinese', 'line', 'words'])\n", "dict_keys(['chinese', 'line', 'words'])\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "2. 用户可以通过编写Schema文件或创建Schema对象，设定数据集格式及特征。\n", "\n", " - 编写Schema文件\n", "\n", " 将数据集格式和特征按JSON格式写入Schema文件。\n", "\n", " - `columns`：列信息字段，需要根据数据集的实际列名定义。上面的示例中，数据集有三组数据，其列均为`chinese`、`line`和`words`。\n", "\n", " 然后在创建`TFRecordDataset`时将Schema文件路径传入。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 9, "source": [ "import os\n", "import json\n", "\n", "data_json = {\n", " \"columns\": {\n", " \"chinese\": {\n", " \"type\": \"uint8\",\n", " \"rank\": 1\n", " },\n", " \"line\": {\n", " \"type\": \"int8\",\n", " \"rank\": 1\n", " },\n", " \"words\": {\n", " \"type\": \"uint8\",\n", " \"rank\": 0\n", " }\n", " }\n", " }\n", "\n", "if not os.path.exists(\"dataset_schema_path\"):\n", " os.mkdir(\"dataset_schema_path\")\n", "SCHEMA_DIR = \"dataset_schema_path/schema.json\"\n", "with open(SCHEMA_DIR, \"w\") as f:\n", " json.dump(data_json, f, indent=4)\n", "\n", "tfrecord_dataset = ds.TFRecordDataset(DATA_FILE, schema=SCHEMA_DIR)\n", "\n", "for tf_data in tfrecord_dataset.create_dict_iterator():\n", " print(tf_data.values())" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "dict_values([Tensor(shape=[57], dtype=UInt8, value= [230, 177, 159, 229, 183, 158, 229, 184, 130, 233, 149, 191, 230, 177, 159, 229, 164, 167, 230, 161, 165, 229, 143, 130, \n", " 229, 138, 160, 228, 186, 134, 233, 149, 191, 230, 177, 159, 229, 164, 167, 230, 161, 165, 231, 154, 132, 233, 128, 154, \n", " 232, 189, 166, 228, 187, 170, 229, 188, 143]), Tensor(shape=[22], dtype=Int8, value= [ 71, 111, 111, 100, 32, 108, 117, 99, 107, 32, 116, 111, 32, 101, 118, 101, 114, 121, 111, 110, 101, 46]), Tensor(shape=[32], dtype=UInt8, value= [229, 165, 179, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 101, 118, 101, 114, 121, 111, 110, 101, \n", " 99, 32, 32, 32, 32, 32, 32, 32])])\n", "dict_values([Tensor(shape=[12], dtype=UInt8, value= [231, 148, 183, 233, 187, 152, 229, 165, 179, 230, 179, 170]), Tensor(shape=[19], dtype=Int8, value= [ 66, 101, 32, 104, 97, 112, 112, 121, 32, 101, 118, 101, 114, 121, 32, 100, 97, 121, 46]), Tensor(shape=[20], dtype=UInt8, value= [ 66, 101, 32, 32, 32, 104, 97, 112, 112, 121, 100, 97, 121, 32, 32, 98, 32, 32, 32, 32])])\n", "dict_values([Tensor(shape=[48], dtype=UInt8, value= [228, 187, 138, 229, 164, 169, 229, 164, 169, 230, 176, 148, 229, 164, 170, 229, 165, 189, 228, 186, 134, 230, 136, 145, \n", " 228, 187, 172, 228, 184, 128, 232, 181, 183, 229, 142, 187, 229, 164, 150, 233, 157, 162, 231, 142, 169, 229, 144, 167\n", " ]), Tensor(shape=[20], dtype=Int8, value= [ 84, 104, 105, 115, 32, 105, 115, 32, 97, 32, 116, 101, 120, 116, 32, 102, 105, 108, 101, 46]), Tensor(shape=[16], dtype=UInt8, value= [ 84, 104, 105, 115, 116, 101, 120, 116, 102, 105, 108, 101, 97, 32, 32, 32])])\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "- 创建Schema对象\n", "\n", " 创建Schema对象，为其添加自定义字段，然后在创建数据集对象时传入。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 10, "source": [ "from mindspore import dtype as mstype\n", "schema = ds.Schema()\n", "schema.add_column('chinese', de_type=mstype.uint8)\n", "schema.add_column('line', de_type=mstype.uint8)\n", "tfrecord_dataset = ds.TFRecordDataset(DATA_FILE, schema=schema)\n", "\n", "for tf_data in tfrecord_dataset.create_dict_iterator():\n", " print(tf_data)" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "{'chinese': Tensor(shape=[12], dtype=UInt8, value= [231, 148, 183, 233, 187, 152, 229, 165, 179, 230, 179, 170]), 'line': Tensor(shape=[19], dtype=UInt8, value= [ 66, 101, 32, 104, 97, 112, 112, 121, 32, 101, 118, 101, 114, 121, 32, 100, 97, 121, 46])}\n", "{'chinese': Tensor(shape=[48], dtype=UInt8, value= [228, 187, 138, 229, 164, 169, 229, 164, 169, 230, 176, 148, 229, 164, 170, 229, 165, 189, 228, 186, 134, 230, 136, 145, \n", " 228, 187, 172, 228, 184, 128, 232, 181, 183, 229, 142, 187, 229, 164, 150, 233, 157, 162, 231, 142, 169, 229, 144, 167\n", " ]), 'line': Tensor(shape=[20], dtype=UInt8, value= [ 84, 104, 105, 115, 32, 105, 115, 32, 97, 32, 116, 101, 120, 116, 32, 102, 105, 108, 101, 46])}\n", "{'chinese': Tensor(shape=[57], dtype=UInt8, value= [230, 177, 159, 229, 183, 158, 229, 184, 130, 233, 149, 191, 230, 177, 159, 229, 164, 167, 230, 161, 165, 229, 143, 130, \n", " 229, 138, 160, 228, 186, 134, 233, 149, 191, 230, 177, 159, 229, 164, 167, 230, 161, 165, 231, 154, 132, 233, 128, 154, \n", " 232, 189, 166, 228, 187, 170, 229, 188, 143]), 'line': Tensor(shape=[22], dtype=UInt8, value= [ 71, 111, 111, 100, 32, 108, 117, 99, 107, 32, 116, 111, 32, 101, 118, 101, 114, 121, 111, 110, 101, 46])}\n" ] } ], "metadata": { "scrolled": true } }, { "cell_type": "markdown", "source": [ "对比上述中的编写和创建步骤，可以看出：\n", "\n", "|步骤|chinese|line|words\n", "|:---|:---|:---|:---\n", "| 编写|UInt8 |Int8|UInt8\n", "| 创建|UInt8 |UInt8|\n", "\n", "示例编写步骤中的`columns`中数据由`chinese`（UInt8）、`line`（Int8）和`words`（UInt8）变为了示例创建步骤中的`chinese`（UInt8）、`line`（UInt8），通过Schema对象，设定数据集的数据类型和特征，使得列中的数据类型和特征相应改变了。" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### CSV数据格式\n", "\n", "下面的样例通过`CSVDataset`加载CSV格式数据集文件，并展示了已加载数据的`keys`。\n", "\n", "下载测试数据`test_csv.zip`并解压到指定位置，执行如下命令：" ], "metadata": {} }, { "cell_type": "code", "execution_count": null, "source": [ "download_dataset(\"https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/datasets/test_csv.zip\", \"./datasets/mindspore_dataset_loading/test_csv/\")" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "解压后数据集文件的目录结构如下：\n", "\n", "```text\n", "./datasets/mindspore_dataset_loading/test_csv/\n", "├── test1.csv\n", "└── test2.csv\n", "```" ], "metadata": {} }, { "cell_type": "markdown", "source": [ "传入数据集路径或CSV文件列表，Text格式数据集文件的加载方式与CSV文件类似。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 15, "source": [ "import mindspore.dataset as ds\n", "\n", "DATA_FILE = [\"./datasets/mindspore_dataset_loading/test_csv/test1.csv\", \"./datasets/mindspore_dataset_loading/test_csv/test2.csv\"]\n", "csv_dataset = ds.CSVDataset(DATA_FILE)\n", "\n", "for csv_data in csv_dataset.create_dict_iterator(output_numpy=True):\n", " print(csv_data.keys())" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "dict_keys(['a', 'b', 'c', 'd'])\n", "dict_keys(['a', 'b', 'c', 'd'])\n", "dict_keys(['a', 'b', 'c', 'd'])\n", "dict_keys(['a', 'b', 'c', 'd'])\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## 自定义数据集加载\n", "\n", "对于目前MindSpore不支持直接加载的数据集，可以通过`GeneratorDataset`接口实现自定义方式的加载，或者将其转换成MindRecord数据格式。`GeneratorDataset`接口接收一个可随机访问对象或可迭代对象，由该对象自定义数据读取的方式。\n", "\n", "> 1. 带`__getitem__`函数的随机访问对象相比可迭代对象，不需进行index递增等操作，逻辑更精简，易于使用。\n", "> 2. 分布式训练场景需对数据集进行切片操作，`GeneratorDataset`初始化时可以接收sampler参数, 也可接收`num_shards、shard_id来指定切片份数和取第几份，后面这种方式更易于使用。\n", "\n", "下面分别展示这两种不同的自定义数据集加载方法，为了便于对比，生成的随机数据保持相同。\n", "\n", "### 构造可随机访问对象\n", "\n", "可随机访问的对象具有__getitem__函数，能够随机访问指定索引位置的数据。定义数据集类的时候重写__getitem__函数，即可使得该类的对象支持随机访问。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 16, "source": [ "import numpy as np\n", "import mindspore.dataset as ds\n", "\n", "class GetDatasetGenerator:\n", " def __init__(self):\n", " np.random.seed(58)\n", " self.__data = np.random.sample((5, 2))\n", " self.__label = np.random.sample((5, 1))\n", "\n", " def __getitem__(self, index):\n", " return (self.__data[index], self.__label[index])\n", "\n", " def __len__(self):\n", " return len(self.__data)\n", "\n", "dataset_generator = GetDatasetGenerator()\n", "dataset = ds.GeneratorDataset(dataset_generator, [\"data\", \"label\"], shuffle=False)\n", "\n", "for data in dataset.create_dict_iterator():\n", " print(data[\"data\"], data[\"label\"])" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[0.36510558 0.45120592] [0.78888122]\n", "[0.49606035 0.07562207] [0.38068183]\n", "[0.57176158 0.28963401] [0.16271622]\n", "[0.30880446 0.37487617] [0.54738768]\n", "[0.81585667 0.96883469] [0.77994068]\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### 构造可迭代对象\n", "\n", "可迭代的对象具有__iter__函数和__next__函数，能够在每次调用时返回一条数据。定义数据集类的时候重写__iter__函数和__next__函数，通过__iter__函数返回迭代器，通过__next__函数定义数据集加载方式，即可使得该类的对象可迭代。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 17, "source": [ "import numpy as np\n", "import mindspore.dataset as ds\n", "\n", "class IterDatasetGenerator:\n", " def __init__(self):\n", " np.random.seed(58)\n", " self.__index = 0\n", " self.__data = np.random.sample((5, 2))\n", " self.__label = np.random.sample((5, 1))\n", "\n", " def __next__(self):\n", " if self.__index >= len(self.__data):\n", " raise StopIteration\n", " else:\n", " item = (self.__data[self.__index], self.__label[self.__index])\n", " self.__index += 1\n", " return item\n", "\n", " def __iter__(self):\n", " self.__index = 0\n", " return self\n", "\n", " def __len__(self):\n", " return len(self.__data)\n", "\n", "dataset_generator = IterDatasetGenerator()\n", "dataset = ds.GeneratorDataset(dataset_generator, [\"data\", \"label\"], shuffle=False)\n", "\n", "for data in dataset.create_dict_iterator():\n", " print(data[\"data\"], data[\"label\"])" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[0.36510558 0.45120592] [0.78888122]\n", "[0.49606035 0.07562207] [0.38068183]\n", "[0.57176158 0.28963401] [0.16271622]\n", "[0.30880446 0.37487617] [0.54738768]\n", "[0.81585667 0.96883469] [0.77994068]\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "需要注意的是，如果数据集本身并不复杂，直接定义一个可迭代的函数即可快速实现自定义加载功能。" ], "metadata": {} }, { "cell_type": "code", "execution_count": 18, "source": [ "import numpy as np\n", "import mindspore.dataset as ds\n", "\n", "np.random.seed(58)\n", "data = np.random.sample((5, 2))\n", "label = np.random.sample((5, 1))\n", "\n", "def GeneratorFunc():\n", " for i in range(5):\n", " yield (data[i], label[i])\n", "\n", "dataset = ds.GeneratorDataset(GeneratorFunc, [\"data\", \"label\"])\n", "\n", "for item in dataset.create_dict_iterator():\n", " print(item[\"data\"], item[\"label\"])" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[0.36510558 0.45120592] [0.78888122]\n", "[0.49606035 0.07562207] [0.38068183]\n", "[0.57176158 0.28963401] [0.16271622]\n", "[0.30880446 0.37487617] [0.54738768]\n", "[0.81585667 0.96883469] [0.77994068]\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### NumPy数据格式\n", "\n", "如果所有数据已经读入内存，可以直接使用`NumpySlicesDataset`类将其加载。\n", "\n", "下面的样例分别介绍了通过`NumpySlicesDataset`加载arrays数据、 list数据和dict数据的方式。\n", "\n", "- 加载NumPy arrays数据" ], "metadata": {} }, { "cell_type": "code", "execution_count": 11, "source": [ "import numpy as np\n", "import mindspore.dataset as ds\n", "\n", "np.random.seed(6)\n", "features, labels = np.random.sample((4, 2)), np.random.sample((4, 1))\n", "\n", "data = (features, labels)\n", "dataset = ds.NumpySlicesDataset(data, column_names=[\"col1\", \"col2\"], shuffle=False)\n", "\n", "for np_arr_data in dataset:\n", " print(np_arr_data[0], np_arr_data[1])" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[0.89286015 0.33197981] [0.33540785]\n", "[0.82122912 0.04169663] [0.62251943]\n", "[0.10765668 0.59505206] [0.43814143]\n", "[0.52981736 0.41880743] [0.73588211]\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "- 加载Python list数据" ], "metadata": {} }, { "cell_type": "code", "execution_count": 12, "source": [ "import mindspore.dataset as ds\n", "\n", "data1 = [[1, 2], [3, 4]]\n", "\n", "dataset = ds.NumpySlicesDataset(data1, column_names=[\"col1\"], shuffle=False)\n", "\n", "for np_list_data in dataset:\n", " print(np_list_data[0])" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[1 2]\n", "[3 4]\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "- 加载Python dict数据" ], "metadata": {} }, { "cell_type": "code", "execution_count": 13, "source": [ "import mindspore.dataset as ds\n", "\n", "data1 = {\"a\": [1, 2], \"b\": [3, 4]}\n", "\n", "dataset = ds.NumpySlicesDataset(data1, column_names=[\"col1\", \"col2\"], shuffle=False)\n", "\n", "for np_dic_data in dataset.create_dict_iterator():\n", " print(np_dic_data)" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "{'col1': Tensor(shape=[], dtype=Int64, value= 1), 'col2': Tensor(shape=[], dtype=Int64, value= 3)}\n", "{'col1': Tensor(shape=[], dtype=Int64, value= 2), 'col2': Tensor(shape=[], dtype=Int64, value= 4)}\n" ] } ], "metadata": {} } ], "metadata": { "interpreter": { "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" }, "kernelspec": { "display_name": "MindSpore", "language": "python", "name": "mindspore" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" } }, "nbformat": 4, "nbformat_minor": 4 }