mindspore.dataset.dataloader.IterableDataset

class mindspore.dataset.dataloader.IterableDataset

Base class for implementing iterable datasets.

Iterable style datasets should inherit from it.

Iterable style datasets represent an iterable over data samples. It is particularly useful when random reads are expensive or even improbable. Subclasses must overwrite __iter__() method, returning an iterator of samples over the dataset.

Examples

>>> from mindspore.dataset.dataloader import IterableDataset, get_worker_info
>>>
>>> class IterableStyleDataset(IterableDataset):
...     def __init__(self, num_samples):
...         self.start = 0
...         self.end = num_samples
...
...     def __iter__(self):
...         worker_info = get_worker_info()
...         if worker_info is None:
...             return iter(range(self.start, self.end))
...         else:
...             worker_id = worker_info.id
...             num_workers = worker_info.num_workers
...             return iter(range(self.start + worker_id, self.end, num_workers))