mindspore.dataset.config.set_error_samples_mode

View Source On AtomGit
mindspore.dataset.config.set_error_samples_mode(error_samples_mode)

Set the method in which erroneous samples should be processed in a dataset pipeline.

Note

  • This error samples feature is only applicable to the Map operation in a dataset pipeline.

  • For ErrorSamplesMode.REPLACE mode, a cache of other samples will be used.

  • If ErrorSamplesMode.SKIP mode is used in a distributed setting, make sure to manually ensure the number of valid samples is the same for each shard (otherwise one may encounter hangs). One technique is to manually concat a dataset of all valid samples plus a take operation for the number of skipped erroneous samples.

Parameters:

error_samples_mode (ErrorSamplesMode) –

The method in which erroneous samples should be processed in a dataset pipeline. It can be any of [ErrorSamplesMode.RETURN, ErrorSamplesMode.REPLACE, ErrorSamplesMode.SKIP].

  • ErrorSamplesMode.RETURN: means an erroneous sample results in an error being raised and returned.

  • ErrorSamplesMode.REPLACE: means an erroneous sample is replaced with a correct sample.

  • ErrorSamplesMode.SKIP: means an erroneous sample is skipped.

Raises:

TypeError – If error_samples_mode is not of type ErrorSamplesMode.

Examples

>>> import mindspore.dataset as ds
>>> ds.config.set_error_samples_mode(ds.config.ErrorSamplesMode.SKIP)