Function mindspore::dataset::IWSLT2016

Function Documentation

inline std::shared_ptr<IWSLT2016Dataset> mindspore::dataset::IWSLT2016(const std::string &dataset_dir, const std::string &usage = "all", const std::vector<std::string> &language_pair = {"de", "en"}, const std::string &valid_set = "tst2013", const std::string &test_set = "tst2014", int64_t num_samples = 0, ShuffleMode shuffle = ShuffleMode::kGlobal, int32_t num_shards = 1, int32_t shard_id = 0, const std::shared_ptr<DatasetCache> &cache = nullptr)

Function to create a IWSLT2016Dataset.

Note

The generated dataset has two columns [“text”, “translation”].

Parameters
  • dataset_dir[in] Path to the root directory that contains the dataset.

  • usage[in] Part of dataset of IWSLT2016, can be “train”, “valid”, “test” or “all” (default = “all”).

  • language_pair[in] List containing src and tgt language (Default = {“de”, “en”}).

  • valid_set[in] A string to identify validation set (Default = “tst2013”).

  • test_set[in] A string to identify test set (Default = “tst2014”).

  • num_samples[in] The number of samples to be included in the dataset. (Default = 0, means all samples).

  • shuffle[in] The mode for shuffling data every epoch (Default=ShuffleMode::kGlobal). Can be any of: ShuffleMode::kFalse - No shuffling is performed. ShuffleMode::kFiles - Shuffle files only. ShuffleMode::kGlobal - Shuffle both the files and samples.

  • num_shards[in] Number of shards that the dataset should be divided into (Default = 1).

  • shard_id[in] The shard ID within num_shards. This argument should be specified only when num_shards is also specified (Default = 0).

  • cache[in] Tensor cache to use (default=nullptr, which means no cache is used).

Returns

Shared pointer to the IWSLT2016Dataset.