Class SentencePieceTokenizer

Defined in File text.h

Inheritance Relationships

Base Type

public mindspore::dataset::TensorTransform (Class TensorTransform)

Class Documentation

class SentencePieceTokenizer : public mindspore::dataset::TensorTransform 

Tokenize a scalar token or a 1-D token to tokens by sentencepiece.

Public Functions

SentencePieceTokenizer(const std::shared_ptr<SentencePieceVocab> &vocab, mindspore::dataset::SPieceTokenizerOutType out_type)

Constructor.

Example

/* Define operations */
std::shared_ptr<Dataset> ds_vocab = TextFile({"/path/to/vocab/file"}, 0, ShuffleMode::kFalse);
std::shared_ptr<SentencePieceVocab> vocab =
    ds_vocab->BuildSentencePieceVocab({}, 0, 0.9995, SentencePieceModel::kUnigram, {});
auto tokenizer_op = text::SentencePieceTokenizer(vocab, mindspore::dataset::SPieceTokenizerOutType::kString);

/* dataset is an instance of Dataset object */
dataset = dataset->Map({tokenizer_op},   // operations
                       {"text"});        // input columns

参数

vocab – [in] a SentencePieceVocab object.
out_type – [in] The type of the output.

inline SentencePieceTokenizer(const std::string &vocab_path, mindspore::dataset::SPieceTokenizerOutType out_type)

Constructor.

Example

/* Define operations */
auto tokenizer_op = text::SentencePieceTokenizer("/path/to/model",
                                                 mindspore::dataset::SPieceTokenizerOutType::kInt);

/* dataset is an instance of Dataset object */
dataset = dataset->Map({tokenizer_op},   // operations
                       {"text"});        // input columns

参数

vocab_path – [in] vocab model file path.
out_type – [in] The type of the output.

SentencePieceTokenizer(const std::vector<char> &vocab_path, mindspore::dataset::SPieceTokenizerOutType out_type)

Constructor.

参数

vocab_path – [in] vocab model file path. type should be char of vector.
out_type – [in] The type of the output.

~SentencePieceTokenizer() = default: Destructor.