mindspore.dataset.text.SPieceTokenizerLoadType

class mindspore.dataset.text.SPieceTokenizerLoadType[source]

An enumeration for loading type of mindspore.dataset.text.SentencePieceTokenizer .

Possible enumeration values are: SPieceTokenizerLoadType.FILE, SPieceTokenizerLoadType.MODEL.

  • SPieceTokenizerLoadType.FILE: Load SentencePiece tokenizer from a Vocab file.

  • SPieceTokenizerLoadType.MODEL: Load SentencePiece tokenizer from a SentencePieceVocab object.