mindspore.ckpt_to_safetensors

mindspore.ckpt_to_safetensors(file_path, save_path=None, name_map=None, file_name_regex=None, processes_num=1)[source]

Converts MindSpore checkpoint files into safetensors format and saves them to save_path. Safetensors is a reliable and portable machine learning model storage format introduced by Huggingface, used for securely storing Tensors with fast speed (zero copy).

Note

The number of multiprocess settings should be set according to the host size, and it is not recommended to set it too large, otherwise it may cause freezing. The safetensors format does not support the encryption verification function. If the checkpoint file has encryption verification enabled, an error will be generated when performing the conversion. The safetensors format currently does not support the crc verification function. If the checkpoint file contains crc verification information, the crc verification information will be lost after conversion to safetensors.

Parameters

file_path (str) – Path to the directory containing checkpoint files or a single checkpoint file (.ckpt).
save_path (str, optional) – Directory path where safetensors files will be saved. Default: None.
name_map (dict, optional) – Dictionary mapping original parameter names to new names. Default: None.
file_name_regex (str, optional) – Regular expression used to match the file that needs to be converted. Default: None.
processes_num (int, optional) – Number of processes to use for parallel processing. Default: 1.

Raises

ValueError – If the input path is invalid or the save_path is not a directory, or the file_path does not end with '.ckpt'.

Supported Platforms:: Ascend GPU CPU

Examples

>>> import mindspore as ms
>>> ms.ckpt_to_safetensors("./ckpt_save_path")
>>> ms.ckpt_to_safetensors("./ckpt_save_path/rank0/checkpoint_0.ckpt")
>>> ms.ckpt_to_safetensors(file_path="./ckpt_save_path/rank0/checkpoint_0.ckpt", save_path="./new_path/")
>>> namemap = {"lin.weight":"new_name"}
>>> ms.ckpt_to_safetensors("./ckpt_save_path/rank0/checkpoint_0.ckpt", "./new_path/", namemap)