[{"data":1,"prerenderedAt":478},["ShallowReactive",2],{"content-doc-\u002Ftechnology-blogs\u002Fzh\u002F2026-11-17-1":3},{"id":4,"title":5,"address":6,"body":7,"category":466,"cover":467,"date":468,"description":16,"end_date":6,"extension":469,"href":6,"isTop":470,"meta":471,"navigation":472,"path":473,"seo":474,"series":6,"start_date":6,"state":6,"stem":475,"type":476,"working":6,"__hash__":477},"technologyBlogs\u002Ftechnology-blogs\u002Fzh\u002F2026-11-17-1.md","昇思 HyperParallel DCP 关键技术拆解：异步持久化、细粒度去冗余、跨切片加载、分片广播",null,{"type":8,"value":9,"toc":458},"minimark",[10,14,17,20,25,35,39,44,47,52,57,60,65,68,73,76,86,89,95,98,102,107,110,113,182,185,189,194,197,232,235,239,242,297,300,340,345,348,352,356,359,364,367,371,374,387,391,394,399,402,405,408,411,414,417,420,424,427,430,433,436,439,442,446,449,454],[11,12,13],"p",{},"千亿至万亿参数模型训练已进入万卡规模，单次 checkpoint 可达数千 GB，断点续训效率成为长稳训练关键瓶颈。传统 checkpoint 存在四大成本：存盘阻塞、落盘冗余、读盘重复、并行策略转换。",[11,15,16],{},"昇思 HyperParallel DCP（Distributed Checkpoint） 将检查点重构为\"规划（Planner）—存储（Storage）—元数据（Metadata）\"三层解耦架构，构建四大关键技术解决以上四大成本，实现零冗余落盘、零阻塞存盘、零重复读盘、零离线转换。",[11,18,19],{},"本文聚焦原理分析，重点阐述四大关键技术。至于接口如何调用、断点续训如何配置等相关操作详见本系列后续实操篇。",[21,22,24],"h1",{"id":23},"_01-传统-checkpoint-四大成本","01 传统 checkpoint 四大成本",[26,27,29],"div",{"style":28},"text-align: center;",[30,31],"img",{"src":32,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F1.jpg","display: block;margin: 0 auto;max-width:60%","",[21,36,38],{"id":37},"_02-hyperparallel-dcp-四大技术","02 HyperParallel DCP 四大技术",[26,40,41],{"style":28},[30,42],{"src":43,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F2.jpg",[11,45,46],{},"传统方案类似于“先拼回完整图像，再重新切分”：保存时要先把各卡分片合并成完整张量，加载时再按新并行策略重新切分。DCP 每张卡仅保存自身部分分片，并附加一份全局元数据（.metadata，相当于一张“座位表”），记录每个分片在完整张量中的坐标位置，该元数据即全局契约。并行策略变化时，无需恢复完整张量，只需根据新分片区域查询元数据求交。",[26,48,49],{"style":28},[30,50],{"src":51,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F3.jpg",[53,54,56],"h2",{"id":55},"_21-异步持久化同步-staging-异步落盘","2.1 异步持久化：同步 staging + 异步落盘",[11,58,59],{},"build_staged_state_dict 将权重D2H拷贝到Host，之后构建plan、集群协同、写盘全交给子进程。在 async_save返回后，训练就可以继续执行。",[26,61,62],{"style":28},[30,63],{"src":64,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F4.jpg",[11,66,67],{},"难点在于子进程做集群协同对齐plan，训练侧HCCL\u002FNCCL 通信域活在 C++ 全局状态里，fork 之后就失效了，子进程不能直接复用用。HyperParallel DCP 为此给了两种协同策略，按环境二选一：",[26,69,70],{"style":28},[30,71],{"src":72,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F5.jpg",[11,74,75],{},"第二种尤其实用：集群本来就有共享文件系统，不必再为 checkpoint 单开一个通信域。在每个 pkl 写完后末尾追加 _COMPLETE_FLAG 标记，读端只认带标记的文件，没有标记就先跳过去读下一个，然后轮询重试，从而保证读到文件都是完整可靠的。",[77,78,83],"pre",{"className":79,"code":81,"language":82},[80],"language-text","# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Fapi.py:223\nuse_gloo=True         → 重建独立的 CPU gloo 通信域\nuse_gloo=False        → 通过共享文件存储交换 plan 与 write result\nuse_collectives=False → 降级选项：完全不跨卡，每 rank 各写各的 {rank}.metadata\n","text",[84,85,81],"code",{"__ignoreMap":34},[11,87,88],{},"子进程还会把 plan 缓存带回主进程。规划结果存在 StandardSavePlanner.cached_save_result 这个类级字典里，可算出它的是子进程，子进程一退出，缓存就跟着没了，下一次 async_save 又得从头 all_gather 一遍。DCP 的做法是让子进程写完盘后，把缓存连同 metadata 一起塞回 result queue，由主进程的 join 线程 merge 进自己的缓存：",[77,90,93],{"className":91,"code":92,"language":82},[80],"# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Fasync_persist.py:389 \u002F 437   子进程侧\nresult_queue.put((AsyncPersistStatus.SUCCESS, (meta, StandardSavePlanner.cached_save_result)))\n\n# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Fasync_persist.py:480        主进程的 join 线程\nStandardSavePlanner.cached_save_result.update(payload[1])\n",[84,94,92],{"__ignoreMap":34},[11,96,97],{},"两条协同策略（gloo \u002F 共享文件存储）都会回传。下一次  async_save fork 出的子进程，继承到的就是这份已经填好的缓存。这意味着同结构的checkpoint 只在第一次做规划，之后每次异步\u002F同步保存都复用缓存。",[53,99,101],{"id":100},"_22-细粒度去冗余分片级去重与负载均衡","2.2 细粒度去冗余：分片级去重与负载均衡",[26,103,104],{"style":28},[30,105],{"src":106,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F6.jpg",[11,108,109],{},"去重的粒度不是\"张量\"而是分片：MetadataIndex(fqn, offsets) 唯一标识\"哪个参数的哪一块\"，坐标相同的分片按定义就是同一份副本，无论被多少张rank持有，全局只有一个 rank 会写它。DP 复制维度、TP 上的 Replicate()参数、fully_shard 的重叠副本，都只会落盘一份。从而保证checkpoint的体积等于模型参数量本身，而不是参数量 × 复制倍数。",[11,111,112],{},"同时去冗余不止是简单丢掉多余副本，它也是一次写侧的负载均衡：",[77,114,118],{"className":115,"code":116,"language":117,"meta":34,"style":34},"language-python shiki shiki-themes github-light github-dark monokai","# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Futil.py:216\nfor item_key, containing_plans in multi_plan_items:\n    if save_to_minimum_rank:\n            target_plan = min(containing_plans)\n    else:\n         target_plan = min(containing_plans, key=lambda p_idx: storage_sizes[p_idx])    \n    entry = item_registry[item_key]\n    storage_sizes[target_plan] += entry.tensor_storage_size() or 1    \n    for p_idx in containing_plans - {target_plan}:     # 其余 plan 丢掉这个副本\n        remaining_items[p_idx].discard(item_key)\n","python",[84,119,120,128,134,140,146,152,158,164,170,176],{"__ignoreMap":34},[121,122,125],"span",{"class":123,"line":124},"line",1,[121,126,127],{},"# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Futil.py:216\n",[121,129,131],{"class":123,"line":130},2,[121,132,133],{},"for item_key, containing_plans in multi_plan_items:\n",[121,135,137],{"class":123,"line":136},3,[121,138,139],{},"    if save_to_minimum_rank:\n",[121,141,143],{"class":123,"line":142},4,[121,144,145],{},"            target_plan = min(containing_plans)\n",[121,147,149],{"class":123,"line":148},5,[121,150,151],{},"    else:\n",[121,153,155],{"class":123,"line":154},6,[121,156,157],{},"         target_plan = min(containing_plans, key=lambda p_idx: storage_sizes[p_idx])    \n",[121,159,161],{"class":123,"line":160},7,[121,162,163],{},"    entry = item_registry[item_key]\n",[121,165,167],{"class":123,"line":166},8,[121,168,169],{},"    storage_sizes[target_plan] += entry.tensor_storage_size() or 1    \n",[121,171,173],{"class":123,"line":172},9,[121,174,175],{},"    for p_idx in containing_plans - {target_plan}:     # 其余 plan 丢掉这个副本\n",[121,177,179],{"class":123,"line":178},10,[121,180,181],{},"        remaining_items[p_idx].discard(item_key)\n",[11,183,184],{},"先把\"只有一个 rank 持有\"的分片入账，再把有副本的分片派给当前写入量最小的那个 rank 。如果一律交给最小号rank，副本都会堆到 DP0 rank上，整体写盘时间就会被这部分卡拖住。",[53,186,188],{"id":187},"_23-分片广播最小-rank-读-组内广播","2.3 分片广播：最小 rank 读 + 组内广播",[26,190,191],{"style":28},[30,192],{"src":193,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F7.jpg",[11,195,196],{},"去冗余在读侧有一个对偶操作：",[77,198,200],{"className":115,"code":199,"language":117,"meta":34,"style":34},"# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Fstandard_planner.py:492\ngroup_ranks = infer_same_shard_ranks_for_dtensor(tensor)   # 由 mesh_shape + tensor_map 推同分片组\nload_rank = min(group_ranks)\nif len(group_ranks) > 1:\n    setattr(tensor, BROADCAST_INFO, BroadcastInfo(group_ranks, load_rank))\n    return self.rank == load_rank\n",[84,201,202,207,212,217,222,227],{"__ignoreMap":34},[121,203,204],{"class":123,"line":124},[121,205,206],{},"# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Fstandard_planner.py:492\n",[121,208,209],{"class":123,"line":130},[121,210,211],{},"group_ranks = infer_same_shard_ranks_for_dtensor(tensor)   # 由 mesh_shape + tensor_map 推同分片组\n",[121,213,214],{"class":123,"line":136},[121,215,216],{},"load_rank = min(group_ranks)\n",[121,218,219],{"class":123,"line":142},[121,220,221],{},"if len(group_ranks) > 1:\n",[121,223,224],{"class":123,"line":148},[121,225,226],{},"    setattr(tensor, BROADCAST_INFO, BroadcastInfo(group_ranks, load_rank))\n",[121,228,229],{"class":123,"line":154},[121,230,231],{},"    return self.rank == load_rank\n",[11,233,234],{},"DP=4 的复制维度上，磁盘读量从 4× 降到 1×，由最小号rank从磁盘读，然后广播给组内其他3张rank，用D2D高速通信替换存储IO。",[53,236,238],{"id":237},"_24-跨切片加载n-维区间求交","2.4 跨切片加载：N 维区间求交",[11,240,241],{},"\"拿着新区域去查座位表\"：每个张量分片都抽象成 N 维半开区间 [start, end)，查表即求交，两个区间交集就是要搬运的那片。load时重切分都归结到这里。",[77,243,245],{"className":115,"code":244,"language":117,"meta":34,"style":34},"# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Freshard.py:88\ndef infer_intersection(area_a, area_b):\n    intersection = []\n    for axis_range_a, axis_range_b in zip(area_a, area_b):    \n        left = max(axis_range_a[0], axis_range_b[0])    \n        right = min(axis_range_a[1], axis_range_b[1])     \n        if left >= right:          # 任一维不相交 → 整体不相交     \n               return None        \n        intersection.append((left, right))    \n    return tuple(intersection)\n",[84,246,247,252,257,262,267,272,277,282,287,292],{"__ignoreMap":34},[121,248,249],{"class":123,"line":124},[121,250,251],{},"# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Freshard.py:88\n",[121,253,254],{"class":123,"line":130},[121,255,256],{},"def infer_intersection(area_a, area_b):\n",[121,258,259],{"class":123,"line":136},[121,260,261],{},"    intersection = []\n",[121,263,264],{"class":123,"line":142},[121,265,266],{},"    for axis_range_a, axis_range_b in zip(area_a, area_b):    \n",[121,268,269],{"class":123,"line":148},[121,270,271],{},"        left = max(axis_range_a[0], axis_range_b[0])    \n",[121,273,274],{"class":123,"line":154},[121,275,276],{},"        right = min(axis_range_a[1], axis_range_b[1])     \n",[121,278,279],{"class":123,"line":160},[121,280,281],{},"        if left >= right:          # 任一维不相交 → 整体不相交     \n",[121,283,284],{"class":123,"line":166},[121,285,286],{},"               return None        \n",[121,288,289],{"class":123,"line":172},[121,290,291],{},"        intersection.append((left, right))    \n",[121,293,294],{"class":123,"line":178},[121,295,296],{},"    return tuple(intersection)\n",[11,298,299],{},"\"我要的\"与\"磁盘上存的\"求交，算出的偏移量直接变成读请求：",[77,301,303],{"className":115,"code":302,"language":117,"meta":34,"style":34},"# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Fstandard_planner.py:396\noverlap = infer_intersection(local_area, saved_area)\nif overlap is None:\n    continue\ndest_offsets    = tuple(overlap[i][0] - local_chunk.offsets[i]   for i in range(len(overlap)))\nstorage_offsets = tuple(overlap[i][0] - storage_chunk.offsets[i] for i in range(len(overlap)))\nlengths         = tuple(overlap[i][1] - overlap[i][0]            for i in range(len(overlap)))\n",[84,304,305,310,315,320,325,330,335],{"__ignoreMap":34},[121,306,307],{"class":123,"line":124},[121,308,309],{},"# hyper_parallel\u002Fcore\u002Fdistributed_checkpoint\u002Fstandard_planner.py:396\n",[121,311,312],{"class":123,"line":130},[121,313,314],{},"overlap = infer_intersection(local_area, saved_area)\n",[121,316,317],{"class":123,"line":136},[121,318,319],{},"if overlap is None:\n",[121,321,322],{"class":123,"line":142},[121,323,324],{},"    continue\n",[121,326,327],{"class":123,"line":148},[121,328,329],{},"dest_offsets    = tuple(overlap[i][0] - local_chunk.offsets[i]   for i in range(len(overlap)))\n",[121,331,332],{"class":123,"line":154},[121,333,334],{},"storage_offsets = tuple(overlap[i][0] - storage_chunk.offsets[i] for i in range(len(overlap)))\n",[121,336,337],{"class":123,"line":160},[121,338,339],{},"lengths         = tuple(overlap[i][1] - overlap[i][0]            for i in range(len(overlap)))\n",[26,341,342],{"style":28},[30,343],{"src":344,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F8.jpg",[11,346,347],{},"save 端只记录每片全局坐标，load 端只关心\"我的新区域压到了哪些旧片\"。TP8 存、TP4 读，或者 fully_shard 的分片换一个 mesh，走的都是这套设计逻辑。\n除以上四大技术外，HyperParallel DCP还提供 huggingface safetensors ⇄ DCP 离线互转配套工具（offline_transform），用于冷启动加载开源权重、训练结束后导出发布。具体用法见本系列实操篇。",[21,349,351],{"id":350},"_03-万卡集群分钟级恢复","03 万卡集群分钟级恢复",[53,353,355],{"id":354},"_31-恢复时间不随卡数线性增长","3.1 恢复时间不随卡数线性增长",[11,357,358],{},"传统checkpoint 恢复耗时有几项随集群规模和拓扑变更成倍放大，DCP逐个消掉：",[26,360,361],{"style":28},[30,362],{"src":363,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F9.jpg",[11,365,366],{},"于是恢复耗时的构成从 O(参数量 × R) \u002F 存储带宽 + 离线转换作业，收敛成 O(参数量) \u002F 存储聚合带宽 + 一次组内广播——读取只与模型大小和存储带宽有关，广播随卡数轻微增长。",[53,368,370],{"id":369},"_32-万卡集群实测","3.2 万卡集群实测",[11,372,373],{},"在盘古 505B 模型万卡集群训练场景下，实测 HyperParallel DCP：",[375,376,377,381,384],"ul",{},[378,379,380],"li",{},"异步保存性能：5s；",[378,382,383],{},"断点续训状态恢复耗时：290s；",[378,385,386],{},"当前持续优化中，预计恢复耗时可进一步压缩至 3 分钟内。",[21,388,390],{"id":389},"_04-业界方案对比","04 业界方案对比",[11,392,393],{},"这套三层解耦思路业界也有类似方案，PyTorch 的 torch.distributed.checkpoint 、Megatron-LM 的 dist_checkpointing 、字节的 ByteCheckpoint 都基于这条路线构建。",[26,395,396],{"style":28},[30,397],{"src":398,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F10.jpg",[11,400,401],{},"HyperParallel DCP 三项核心竞争力",[11,403,404],{},"内置广播能力：：",[11,406,407],{},"HyperParallel DCP 根据 DTensor本地推导同一分片副本rank，只让其中一张rank读盘，再广播给其余副本。对照之下：PyTorch DCP 未做读侧去重；Megatron 必须先 all_gather 分片元数据才能去重；ByteCheckpoint 也还未实现该能力。",[11,409,410],{},"异步落盘整链路搬离主进程：",[11,412,413],{},"HyperParallel DCP 把规划、通信、写盘都交给子进程，并提供两种跨卡协同策略：重建CPU gloo 组和共享文件系统通信。大集群本来就有共享存储，选后者就不必为 checkpoint 另开通信域。PyTorch DCP子进程只支持重建CPU gloo；Megatron 子进程只负责写盘，集合通信与 .metadata 落盘退回主进程的 finalize；ByteCheckpoint 则在主进程完成规划。",[11,415,416],{},"一份实现覆盖双框架多硬件：",[11,418,419],{},"HyperParallel DCP 架构不绑定框架和硬件，torch\u002FMindSpore + NPU\u002FGPU 共用。业界其他架构主要支持torch + GPU。",[21,421,423],{"id":422},"_05-结语","05 结语",[11,425,426],{},"DCP 是 HyperParallel\"故障快速恢复\"这条主线的第一块拼图，它先把\"存得快、存得省、读得省、切分可变\"做扎实。后续故障快恢、临终遗言、SDC 检测等都会基于这套架构构建，只要每片数据全局坐标是可描述的，恢复就是一次区间求交。",[11,428,429],{},"DCP 后续规划集中在以下两个关键事项上。",[11,431,432],{},"一、极致性能",[11,434,435],{},"存侧性能已近乎极致，读侧还有优化空间。其一是读侧负载均衡：同一分片有多个副本 rank 时，\"谁去读\"可以按rank读取量来均衡；其二是读取和广播异步流水化：读完的分片立刻广播，与后面分片的读盘重叠。这两步做完，恢复耗时还能再压一个台阶。",[11,437,438],{},"二、独立交付：",[11,440,441],{},"DCP 是一个相对独立模块，不与 HyperParallel 并行实现绑定，只依赖\"张量分片报出自己全局坐标\"这一个约定。因此，上层不管是 HyperParallel 还是其他基于torch生态构建的加速方案，都可以接入HyperParallel DCP，享受关键技术优势。",[21,443,445],{"id":444},"_06-欢迎加入-hyperparallel","06 欢迎加入 HyperParallel",[11,447,448],{},"我们诚挚邀请各位开发者、研究者加入HyperParallel。无论是贡献代码、完善文档，还是提出改进建议，您的参与都将推动大模型分布式并行技术的边界。让我们一起，让大模型训练更简单、更快速、更智能！",[26,450,451],{"style":28},[30,452],{"src":453,"style":33,"alt":34},"\u002Fcategory\u002Finformation\u002Ftechnology-blogs\u002Fbanner\u002F2026-11-17-1\u002F11.jpg",[455,456,457],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}",{"title":34,"searchDepth":142,"depth":142,"links":459},[460,461,462,463,464,465],{"id":55,"depth":130,"text":56},{"id":100,"depth":130,"text":101},{"id":187,"depth":130,"text":188},{"id":237,"depth":130,"text":238},{"id":354,"depth":130,"text":355},{"id":369,"depth":130,"text":370},"技术解读","https:\u002F\u002Fobs-mindspore-file.obs.cn-north-4.myhuaweicloud.com\u002Ffile\u002F2024\u002F11\u002F28\u002F8e0e0150508a4c5ba4287fa3bec8ea3f.png","2026-9-23","md",false,{},true,"\u002Ftechnology-blogs\u002Fzh\u002F2026-11-17-1",{"title":5,"description":16},"technology-blogs\u002Fzh\u002F2026-11-17-1","technology-blogs","98Hz9GJNtRe2yg_HGC4hHHGtyQ6fIy6paY8Iuhcq0rs",1795123617239]