[{"data":1,"prerenderedAt":214},["ShallowReactive",2],{"content-query-yNWMh8SLNr":3},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"cover":11,"type":12,"body":13,"_type":208,"_id":209,"_source":210,"_file":211,"_stem":212,"_extension":213},"/technology-blogs/en/2974","en",false,"","Idea Sharing | MindSpore Improves the Crystal Material Property Prediction Using Periodic Graph Transformer","November 28, 2023","2023-11-28","https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/02/04/78972c3420df4f8f8803d25fdb1cc890.png","technology-blogs",{"type":14,"children":15,"toc":205},"root",[16,24,34,39,44,52,57,62,67,72,80,85,90,95,100,108,113,118,125,130,135,140,147,152,157,164,169,177,182,190,195,200],{"type":17,"tag":18,"props":19,"children":21},"element","h1",{"id":20},"idea-sharing-mindspore-improves-the-crystal-material-property-prediction-using-periodic-graph-transformer",[22],{"type":23,"value":8},"text",{"type":17,"tag":25,"props":26,"children":27},"p",{},[28],{"type":17,"tag":29,"props":30,"children":31},"strong",{},[32],{"type":23,"value":33},"Background",{"type":17,"tag":25,"props":35,"children":36},{},[37],{"type":23,"value":38},"Crystalline materials are a family of compounds formed by atom periodic repetitions in 3D space. Crystal materials are the basis materials for many industrial applications, such as semiconductor electronics, solar cells, and chemical cells. By predicting material properties (for example, formation energy) and designing new materials with desired properties, Materials science meets the enormous demands of various industries. But its development is limited by expensive experiments or time-consuming material simulations. With the success of AI, people attempt to apply these AI models to crystal material prediction. Unlike molecules, there are complex repeated units and atom periodic arrangement in the atomic arrangement in crystal materials. Developing effective AI models requires explicit symmetry information in these structures.",{"type":17,"tag":25,"props":40,"children":41},{},[42],{"type":23,"value":43},"A crystal architecture can be represented by a lattice vector, L=(L1, L2, L3), and a unit cell, (Ai, Pi). L represents a lattice repetition period in 3D space, a cell is a minimum repetition unit of the crystal, Ai is the type of the _i_th atom, and Pi is the coordinate position. When the crystal architecture is represented by a graph structure, periodic invariance is required, that is, the learned graph representation keeps unchanged for the scale expansion of the minimum repeatable unit and the periodic boundary change.",{"type":17,"tag":25,"props":45,"children":46},{},[47],{"type":17,"tag":29,"props":48,"children":49},{},[50],{"type":23,"value":51},"1. CGCNN",{"type":17,"tag":25,"props":53,"children":54},{},[55],{"type":23,"value":56},"CGCNN is an early model that introduces periodicity in the crystal graph representation. See figure 1 for its constructed graph representation.",{"type":17,"tag":25,"props":58,"children":59},{},[60],{"type":23,"value":61},"(1) The number of nodes in the graph is equal to the number of atoms in the crystal cell (the atoms at the edge of the cell are evenly divided according to the number of cells to which the atoms belong).",{"type":17,"tag":25,"props":63,"children":64},{},[65],{"type":23,"value":66},"(2) Atoms that are periodically repeated correspond to the same node in the graph.",{"type":17,"tag":25,"props":68,"children":69},{},[70],{"type":23,"value":71},"(3) The connection between nodes consists of the connection between atoms and all atoms within the truncated radius. Because an atom can be connected to the same atom in multiple cells that are near, these atoms are represented by the same node in the crystal graph, and multiple different edges may exist between the nodes.",{"type":17,"tag":25,"props":73,"children":74},{},[75],{"type":17,"tag":76,"props":77,"children":79},"img",{"alt":7,"src":78},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/02/04/eb4344f0e22d4aa0af90f1f9f3d901e3.png",[],{"type":17,"tag":25,"props":81,"children":82},{},[83],{"type":23,"value":84},"Figure 1 CGCNN constructed graph representation",{"type":17,"tag":25,"props":86,"children":87},{},[88],{"type":23,"value":89},"The advantage of the representation is that the same crystal graph can be obtained when the edges of the cells are different, which ensures the periodic invariance. However, there are drawbacks in the CGCNN paper too:",{"type":17,"tag":25,"props":91,"children":92},{},[93],{"type":23,"value":94},"(1) There is no explicit periodic information. The same microstructure but different repetition periods may indicate different crystals.",{"type":17,"tag":25,"props":96,"children":97},{},[98],{"type":23,"value":99},"(2) In specific implementation, nearest neighboring atoms of number n are used to construct a connection for each atom. When there are different atoms at the same distance, periodic invariance may be violated.",{"type":17,"tag":25,"props":101,"children":102},{},[103],{"type":17,"tag":29,"props":104,"children":105},{},[106],{"type":23,"value":107},"2. Matformer",{"type":17,"tag":25,"props":109,"children":110},{},[111],{"type":23,"value":112},"In response to these shortcomings, the paper about Matformer published in 2022 made improvements. In the paper, periodicity is explicitly added to the crystal graph representation, and the attention structure is used to convey the graph message. The crystal property prediction is conducted on multiple datasets such as Materials Project and JARVIS with good results obtained.",{"type":17,"tag":25,"props":114,"children":115},{},[116],{"type":23,"value":117},"Matformer uses a method similar to that of CGCNN to construct a multi-edge graph. The edge connection of the nearest neighbor n is replaced with the method in which if less than the minimum distance r, an edge exists. This ensures periodic invariance. In figure 2, (a) shows the change of graph representation along with the change of periodic boundaries, when multi-edge graph is not used.",{"type":17,"tag":25,"props":119,"children":120},{},[121],{"type":17,"tag":76,"props":122,"children":124},{"alt":7,"src":123},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/02/04/9440a50c94c14679976460bf69daeb4d.png",[],{"type":17,"tag":25,"props":126,"children":127},{},[128],{"type":23,"value":129},"Figure 2 (a) Graph representation that satisfies, or does not satisfy periodic invariance when changing periodic boundaries; (b) Graph representation with explicit addition of periodic representation.",{"type":17,"tag":25,"props":131,"children":132},{},[133],{"type":23,"value":134},"The lattice vector is then explicitly added to the graph representation through self-connecting edges. As shown in Figure 2 (b), the atom is connected and encoded with its adjacent replications in three lattices. Because using angles to represent spatial information is more complex, three diagonals on the lattice surface are additionally used to implicitly represent an angle relationship of a lattice vector.",{"type":17,"tag":25,"props":136,"children":137},{},[138],{"type":23,"value":139},"Figure 3 shows the architecture of Matformer. The node information represented by the atom sequence number and the edge information represented by the distance between atoms goes to the multiple Matformer layers after being encoded. Based on the attention architecture, each Matformer layer each aggregates information about nodes and edges connected to each node and updates the node. For a multi-edge graph, each edge corresponds to an attention head. In the figure, h indicates the edge sequence number. When K and V in attention are generated, edge features and node features are combined. After Q and K are multiplied, sigmoid is used to replace softmax to calculate the weight. In this way, the impact of different degrees (number of connections) of nodes can be considered, and message aggregation and communication between adjacent nodes are reduced, improving calculation efficiency. The ablation experiment also shows that Sigmoid can get better test precision.",{"type":17,"tag":25,"props":141,"children":142},{},[143],{"type":17,"tag":76,"props":144,"children":146},{"alt":7,"src":145},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/02/04/7d63d376fd384989aed416bc3ea8c6d8.png",[],{"type":17,"tag":25,"props":148,"children":149},{},[150],{"type":23,"value":151},"Figure 3 (a) Overall architecture of Matformer; (b) Architecture of Matformer Layer; (c) Ablation experiment of replacing Sigmoid with Softmax",{"type":17,"tag":25,"props":153,"children":154},{},[155],{"type":23,"value":156},"Matformer shows advantages in predicting multiple attributes on the Materials Project and JARVIS datasets. This is because of the introduction of its explicit periodicity and the design of the attention structure for multi-edge graphs. In terms of computational efficiency, Matformer is much higher than ALIGNN. Figure 4 shows the comparison of prediction precision and calculation efficiency between different models.",{"type":17,"tag":25,"props":158,"children":159},{},[160],{"type":17,"tag":76,"props":161,"children":163},{"alt":7,"src":162},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/02/04/2ae6de64ebc94539a6da7020efa10757.png",[],{"type":17,"tag":25,"props":165,"children":166},{},[167],{"type":23,"value":168},"Figure 4 Comparison of prediction precision and calculation efficiency between Matformer and ALIGNN",{"type":17,"tag":25,"props":170,"children":171},{},[172],{"type":17,"tag":29,"props":173,"children":174},{},[175],{"type":23,"value":176},"3. Summary",{"type":17,"tag":25,"props":178,"children":179},{},[180],{"type":23,"value":181},"Through the analysis of this blog, it can be seen that the addition of prior physical knowledge, such as periodic invariance and periodic repetition, can greatly improve the model predictions on the crystal material property. At the same time, the design of the attention structure for the crystal multi-edge graph can well capture the features of the crystal structure and optimize the computational efficiency. Currently, based on its advantages, the MindSpore AI framework is carrying out research and development of periodic graph transformer to improve crystal property prediction and computing efficiency. Let's look forward to it.",{"type":17,"tag":25,"props":183,"children":184},{},[185],{"type":17,"tag":29,"props":186,"children":187},{},[188],{"type":23,"value":189},"References",{"type":17,"tag":25,"props":191,"children":192},{},[193],{"type":23,"value":194},"[1] Keith T Butler, Daniel W Davies, Hugh Cartwright, Olexandr Isayev, and Aron Walsh. 2018. Machine learning for molecular and materials science. Nature 559, 7715 (2018), 547–555.",{"type":17,"tag":25,"props":196,"children":197},{},[198],{"type":23,"value":199},"[2] Tian Xie and Jeffrey C Grossman. 2018. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical Review Letters 120, 14 (2018), 145301.",{"type":17,"tag":25,"props":201,"children":202},{},[203],{"type":23,"value":204},"[3] Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. 2022. Periodic Graph Transformers for Crystal Material Property Prediction. In The 36th Annual Conference on Neural Information Processing Systems. 15066–15080.",{"title":7,"searchDepth":206,"depth":206,"links":207},4,[],"markdown","content:technology-blogs:en:2974.md","content","technology-blogs/en/2974.md","technology-blogs/en/2974","md",1776506108390]