【MIIT program】Add DeepH model integration - #289
Open
Belugaaaa wants to merge 1 commit into
Open
Conversation
|
Thanks for your contribution! |
Collaborator
|
#258 请参考该文档 |
Author
|
已根据 issue #258 的要求完成更新。
主要更新：
烦请再次审阅，谢谢！ |
leeleolay
requested changes
Jun 24, 2026
| np.asarray(structure.frac_coords), | ||
| dtype=torch.get_default_dtype(), | ||
| ) | ||
| numbers = torch.tensor(structure.atomic_numbers) |
Collaborator
There was a problem hiding this comment.
使用已有的predictor trainer等
Comment on lines
+31
to
+41
| DeepH status note: | ||
|
|
||
| - `ppmat/models/deeph` | ||
| - `ppmat/datasets/deeph_dataset.py` | ||
| - `ppmat/datasets/collate_fn.py::DeepHCollator` | ||
| - `electronic_structure/configs/deeph/deeph_graphene.yaml` | ||
|
|
||
| These pieces already allow DeepH to enter the standard PaddleMaterials | ||
| `build_dataloader -> build_model -> BaseTrainer` path. Predictor / sampler | ||
| entries are available, while compiler benchmarking still needs final GPU timing | ||
| numbers for the MIIT acceptance material. |
| ) | ||
|
|
||
|
|
||
| class DeepHCollator: |
Collaborator
There was a problem hiding this comment.
默认的不可以使用吗？兼容默认的collator
| if isinstance(lattice, list): | ||
| lattice = np.asarray(lattice) | ||
| lengths, angles = lattices_to_params_shape_numpy(lattice) | ||
| lengths, angles = lattices_to_params_shape_numpy(lattice) |
Comment on lines
+112
to
+115
| if jAtoms is None: | ||
| raise ModuleNotFoundError( | ||
| "BuildStructure(format='jarvis') requires the optional 'jarvis' package." | ||
| ) |
Comment on lines
+137
to
+157
| if p_map is not None: | ||
| canonical_crystal = p_map( | ||
| BuildStructure.build_one, | ||
| crystals_data, | ||
| [self.format] * len(crystals_data), | ||
| [self.primitive] * len(crystals_data), | ||
| [self.niggli] * len(crystals_data), | ||
| [self.canocial] * len(crystals_data), | ||
| num_cpus=self.num_cpus, | ||
| ) | ||
| else: | ||
| canonical_crystal = [ | ||
| BuildStructure.build_one( | ||
| crystal_data, | ||
| self.format, | ||
| self.primitive, | ||
| self.niggli, | ||
| self.canocial, | ||
| ) | ||
| for crystal_data in crystals_data | ||
| ] |
| self.primitive, | ||
| self.niggli, | ||
| self.canocial, | ||
| num_cpus=self.num_cpus, |
Collaborator
There was a problem hiding this comment.
参考已有的dataset格式，需要重新整理且使用已有的实现，保持实现逻辑的一致
Author
|
已根据意见进一步调整，烦请再次审阅，谢谢！ |
Author
|
已移除数据处理和图构建中的 torch 依赖，麻烦再次审阅，谢谢！ |
leeleolay
reviewed
Jun 25, 2026
Comment on lines
+3
to
+7
| ## Task | ||
|
|
||
| DeepH is integrated into the PaddleMaterials machine-learning electronic | ||
| structure (MLES) task. It predicts edge-level Hamiltonian matrix elements on | ||
| crystal graphs and complements electron-density-oriented MLES models. |
| structure (MLES) task. It predicts edge-level Hamiltonian matrix elements on | ||
| crystal graphs and complements electron-density-oriented MLES models. | ||
|
|
||
| ## Model and Dataset |
Comment on lines
+24
to
+32
| ## Environment | ||
|
|
||
| Use an official PaddlePaddle release that is compatible with PaddleMaterials. | ||
| This integration is intended for PaddlePaddle `3.2.2` or later and does not | ||
| intentionally rely on develop-only Paddle APIs. For AI Studio reproduction, use | ||
| PaddlePaddle `3.2.2` or later and the TianShu hardware option. | ||
|
|
||
| Additional Python dependencies are the standard PaddleMaterials dependencies. | ||
|
|
Comment on lines
+33
to
+61
| ## Data Preparation | ||
|
|
||
| The `graphene` example expects a DeepH data package with the following entry | ||
| file: | ||
|
|
||
| ```text | ||
| ./data/deeph/graphene/config.ini | ||
| ``` | ||
|
|
||
| The `config.ini` file should point to the DeepH processed raw data directory and | ||
| graph cache directory. The processed structure folders are expected to contain | ||
| DeepH files such as: | ||
|
|
||
| ```text | ||
| lat.dat | ||
| site_positions.dat | ||
| element.dat | ||
| rc.npz | ||
| ``` | ||
|
|
||
| For this PR, dataset files, pretrained weights, logs, and alignment evidence are | ||
| being provided to reviewers for official BCE link backfill before merge. | ||
|
|
||
| Dataset split follows the DeepH config ratios used by the graphene baseline: | ||
|
|
||
| - train ratio: `0.6` | ||
| - validation ratio: `0.2` | ||
| - test ratio: `0.2` | ||
| - split seed: `42` |
Comment on lines
+32
to
+68
| def _as_tuple(config_files) -> Tuple[str, ...]: | ||
| if isinstance(config_files, str): | ||
| return (config_files,) | ||
| return tuple(config_files) | ||
|
|
||
|
|
||
| def _load_config(config_files) -> ConfigParser: | ||
| config = ConfigParser() | ||
| read_files = config.read(list(_as_tuple(config_files))) | ||
| if not read_files: | ||
| raise FileNotFoundError(f"Can not read DeepH config files: {config_files}") | ||
| return config | ||
|
|
||
|
|
||
| def _np_dtype(dtype: str): | ||
| if dtype == "float32": | ||
| return np.float32 | ||
| if dtype == "float16": | ||
| return np.float16 | ||
| if dtype == "float64": | ||
| return np.float64 | ||
| raise ValueError(f"Unknown DeepH dtype: {dtype}") | ||
|
|
||
|
|
||
| def _array_to_paddle(array, dtype: str) -> paddle.Tensor: | ||
| return paddle.to_tensor(np.asarray(array), dtype=dtype) | ||
|
|
||
|
|
||
| def _list_structure_folders(raw_data_dir: str, interface: str, nums: int | None = None): | ||
| if interface != "npz": | ||
| raise NotImplementedError( | ||
| "DeepHDataset currently supports the validated npz Hamiltonian format." | ||
| ) | ||
| marker = "rc.npz" | ||
| folder_list = [] | ||
| for root, _, files in os.walk(raw_data_dir): | ||
| if marker in files: |
| ) | ||
| return graphs, info | ||
|
|
||
| def _build_graph_from_folder(self, folder: str, config): |
Author
There was a problem hiding this comment.
这里已按 PaddleMaterials Dataset 形式重新整理了一版，目前保留 deeph_dataset.py + deeph_graph.py 这两个文件主要是因为 DeepH 数据格式和监督目标比较特殊。
DeepH processed data 不是通用的 CIF/JSON/CSV 数据格式，而是由 lat.dat、site_positions.dat、element.dat、orbital_types.dat、rh.npz、rc.npz 等文件共同描述一个样本。其中 rh.npz/rc.npz 还需要和周期边、orbital block、local coordinate matrix、LCMP subgraph 对齐，才能生成模型训练所需的 label/mask/subgraph metadata。
因此当前实现中：
- 通用结构构建已复用 BuildStructure；
- 周期邻接图构建已复用 FindPointsInSpheres；
- cache 已按 PaddleMaterials 风格整理为 dataset_cfg.pkl + graphs.pkl；
- collate 已回到 DefaultCollator；
- DeepH 特有的 Hamiltonian label、orbital mask、LCMP subgraph 和 onsite edge 逻辑保留在 deeph_graph.py。
也就是说，通用能力已经尽量复用套件已有实现；保留下来的部分主要是 DeepH 数据格式到模型输入之间的必要转换逻辑，暂时没有现成通用模块可以直接替代。
Collaborator
There was a problem hiding this comment.
lcmp subgraph 也写成一个graph_convert吧，然后在这里调用
Collaborator
There was a problem hiding this comment.
建议按“通用能力”和“DeepH 特有逻辑”拆分，尽量复用套件已有实现，减少重复造轮子：
尽量复用已有 graph_converter，尽量复用套件已有 dataset / build / cache 处理逻辑
Author
|
已根据意见继续调整并验证。已在 PaddlePaddle GPU 3.2.2 环境下验证当前代码训推链路，请再次核实，谢谢！ |
leeleolay
reviewed
Jun 25, 2026
leeleolay
reviewed
Jun 25, 2026
| f"dict, list, number, None, pgl.Graph, but got {type(sample)}" | ||
| ) | ||
|
|
||
|
|
| elif isinstance(sample, Data): | ||
| # Geometric `Data` objects: batch them into a single `Batch` | ||
| return Batch.from_data_list(batch) | ||
| return Batch.from_data_list(batch).to_dict() |
Collaborator
There was a problem hiding this comment.
这里是原始的有错误吗，建议修改dataset的数据处理格式，这个是公共函数
Comment on lines
+37
to
+75
| def _as_tuple(config_files) -> Tuple[str, ...]: | ||
| if isinstance(config_files, str): | ||
| return (config_files,) | ||
| return tuple(config_files) | ||
|
|
||
|
|
||
| def _load_config(config_files) -> ConfigParser: | ||
| config = ConfigParser() | ||
| read_files = config.read(list(_as_tuple(config_files))) | ||
| if not read_files: | ||
| raise FileNotFoundError(f"Can not read DeepH config files: {config_files}") | ||
| return config | ||
|
|
||
|
|
||
| def _np_dtype(dtype: str): | ||
| if dtype == "float32": | ||
| return np.float32 | ||
| if dtype == "float16": | ||
| return np.float16 | ||
| if dtype == "float64": | ||
| return np.float64 | ||
| raise ValueError(f"Unknown DeepH dtype: {dtype}") | ||
|
|
||
|
|
||
| def _array_to_paddle(array, dtype: str) -> paddle.Tensor: | ||
| return paddle.to_tensor(np.asarray(array), dtype=dtype) | ||
|
|
||
|
|
||
| def _list_structure_folders(raw_data_dir: str, interface: str, nums: int | None = None): | ||
| if interface != "npz": | ||
| raise NotImplementedError( | ||
| "DeepHDataset currently supports the validated npz Hamiltonian format." | ||
| ) | ||
| folder_list = sorted( | ||
| root for root, _, files in os.walk(raw_data_dir) if "rc.npz" in files | ||
| ) | ||
| if nums is not None: | ||
| folder_list = folder_list[:nums] | ||
| return folder_list |
Collaborator
There was a problem hiding this comment.
这部分都用到了吗，可以被已有的功能替代吗，还有数据类型转换建议在实现中修正
| ) | ||
| return graphs, info | ||
|
|
||
| def _build_graph_from_folder(self, folder: str, config): |
Collaborator
There was a problem hiding this comment.
lcmp subgraph 也写成一个graph_convert吧，然后在这里调用
Author
|
已修改，烦请再次审阅，谢谢！ |
leeleolay
reviewed
Jun 26, 2026
Comment on lines
+37
to
+75
| def _as_tuple(config_files) -> Tuple[str, ...]: | ||
| if isinstance(config_files, str): | ||
| return (config_files,) | ||
| return tuple(config_files) | ||
|
|
||
|
|
||
| def _load_config(config_files) -> ConfigParser: | ||
| config = ConfigParser() | ||
| read_files = config.read(list(_as_tuple(config_files))) | ||
| if not read_files: | ||
| raise FileNotFoundError(f"Can not read DeepH config files: {config_files}") | ||
| return config | ||
|
|
||
|
|
||
| def _np_dtype(dtype: str): | ||
| if dtype == "float32": | ||
| return np.float32 | ||
| if dtype == "float16": | ||
| return np.float16 | ||
| if dtype == "float64": | ||
| return np.float64 | ||
| raise ValueError(f"Unknown DeepH dtype: {dtype}") | ||
|
|
||
|
|
||
| def _array_to_paddle(array, dtype: str) -> paddle.Tensor: | ||
| return paddle.to_tensor(np.asarray(array), dtype=dtype) | ||
|
|
||
|
|
||
| def _list_structure_folders(raw_data_dir: str, interface: str, nums: int | None = None): | ||
| if interface != "npz": | ||
| raise NotImplementedError( | ||
| "DeepHDataset currently supports the validated npz Hamiltonian format." | ||
| ) | ||
| folder_list = sorted( | ||
| root for root, _, files in os.walk(raw_data_dir) if "rc.npz" in files | ||
| ) | ||
| if nums is not None: | ||
| folder_list = folder_list[:nums] | ||
| return folder_list |
|
|
||
| def _build_graph_converter(self, config): | ||
| cfg = self._graph_converter_cfg(config) | ||
| if cfg["__class_name__"] != "FindPointsInSpheres": |
Collaborator
There was a problem hiding this comment.
这些逻辑不要在这里实现，按照套件内原有的数据集模式来重构下dataset @Belugaaaa
| ) | ||
|
|
||
|
|
||
| class DeepHGraphConverter: |
| import paddle.nn.functional as F | ||
|
|
||
|
|
||
| def scatter_add(src, index, dim_size=None): |
| return paddle.scatter_nd_add(out, index, src) | ||
|
|
||
|
|
||
| class MaskMSELoss(nn.Layer): |
| self.target_name: pred, | ||
| } | ||
|
|
||
| def sample(self, batch, **kwargs): |
Collaborator
There was a problem hiding this comment.
sample和predict的差别，为什么要单独写个sample
Collaborator
|
添加数据处理和模型推理可完整应用在dft计算的逻辑实现 |
Collaborator
|
调用第三方科学计算软件（DFT）优先考虑ase以及ppmatSim里的实现方式，保持兼容ppmatSim |
Author
|
已根据意见继续补充了一版，麻烦老师再帮忙看下。 目前我们这边新增了 DeepH 推理结果到 Hamiltonian block 的导出逻辑：
另外也补了 ASE 兼容入口：
目前已经补了对应单测，并验证过：
想再确认一下老师这里说的“完整应用在 DFT 计算的逻辑实现”，是希望我们做到：
如果是第 2 种，我们可能需要再补一个更完整的 workflow 层实现。 |
Collaborator
是第二种 |
Author
|
已修改，麻烦老师再次审阅，谢谢！ |
leeleolay
reviewed
Jul 20, 2026
Comment on lines
+44
to
+49
| ## Environment | ||
|
|
||
| The reproduced results use the official PaddlePaddle GPU 3.2.2 release and | ||
| Python 3.10. Install PaddlePaddle for the local CUDA platform, then install the | ||
| PaddleMaterials dependencies: | ||
|
|
Comment on lines
+50
to
+53
| ```bash | ||
| python -m pip install paddlepaddle-gpu==3.2.2 | ||
| python -m pip install -r requirements.txt | ||
| ``` |
Comment on lines
+75
to
+85
| The PaddleMaterials dataset adapter: | ||
|
|
||
| - reads DeepH processed folders from `raw_dir`; | ||
| - reconstructs canonical `pymatgen.Structure` objects with `BuildStructure`; | ||
| - builds periodic crystal graphs with `FindPointsInSpheres`; | ||
| - attaches DeepH Hamiltonian labels and LCMP subgraph metadata; | ||
| - caches parsed graph samples under the configured `graph_dir`. | ||
|
|
||
| The graphene dataset, pretrained Paddle checkpoint, and acceptance logs are in | ||
| the [DeepH handoff package](https://pan.baidu.com/s/1dkXlzYV_3DLQugoQDKjsbA?pwd=vi6r) | ||
| (password: `vi6r`). Prepare the repository paths with: |
leeleolay
reviewed
Jul 20, 2026
Comment on lines
+27
to
+43
| def _to_numpy(data): | ||
| if paddle.is_tensor(data): | ||
| return data.numpy() | ||
| return np.asarray(data) | ||
|
|
||
|
|
||
| def _get_structure_folder(batch) -> str: | ||
| folder = batch.get("structure_folder") | ||
| if isinstance(folder, (list, tuple)): | ||
| if len(folder) != 1: | ||
| raise ValueError( | ||
| "DeepH Hamiltonian export currently expects batch_size=1." | ||
| ) | ||
| folder = folder[0] | ||
| if not isinstance(folder, str): | ||
| raise TypeError("DeepH Hamiltonian export requires `structure_folder`.") | ||
| return folder |
| return folder | ||
|
|
||
|
|
||
| def build_deeph_hamiltonian_npz( |
| return hamiltonian | ||
|
|
||
|
|
||
| def save_deeph_hamiltonian_npz( |
Author
|
老师您好， 这里暂时没有直接合并到 yaml 中，主要是因为 orbital mapping 较长，放在 ini 中更方便和原始 DeepH 配置保持对应，也避免训练 yaml 过于臃肿。如果您觉得这部分也需要统一迁到 yaml，我可以继续调整。 其余问题均已完成修改，烦请再次审阅 |
Collaborator
|
放到yaml中吧，我理解orbital mapping信息是预先缓存的orbital信息，方便训练的时候使用吗？ |
Author
|
已完成迁移 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds DeepH integration into PaddleMaterials for electronic-structure Hamiltonian prediction.
Main changes:
DeepHHamiltonianmodel underppmat/models/deephAcceptance Evidence
Notes
Dataset, pretrained model, and log links are pending official BCE link backfill before merge.