Skip to content

Commit

Permalink
Merge pull request #64 from ZLI-afk/v1.2
Browse files Browse the repository at this point in the history
V1.2 update
  • Loading branch information
kevinwenminion authored May 6, 2024
2 parents 109af0d + 92b9ea6 commit f2afb74
Show file tree
Hide file tree
Showing 10 changed files with 122 additions and 28 deletions.
31 changes: 25 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@
</div>

# APEX: Alloy Property EXplorer
[![](https://img.shields.io/badge/release-1.2.0-blue.svg)](https://github.com/deepmodeling/APEX)

[APEX](https://github.com/deepmodeling/APEX): Alloy Property EXplorer is a component of the [AI Square](https://aissquare.com/) project that involves the restructuring of the [DP-GEN](https://github.com/deepmodeling/dpgen) `auto_test` module to develop a versatile and extensible Python package for general alloy property calculations. This package enables users to conveniently establish a wide range of cloud-native property-test workflows by utilizing various computational approaches, including LAMMPS, VASP, ABACUS, and others.

## v1.2 Main Features Update
* Add a `retrieve` sub-command to allow results to be retrieved independently and manually for multiple properties (Remove `Distributor` and `Collector` OP)
* Support common **dflow operations** with terminal commands
* Incorporate results `archive` function to both local paths and NoSQL database ([MongoDB](https://www.mongodb.com/) and [DynamoDB](https://aws.amazon.com/cn/dynamodb/))
Expand All @@ -15,9 +17,24 @@
* Add four additional **ML** pair styles (`snap`, `gap`, `rann` and `mace`) and an extra `meam-spline` in LAMMPS interation type support
* Modify the single-step run command from `test` to `do` for improved clarity and consistencey

## APEX Bohrium App
[![](https://img.shields.io/badge/APP-BohriumApp-orange.svg)](https://app.bohrium.dp.tech/apex/)

APEX also provides a web-based [Bohrium App](https://app.bohrium.dp.tech/apex/) for rapid and easy alloy property calculations without intensive JSON configuration (Note: one will need a Bohrium account to access this service).

## How to cite APEX
[![](https://img.shields.io/badge/DOI-10.48550/arXiv.2404.17330-red.svg)](https://doi.org/10.48550/arXiv.2404.17330)

If you use APEX in your research, please cite the following paper for general purpose:

> Z. Li, T. Wen, Y. Zhang, X. Liu, C. Zhang, A. S. L. S. Pattamatta, X. Gong, B. Ye, H.Wang, L. Zhang, D. J. Srolovitz, An extendable cloud-native alloy property explorer (2024). arXiv:2404.17330.
## Table of Contents

- [APEX: Alloy Property EXplorer](#apex-alloy-property-explorer)
- [v1.2 Main Features Update](#v12-main-features-update)
- [APEX Bohrium App](#apex-bohrium-app)
- [How to cite APEX](#how-to-cite-apex)
- [Table of Contents](#table-of-contents)
- [1. Overview](#1-overview)
- [2. Easy Install](#2-easy-install)
Expand Down Expand Up @@ -109,7 +126,8 @@ The instructions regarding global configuration, [dflow](https://github.com/deep
| group_size | Int | 1 | Number of tasks per parallel run group |
| pool_size | Int | 1 | For multi tasks per parallel group, the pool size of multiprocessing pool to handle each task (1 for serial, -1 for infinity) |
| upload_python_package | Optional[List] | None | Additional python packages required in the container |
| debug_pool_workers | Int | 1 | Pool size of parallel tasks running in the debug mode |
| debug_pool_workers | Int | 1 | Pool size of parallel tasks running in the debug mode |
| flow_name | String | None | Specify name of workflow to be submitted (default: work path name) |
| submit_only | Bool | False | Submit workflow only without automatic result retrieving |

* **Dflow config**
Expand Down Expand Up @@ -252,11 +270,12 @@ Below are three examples (for detailed explanations of each parameter, please re
| vol_abs | Bool | False | Whether to treat vol_start and vol_end as absolute volume, default = False |

##### 3.1.2.2. Elastic
| Key words | Data structure | Example | Description |
|:-------------|----------------|---------|----------------------------------------------------|
| norm_deform | Float | 0.01 | The deformation in xx, yy, zz, defaul = 1e-2 |
| shear_deform | Float | 0.01 | The deformation in other directions, default = 1e-2 |
| conventional | Bool | False | Whether adopt conventional cell for deformation |
| Key words | Data structure | Example | Description |
|:-------------|----------------|---------|-----------------------------------------------------------------------------------------------------------------------------------|
| norm_deform | Float | 0.01 | The deformation in xx, yy, zz, defaul = 1e-2 |
| shear_deform | Float | 0.01 | The deformation in other directions, default = 1e-2 |
| conventional | Bool | False | Whether adopt conventional cell for deformation |
| ieee | Bool | True | Whether rotate relaxed structure into IEEE-standard format before deformation ([ref](https://ieeexplore.ieee.org/document/26560)) |

##### 3.1.2.3. Surface
| Key words | Data structure | Example | Description |
Expand Down
4 changes: 4 additions & 0 deletions apex/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,9 @@ def header():
header_str += " AAA AAA PPP EEEEEEEEEE XXX XXX\n"
header_str += "---------------------------------------------------------------\n"
header_str += f"==>> Alloy Property EXplorer using simulations (v{__version__})\n"
header_str += "Please cite DOI: 10.48550/arXiv.2404.17330\n"
header_str += "Li et al, An extendable cloud-native alloy property explorer (2024).\n"
header_str += "See https://github.com/deepmodeling/APEX for more information.\n"
header_str += "---------------------------------------------------------------\n"
header_str += "Checking input files..."
print(header_str)
2 changes: 1 addition & 1 deletion apex/archive.py
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ def archive2db_from_json(config, json_file):
if config.archive_key:
data_id = config.archive_key
else:
data_id = str(data_dict["work_path"])
data_id = data_dict['archive_key']
data_dict['_id'] = data_id

archive2db(config, data_dict, data_id)
Expand Down
5 changes: 4 additions & 1 deletion apex/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ class Config:
remote_password: str = None
port: int = 22

# calculator config
# basic run config
run_image_name: str = None
run_command: str = None
apex_image_name: str = "zhuoyli/apex_amd64"
Expand All @@ -63,8 +63,11 @@ class Config:
vasp_run_command: str = None
abacus_image_name: str = None
abacus_run_command: str = None

# common APEX config
is_bohrium_dflow: bool = False
submit_only: bool = False
flow_name: str = None

database_type: str = 'local'
archive_method: str = 'sync'
Expand Down
14 changes: 12 additions & 2 deletions apex/core/property/Elastic.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@
import re
from shutil import copyfile

import numpy as np
from monty.serialization import dumpfn, loadfn
from pymatgen.analysis.elasticity.elastic import ElasticTensor
from pymatgen.analysis.elasticity.strain import DeformedStructureSet, Strain
from pymatgen.analysis.elasticity.stress import Stress
from pymatgen.core.structure import Structure
from pymatgen.core.tensors import Tensor
from pymatgen.core.operations import SymmOp
from pymatgen.io.vasp import Incar, Kpoints

from apex.core.calculator.lib import abacus_utils
Expand All @@ -32,6 +33,8 @@ def __init__(self, parameter, inter_param=None):
self.shear_deform = parameter["shear_deform"]
parameter.setdefault("conventional", False)
self.conventional = parameter["conventional"]
parameter.setdefault("ieee", True)
self.ieee = parameter["ieee"]
parameter.setdefault("cal_type", "relaxation")
self.cal_type = parameter["cal_type"]
default_cal_setting = {
Expand Down Expand Up @@ -141,6 +144,13 @@ def make_confs(self, path_to_work, path_to_equi, refine=False):
ss = st.conventional_structure
ss.to(os.path.join(path_to_work, "POSCAR.conv"), "POSCAR")

# convert to IEEE-standard
if self.ieee:
rot = Tensor.get_ieee_rotation(ss)
op = SymmOp.from_rotation_and_translation(rot)
ss.apply_operation(op)
ss.to(os.path.join(path_to_work, "POSCAR.ieee"), "POSCAR")

dfm_ss = DeformedStructureSet(
ss,
symmetry=False,
Expand Down Expand Up @@ -169,7 +179,7 @@ def make_confs(self, path_to_work, path_to_equi, refine=False):
dfm_ss.deformed_structures[ii].to("POSCAR", "POSCAR")
if self.inter_param["type"] == "abacus":
abacus_utils.poscar2stru("POSCAR", self.inter_param, "STRU")
os.remove("POSCAR")
#os.remove("POSCAR")
# record strain
df = Strain.from_deformation(dfm_ss.deformations[ii])
dumpfn(df.as_dict(), "strain.json", indent=4)
Expand Down
37 changes: 33 additions & 4 deletions apex/flow.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,23 @@ def __init__(
self.executor = executor
self.upload_python_packages = upload_python_packages

@staticmethod
def regulate_name(name):
"""
Adjusts the given workflow name to conform to RFC 1123 subdomain requirements.
It ensures the name is lowercase, contains only alphanumeric characters and hyphens,
and starts and ends with an alphanumeric character.
"""
# lowercase the name
name = name.lower()
# substitute invalid characters with hyphens
name = re.sub(r'[^a-z0-9\-]', '-', name)
# make sure the name starts and ends with an alphanumeric character
name = re.sub(r'^[^a-z0-9]+', '', name)
name = re.sub(r'[^a-z0-9]+$', '', name)

return name

def _monitor_relax(self):
print('Waiting for relaxation result...')
while True:
Expand All @@ -77,7 +94,10 @@ def _monitor_relax(self):
wf_status = self.workflow.query_status()
if wf_status == 'Failed':
raise RuntimeError(f'Workflow failed (ID: {self.workflow.id}, UID: {self.workflow.uid})')
relax_post = step_info.get_step(name='relaxation-cal')[0]
try:
relax_post = step_info.get_step(name='relaxation-cal')[0]
except IndexError:
continue
if relax_post['phase'] == 'Succeeded':
print(f'Relaxation finished (ID: {self.workflow.id}, UID: {self.workflow.uid})')
print('Retrieving completed tasks to local...')
Expand Down Expand Up @@ -246,12 +266,15 @@ def submit_relax(
download_path: Union[os.PathLike, str],
relax_parameter: dict,
submit_only: bool = False,
name: Optional[str] = None,
labels: Optional[dict] = None
) -> str:
self.upload_path = upload_path
self.download_path = download_path
self.relax_param = relax_parameter
self.workflow = Workflow(name='relaxation', labels=labels)
flow_name = name if name else self.regulate_name(os.path.basename(download_path))
flow_name += '-relax'
self.workflow = Workflow(name=flow_name, labels=labels)
relaxation = self._set_relax_flow(
input_work_dir=upload_artifact(upload_path),
relax_parameter=relax_parameter
Expand All @@ -272,12 +295,15 @@ def submit_props(
download_path: Union[os.PathLike, str],
props_parameter: dict,
submit_only: bool = False,
name: Optional[str] = None,
labels: Optional[dict] = None
) -> str:
self.upload_path = upload_path
self.download_path = download_path
self.props_param = props_parameter
self.workflow = Workflow(name='property', labels=labels)
flow_name = name if name else self.regulate_name(os.path.basename(download_path))
flow_name += '-props'
self.workflow = Workflow(name=flow_name, labels=labels)
subprops_list, subprops_key_list = self._set_props_flow(
input_work_dir=upload_artifact(upload_path),
props_parameter=props_parameter
Expand All @@ -299,13 +325,16 @@ def submit_joint(
relax_parameter: dict,
props_parameter: dict,
submit_only: bool = False,
name: Optional[str] = None,
labels: Optional[dict] = None
) -> str:
self.upload_path = upload_path
self.download_path = download_path
self.relax_param = relax_parameter
self.props_param = props_parameter
self.workflow = Workflow(name='joint', labels=labels)
flow_name = name if name else self.regulate_name(os.path.basename(download_path))
flow_name += '-joint'
self.workflow = Workflow(name=flow_name, labels=labels)
relaxation = self._set_relax_flow(
input_work_dir=upload_artifact(upload_path),
relax_parameter=self.relax_param
Expand Down
6 changes: 6 additions & 0 deletions apex/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,11 @@ def parse_args():
choices=['relax', 'props', 'joint'],
help="(Optional) Specify type of workflow to submit: (relax | props | joint)"
)
parser_submit.add_argument(
"-n", "--name",
type=str, default=None,
help="(Optional) Specify name of the workflow",
)

##########################################
# Do single step locally
Expand Down Expand Up @@ -526,6 +531,7 @@ def main():
config_file=args.config,
work_dirs=args.work,
indicated_flow_type=args.flow,
flow_name=args.name,
submit_only=args.submit_only,
is_debug=args.debug
)
Expand Down
1 change: 1 addition & 0 deletions apex/report.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ def report_local(input_path_list):
data_dict = loadfn(kk)
try:
workdir_id = data_dict.pop('work_path')
_ = data_dict.pop('archive_key')
except KeyError:
logging.warning(msg=f'Invalid json for result archive, will skip: {kk}')
continue
Expand Down
31 changes: 24 additions & 7 deletions apex/submit.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,15 @@ def pack_upload_dir(
os.chdir(work_dir)
relax_confs = relax_param.get("structures", []) if relax_param else []
prop_confs = prop_param.get("structures", []) if prop_param else []
relax_prefix = relax_param["interaction"].get("potcar_prefix", None) if relax_param else None
prop_prefix = prop_param["interaction"].get("potcar_prefix", None) if prop_param else None
include_dirs = set()
if relax_prefix:
relax_prefix_base = relax_prefix.split('/')[0]
include_dirs.add(relax_prefix_base)
if prop_prefix:
prop_prefix_base = prop_prefix.split('/')[0]
include_dirs.add(prop_prefix_base)
confs = relax_confs + prop_confs
assert len(confs) > 0, "No configuration path indicated!"
conf_dirs = []
Expand All @@ -66,13 +75,11 @@ def pack_upload_dir(
backup_path(path_to_prop)

"""copy necessary files and directories into temp upload directory"""
# exclude 'all_result.json' from copy
conf_root_list = [conf.split('/')[0] for conf in conf_dirs]
conf_root_list = list(set(conf_root_list))
conf_root_list.sort()
ignore_copy_list = conf_root_list
ignore_copy_list.append("all_result.json")
copy_all_other_files(work_dir, upload_dir, ignore_list=ignore_copy_list)
copy_all_other_files(
work_dir, upload_dir,
exclude_files=["all_result.json"],
include_dirs=list(include_dirs)
)
for ii in conf_dirs:
build_conf_path = os.path.join(upload_dir, ii)
os.makedirs(build_conf_path, exist_ok=True)
Expand Down Expand Up @@ -131,13 +138,15 @@ def submit(
)

flow_id = None
flow_name = wf_config.flow_name
submit_only = wf_config.submit_only
if flow_type == 'relax':
flow_id = flow.submit_relax(
upload_path=tmp_dir,
download_path=work_dir,
relax_parameter=relax_param,
submit_only=submit_only,
name=flow_name,
labels=labels
)
elif flow_type == 'props':
Expand All @@ -146,6 +155,7 @@ def submit(
download_path=work_dir,
props_parameter=props_param,
submit_only=submit_only,
name=flow_name,
labels=labels
)
elif flow_type == 'joint':
Expand All @@ -155,6 +165,7 @@ def submit(
props_parameter=props_param,
relax_parameter=relax_param,
submit_only=submit_only,
name=flow_name,
labels=labels
)

Expand All @@ -169,6 +180,7 @@ def submit_workflow(
config_dict: dict,
work_dirs: List[os.PathLike],
indicated_flow_type: str,
flow_name: str = None,
submit_only=False,
is_debug=False,
labels=None
Expand All @@ -188,6 +200,9 @@ def submit_workflow(
config["debug_workdir"] = config_dict.get("debug_workdir", tmp_work_dir.name)
s3_config["storage_client"] = None

if flow_name:
wf_config.flow_name = flow_name

# judge basic flow info from user indicated parameter files
(run_op, calculator, flow_type,
relax_param, props_param) = judge_flow(parameter_dicts, indicated_flow_type)
Expand Down Expand Up @@ -269,6 +284,7 @@ def submit_from_args(
config_file: os.PathLike,
work_dirs,
indicated_flow_type: str,
flow_name: str = None,
submit_only=False,
is_debug=False,
):
Expand All @@ -278,6 +294,7 @@ def submit_from_args(
config_dict=load_config_file(config_file),
work_dirs=work_dirs,
indicated_flow_type=indicated_flow_type,
flow_name=flow_name,
submit_only=submit_only,
is_debug=is_debug,
)
Expand Down
Loading

0 comments on commit f2afb74

Please sign in to comment.