Allow user defined forward_common_files and backward_files

README.md

@@ -25,7 +25,7 @@
 [![conda install](https://img.shields.io/conda/dn/conda-forge/dpgen?label=conda%20install)](https://anaconda.org/conda-forge/dpgen)
 [![pip install](https://img.shields.io/pypi/dm/dpgen?label=pip%20install)](https://pypi.org/project/dpgen)
 
-DP-GEN (Deep Generator)  is a software written in Python, delicately designed to generate a deep learning based model of interatomic potential energy and force field. DP-GEN is depedent on DeepMD-kit (https://github.com/deepmodeling/deepmd-kit/blob/master/README.md). With highly scalable interface with common softwares for molecular simulation, DP-GEN is capable to  automatically prepare scripts and maintain job queues on HPC machines (High Performance Cluster) and analyze results.
+DP-GEN (Deep Generator)  is a software written in Python, delicately designed to generate a deep learning based model of interatomic potential energy and force field. DP-GEN is dependent on [DeepMD-kit](https://github.com/deepmodeling/deepmd-kit/blob/master/README.md). With highly scalable interface with common softwares for molecular simulation, DP-GEN is capable to  automatically prepare scripts and maintain job queues on HPC machines (High Performance Cluster) and analyze results.
 
 If you use this software in any publication, please cite:
 
@@ -34,7 +34,7 @@ Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and
 ### Highlighted features
 + **Accurate and efficient**: DP-GEN is capable to sample more than tens of million structures and select only a few for first principles calculation. DP-GEN will finally obtain a uniformly accurate model.
 + **User-friendly and automatic**: Users may install and run DP-GEN easily. Once succusefully running, DP-GEN can dispatch and handle all jobs on HPCs, and thus there's no need for any personal effort.
-+ **Highly scalable**: With modularized code structures, users and developers can easily extend DP-GEN for their most relevant needs. DP-GEN currently supports for HPC systems (Slurm, PBS, LSF and cloud machines ), Deep Potential interface with DeePMD-kit, MD interface with LAMMPS  and *ab-initio* calculation interface with VASP, PWSCF,SIESTA and Gaussian. We're sincerely welcome and embraced to users' contributions, with more possibilities and cases to use DP-GEN.
++ **Highly scalable**: With modularized code structures, users and developers can easily extend DP-GEN for their most relevant needs. DP-GEN currently supports for HPC systems (Slurm, PBS, LSF and cloud machines ), Deep Potential interface with DeePMD-kit, MD interface with [LAMMPS](https://www.lammps.org/), [Gromacs](http://www.gromacs.org/)   and *ab-initio* calculation interface with VASP, PWSCF, CP2K, SIESTA and Gaussian, Abacus, PWMAT, etc . We're sincerely welcome and embraced to users' contributions, with more possibilities and cases to use DP-GEN.
 
 ### Code structure and interface
 + dpgen:
@@ -43,7 +43,8 @@ Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and
     * generator: source codes for main process of deep generator.
 
     * auto_test : source code for undertaking materials property analysis.
-    * remote : source code for automatically submiting scripts,maintaining job queues and collecting results.
+    * remote and dispatcher : source code for automatically submiting scripts,maintaining job queues and collecting results.
+    Notice this part hase been integrated into [dpdispatcher](https://github.com/deepmodeling/dpdispatcher) 
     * database : source code for collecting data generated by DP-GEN and interface with database.
 + examples : providing example JSON files.
 
@@ -63,6 +64,15 @@ Options for TASK:
 * `test`: Auto-test for Deep Potential.
 * `db`: Collecting data from DP-GEN.
 
+
+[Here](examples) are examples you can refer to. You should make sure that provide a correct [JSON](https://docs.python.org/3/library/json.html) file.
+You can use following command to check your JSON file.
+```python
+import json
+#Specify machine parameters in machine.json
+json.load(open("machine.json"))
+```
+
 ## Download and Install
 One can download the source code of dpgen by
 ```bash
@@ -1322,7 +1332,9 @@ mem_limit | Interger | 16 | Maximal memory permitted to apply for the job.
 | # End of resources
 | command | String | "lmp_serial" | Executable path of software, such as `lmp_serial`, `lmp_mpi` and `vasp_gpu`, `vasp_std`, etc.
 | group_size | Integer | 5 | DP-GEN will put these jobs together in one submitting script.
-
+| user_forward_files | List of str | ["/path_to/vdw_kernel.bindat"] | These files will be uploaded in each calculation task. You should make sure provide the path exists. 
+| user_backward_files | List of str | ["HILLS"] | Besides DP-GEN's normal output, these files will be downloaded after each calculation. You should make sure these files can be generated.
+
 ## Troubleshooting
 1. The most common problem is whether two settings correspond with each other, including:
     - The order of elements in `type_map` and `mass_map` and **`fp_pp_files`**.

doc/CONTRIBUTING.md

@@ -0,0 +1,10 @@
+# DP-GEN Contributing Guide
+Welcome to [DP-GEN](https://github.com/deepmodeling/dpgen/tree/master/dpgen) !
+
+
+## How to contribute
+DP-GEN adopts the same convention as other softwares in DeepModeling Community. 
+You can first refer to DeePMD-kit's
+[Contributing guide](https://github.com/deepmodeling/deepmd-kit/edit/devel/CONTRIBUTING.md)
+and [Developer guide](https://github.com/deepmodeling/deepmd-kit/edit/devel/doc/development/index.md).
+

dpgen/auto_test/common_equi.py

@@ -9,10 +9,9 @@
 from dpgen.auto_test.calculator import make_calculator
 from dpgen.auto_test.mpdb import get_structure
 from dpgen.dispatcher.Dispatcher import make_dispatcher
-from dpgen.remote.decide_machine import decide_fp_machine, decide_model_devi_machine
 from distutils.version import LooseVersion
 from dpgen.dispatcher.Dispatcher import make_submission
-
+from dpgen.remote.decide_machine import convert_mdata
 lammps_task_type = ['deepmd', 'meam', 'eam_fs', 'eam_alloy']
 
 
@@ -133,9 +132,9 @@ def run_equi(confs,
     inter_type = inter_param['type']
     # vasp
     if inter_type == "vasp":
-        mdata = decide_fp_machine(mdata)
+        mdata = convert_mdata(mdata, ["fp"])
     elif inter_type in lammps_task_type:
-        mdata = decide_model_devi_machine(mdata)
+        mdata = convert_mdata(mdata, ["model_devi"])
     else:
         raise RuntimeError("unknown task %s, something wrong" % inter_type)
 

dpgen/auto_test/common_prop.py

@@ -13,9 +13,8 @@
 from dpgen.auto_test.Vacancy import Vacancy
 from dpgen.auto_test.calculator import make_calculator
 from dpgen.dispatcher.Dispatcher import make_dispatcher
-from dpgen.remote.decide_machine import decide_fp_machine, decide_model_devi_machine
 from dpgen.dispatcher.Dispatcher import make_submission
-
+from dpgen.remote.decide_machine import convert_mdata
 lammps_task_type = ['deepmd', 'meam', 'eam_fs', 'eam_alloy']
 
 
@@ -150,9 +149,9 @@ def run_property(confs,
             inter_type = inter_param_prop['type']
             # vasp
             if inter_type == "vasp":
-                mdata = decide_fp_machine(mdata)
+                mdata = convert_mdata(mdata, ["fp"])
             elif inter_type in lammps_task_type:
-                mdata = decide_model_devi_machine(mdata)
+                mdata = convert_mdata(mdata, ["model_devi"])
             else:
                 raise RuntimeError("unknown task %s, something wrong" % inter_type)
 

dpgen/auto_test/lib/util.py

@@ -77,11 +77,11 @@ def get_machine_info(mdata,task_type):
         command = vasp_exec
         command = cmd_append_log(command, "log")
     elif task_type in lammps_task_type:
-        lmp_exec = mdata['lmp_command']
+        model_devi_exec = mdata['model_devi_command']
         group_size = mdata['model_devi_group_size']
         resources = mdata['model_devi_resources']
         machine=mdata['model_devi_machine']
-        command = lmp_exec + " -i in.lammps"
+        command = model_devi_exec + " -i in.lammps"
         command = cmd_append_log(command, "model_devi.log")
     return machine, resources, command, group_size
 

dpgen/data/gen.py

@@ -24,7 +24,7 @@
 from dpgen.generator.lib.vasp import incar_upper
 from pymatgen.core import Structure
 from pymatgen.io.vasp import Incar
-from dpgen.remote.decide_machine import  decide_fp_machine
+from dpgen.remote.decide_machine import  convert_mdata
 from dpgen import ROOT_PATH
 from dpgen.dispatcher.Dispatcher import Dispatcher, make_dispatcher, make_submission
 
@@ -694,7 +694,7 @@ def gen_init_bulk(args) :
 
     if args.MACHINE is not None:
        # Selecting a proper machine
-       mdata = decide_fp_machine(mdata)
+       mdata = convert_mdata(mdata, ["fp"])
        #disp = make_dispatcher(mdata["fp_machine"])
 
     # Decide work path

dpgen/data/surf.py

@@ -11,7 +11,7 @@
 import dpgen.data.tools.bcc as bcc
 from dpgen import dlog
 from dpgen import ROOT_PATH
-from dpgen.remote.decide_machine import  decide_fp_machine
+from dpgen.remote.decide_machine import  convert_mdata
 from dpgen.dispatcher.Dispatcher import Dispatcher, make_dispatcher
 #-----PMG---------
 from pymatgen.io.vasp import Poscar
@@ -596,7 +596,7 @@ def gen_init_surf(args):
 
     if args.MACHINE is not None:
        # Decide a proper machine
-       mdata = decide_fp_machine(mdata)
+       mdata = convert_mdata(mdata, ["fp"])
        # disp = make_dispatcher(mdata["fp_machine"])
 
     #stage = args.STAGE

dpgen/generator/ch4/machine.json

@@ -21,7 +21,7 @@
 	"_comment":	"that's all"
     },
 
-    "lmp_command":	"/sharedext4/softwares/lammps/bin/lmp_serial",
+    "model_devi_command":	"/sharedext4/softwares/lammps/bin/lmp_serial",
     "model_devi_group_size":	1,
     "_comment":		"model_devi on localhost",
     "model_devi_machine":	{

dpgen/generator/lib/utils.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
 
 import os, re, shutil, logging
+import glob
 
 iter_format = "%06d"
 task_format = "%02d"
@@ -60,4 +61,35 @@ def log_task (message) :
 
 def record_iter (record, ii, jj) :
     with open (record, "a") as frec :
-        frec.write ("%d %d\n" % (ii, jj)) 
+        frec.write ("%d %d\n" % (ii, jj))
+
+def symlink_user_forward_files(mdata, task_type, work_path):
+    '''
+    Symlink user-defined forward_common_files
+    Current path should be work_path, such as 00.train
+
+    Parameters
+    ---------
+    mdata : dict
+        machine parameters
+    task_type: str
+        task_type, such as "train"
+    work_path : str
+        work_path, such as "iter.000001/00.train"
+    Returns
+    -------
+    None
+    '''
+    user_forward_files = mdata.get(task_type + "_" + "user_forward_files", [])
+    #Angus: In the future, we may unify the task format.
+    task_format = {"train" : "0*", "model_devi" : "task.*", "fp": "task.*"}
+    for file in user_forward_files:
+        assert os.path.isfile(file)  ,\
+            "user_forward_common_file %s of %s stage doesn't exist. " % (file, task_type)
+        tasks = glob.glob(os.path.join(work_path, task_format[task_type]))
+        for task in tasks:
+            if os.path.isfile(os.path.join(task, os.path.basename(file))):
+                os.remove(os.path.join(task, os.path.basename(file)))
+            os.symlink(file, os.path.join(task, os.path.basename(file)))
+    return
+