Implementation of adaptive supercell settings and improvement of the documentation #80

QuantumChemist · 2024-07-11T20:00:49Z

Implemented adaptive settings for the supercells and improved the documentation.

The PR is in principle ready to be merged. 😃

I also added

making data_preprocessing continue even if not all data calculations were successfull. Making it abort the workflow if the percantance is below 95%

in this PR and have to fix the unit tests accordingly once the atomate2 VASP input issues have been fixed.

autoplex/data/common/utils.py

autoplex/data/phonons/flows.py

…were successfull. Making it abort the workflow if the percantance is below 95%

QuantumChemist · 2024-07-31T08:58:18Z

autoplex/data/phonons/flows.py

+                    jobs.append(supercell_job)
+                    supercell_matrix = supercell_job.output
+                except AttributeError:
+                    supercell_matrix = generate_supercell_matrix(


@JaGeo what do you think about keeping it as an "if everything else fails" option? To keep the workflow from aborting. As it's really annoying if most of your materials are running fine, but it's just one or two structures where the settings didn't match that well.

QuantumChemist · 2024-07-31T09:16:36Z

autoplex/data/phonons/flows.py

+                        structure, supercell_matrix
+                    )
+                    warnings.warn(
+                        message="Falling back to a simple supercell size schema."


with a warning that tells user to check their data in case this happened?

JaGeo · 2024-07-31T09:19:12Z

autoplex/auto/phonons/jobs.py

        phonon_displacement_maker = TightDFTStaticMakerBigSupercells()
+    if adaptive_phonopy_supercell_settings:
+        lattice_avg = sum(structure.lattice.abc) / 3
+        if lattice_avg > 10:


Schon bei 10?

yes, if the lattice parameters of the unit cell have been larger than 10 on average (!), it created those 1000+ atoms supercells with the usual supercell size settings that are hard to get converge. One example is mp-971662, where we arrive at 1242 atoms with a 3x3x3 supercell matrix. It has lattice parameters a = b = c = 10.18 in the primitive unit cell.

Can't you just use the supercell size itself to determine the maker?

The 3x3x3 supercell matrix was just an example here. But using min_length=18 created a similar big supercell. The CubicSupercellTransformation functions allows for up to 1200 atoms in the prefer_90_degrees=True case.

In this case I can't use the supercell size because the determination of the supercell only happens within the atomate2 PhononMaker. The only way to control it is by using min_length. I could add a get_supercell_size call maybe?

and without force_90_degree and the orthorhombic option?

I'm working on a unit test to check this.

You can also subsequently make the requirement smaller: i.e, test 13 Angstrom or 10 as a min_length

Or even iterate over it in steps of 1 or 2 Angstrom, maybe until 10 angstrom

mp-971662 was not among the troublesome phonopy supercells (but the rattled ones), but mp-1200830 was among the materials that caused this issue, the lattice parameter of the primitive cell is a = b = c = 10.76. I'm going to push the unit test now.

QuantumChemist · 2024-07-31T10:44:01Z

tests/data/test_datagen_utils.py

+                )
+
+    assert new_matrix == expected_matrix


@JaGeo
when I run this unit test, I get

============================== 1 passed in 15.04s ============================== PASSED [100%]/home/certural/miniconda3/envs/auto_debug/lib/python3.10/site-packages/_pytest/python.py:159: UserWarning: Falling back to orthorhombic, prefer 90. result = testfunction(**testargs) /home/certural/miniconda3/envs/auto_debug/lib/python3.10/site-packages/_pytest/python.py:159: UserWarning: Falling back to orthorhombic. result = testfunction(**testargs) /home/certural/miniconda3/envs/auto_debug/lib/python3.10/site-packages/_pytest/python.py:159: UserWarning: Falling back to a simple supercell size schema. Check if this is ok for your use case. result = testfunction(**testargs) Process finished with exit code 0

(you have to scroll sideways to see the warnings)

In my actual calculation for mp-1200830, I had

supercell_matrix: - [ -2, -2, 0 ] - [ -2, 0, -2 ] - [ 0, -2, -2 ]

It created a supercell with 656 atoms, which I could only converge with reduced settings.

I think I will then do the following:
start with cubic, then as you suggested above using allow_orthorhombic=True and prefer_90_degrees=False and then also reduce min_length as for the phonopy supercells, those are the only elements, I can control the supercell size with

But my question would still be how would that influence the DFT phonon calculation quality, if usually a min_length of around 20 is recommended? That's why I opted for reducing the calculation accuracy depending on the average unit cell lattice parameter

Yeah, it will reduce the quality but if the k point density is roughly stable, it should not be too bad.

Alright! :)

…ettings

…ate2

add a draft implementation for the supercell search

QuantumChemist · 2024-08-26T15:37:33Z

Hey @JaGeo , what is left now for this PR is that I fix all the atomate2 related VASP errors in the unit tests.

QuantumChemist · 2024-08-26T15:42:15Z

autoplex/data/phonons/utils.py

+
+    matrix = np.array([[a_factor, 0, 0], [0, b_factor, 0], [0, 0, c_factor]])
+    return matrix.transpose().tolist()
+    # despite the warning Expected type 'list', got 'object' instead, this is of <class 'list'>


and here I checked very carefully, that it really returns a list (it does). I couldn't find out why PyCharm flags it as object, so I left it like that. I don't think this will lead to code inconsistencies or bugs.

…a job

QuantumChemist · 2024-08-27T14:08:03Z

Hey @JaGeo 👋🏻
so the unit test run before my last push was sucessful, but then I noticed an issue with database_dir which I also fixed now. But I ran out of action time. I'm not sure if you planned to merge this before I tested a few supercells on supermuc, but in case you want to merge it, shall I repeat the unit test wf for the last push in e.g. Aakash's repo or is it ok like this?

QuantumChemist · 2024-08-27T14:13:04Z

autoplex/fitting/common/flows.py

+        # this will only run if train.extxyz and test.extxyz files are present in the database_dir
        mlip_fit_job = machine_learning_fit(
            database_dir=database_dir,
            isolated_atoms_energies=isolated_atoms_energies,
            num_processes_fit=num_processes_fit,
            auto_delta=auto_delta,
            glue_xml=glue_xml,
            mlip_type=self.mlip_type,
-            HPO=self.HPO,
+            hyper_param_optimization=self.hyper_param_optimization,
            ref_energy_name=self.ref_energy_name,
            ref_force_name=self.ref_force_name,
            ref_virial_name=self.ref_virial_name,


Hey @YuanbinLiu,
I have another question. I noticed that this part will only run successfully, when there is a train.extxyz and test.extyz file present (which are usually generated in the data_prep step), and I guess that this is the case for when you do the fit with the RSS-generated data. Is that correct?

JaGeo · 2024-08-27T14:57:25Z

@QuantumChemist I will merge it here. If a unit test fails, I assume you will take care of it :).

QuantumChemist and others added 9 commits July 6, 2024 13:51

improved jobflow-remote setup tutorial

ed4f05e

improved jobflow-remote setup tutorial

cff92f0

fix minor typo

e62eeef

minor changes

f988064

improving jobflow-remote setup text

1eec958

improving the jfr tutorial

16f1456

prevent too big ratlled supercells

e4e8c24

prevent too tight phonopy supercell settings

ca798b9

improved unit test

a50b3b3

QuantumChemist added documentation Improvements or additions to documentation enhancement New feature or request labels Jul 11, 2024

QuantumChemist requested a review from JaGeo July 11, 2024 20:01

JaGeo reviewed Jul 11, 2024

View reviewed changes

autoplex/data/common/utils.py Outdated Show resolved Hide resolved

QuantumChemist and others added 4 commits July 18, 2024 12:09

Merge branch 'JaGeo:main' into main

8c5380e

minor changes

431bd49

Merge branch 'main' of https://github.com/QuantumChemist/autoplex

f2019c2

adjusted adaptive supercell settings

41e3224

QuantumChemist commented Jul 18, 2024

View reviewed changes

autoplex/data/phonons/flows.py Outdated Show resolved Hide resolved

QuantumChemist added 3 commits July 25, 2024 21:27

making data_preprocessing continue even if not all data calculations …

a7a2472

…were successfull. Making it abort the workflow if the percantance is below 95%

change adaptive supercell routine

21e4c61

update jfremote installation instruction

b0477d1

QuantumChemist commented Jul 31, 2024

View reviewed changes

add warning

87a5685

QuantumChemist commented Jul 31, 2024

View reviewed changes

JaGeo reviewed Jul 31, 2024

View reviewed changes

QuantumChemist added 2 commits July 31, 2024 12:10

add warning

343b197

unit test to check get_supercell_size

fa00767

QuantumChemist commented Jul 31, 2024

View reviewed changes

QuantumChemist added 2 commits July 31, 2024 15:50

adjust unit tests

c381339

change benchmark DFT calcs to be also ffected by adaptive supercell s…

2808165

…ettings

QuantumChemist and others added 12 commits August 13, 2024 15:12

added step_size

26241c5

adjust atomate2 and pymatgen

ee3ae24

change matgl version because of dependencies conflicts caused by atom…

e395cb3

…ate2

changed one value

4aed55e

add a draft implementation

d59d968

add one fall back

698f2d8

change documentation

22703b5

Apply suggestions from code review

e5072c8

cleanup

e8a8f9f

replacing print with logging

9debe3b

minor changes

3071c15

Merge pull request #5 from QuantumChemist/supercell_options_jg

e90111d

add a draft implementation for the supercell search

QuantumChemist commented Aug 26, 2024

View reviewed changes

QuantumChemist added 12 commits August 26, 2024 17:56

small change in dosctring

c7887bc

fixed several issues with the unit tests. Made reduce_supercell_size …

9ef0b18

…a job

fixed wrong logic

2083467

more fixes

de9b72b

fixed unit test with new atomate2 vasp input format

d590116

add missing dosctring

bec0bfe

remove mlip_haper and adjust documentation

e4f516c

fix mock vasp

28b504f

add device=cpu for MACE test

9d5bd25

add device=cuda for MACE documentation

2ce7f14

fixing docstrings, variable names and types in the fit part

8db454f

fixing a few things regarding database_dir

a85dfad

QuantumChemist commented Aug 27, 2024

View reviewed changes

JaGeo merged commit f4fbdf2 into autoatml:main Aug 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of adaptive supercell settings and improvement of the documentation #80

Implementation of adaptive supercell settings and improvement of the documentation #80

QuantumChemist commented Jul 11, 2024 •

edited

Loading

QuantumChemist Jul 31, 2024

QuantumChemist Jul 31, 2024

JaGeo Jul 31, 2024

QuantumChemist Jul 31, 2024 •

edited

Loading

JaGeo Jul 31, 2024

QuantumChemist Jul 31, 2024

QuantumChemist Jul 31, 2024

JaGeo Jul 31, 2024 •

edited

Loading

QuantumChemist Jul 31, 2024

JaGeo Jul 31, 2024

JaGeo Jul 31, 2024

QuantumChemist Jul 31, 2024 •

edited

Loading

QuantumChemist Jul 31, 2024 •

edited

Loading

QuantumChemist Jul 31, 2024

QuantumChemist Jul 31, 2024

QuantumChemist Jul 31, 2024

JaGeo Jul 31, 2024

QuantumChemist Jul 31, 2024

QuantumChemist commented Aug 26, 2024 •

edited

Loading

QuantumChemist Aug 26, 2024 •

edited

Loading

QuantumChemist commented Aug 27, 2024

QuantumChemist Aug 27, 2024 •

edited

Loading

JaGeo commented Aug 27, 2024

Implementation of adaptive supercell settings and improvement of the documentation #80

Implementation of adaptive supercell settings and improvement of the documentation #80

Conversation

QuantumChemist commented Jul 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuantumChemist Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JaGeo Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuantumChemist Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

QuantumChemist Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuantumChemist commented Aug 26, 2024 • edited Loading

QuantumChemist Aug 26, 2024 • edited Loading

Choose a reason for hiding this comment

QuantumChemist commented Aug 27, 2024

QuantumChemist Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

JaGeo commented Aug 27, 2024

QuantumChemist commented Jul 11, 2024 •

edited

Loading

QuantumChemist Jul 31, 2024 •

edited

Loading

JaGeo Jul 31, 2024 •

edited

Loading

QuantumChemist Jul 31, 2024 •

edited

Loading

QuantumChemist Jul 31, 2024 •

edited

Loading

QuantumChemist commented Aug 26, 2024 •

edited

Loading

QuantumChemist Aug 26, 2024 •

edited

Loading

QuantumChemist Aug 27, 2024 •

edited

Loading