PSA: Pydantic v1/v2 and the QCArchive + Psi4 stack

[loriab]

software	pydantic v1 only (API v1) 1	pydantic v1/v2 tolerant (API v1)	pydantic v2 only (API v2)
QCElemental	thru 0.25.1	0.26.0 thru 0.27.1	WIP #321
QCEngine	thru 0.26.0	0.27.0 thru 0.29.0	WIP MolSSI/QCEngine#425
QCFractal next	0.5 beta13 thru 0.51	WIP MolSSI/QCFractal#787 (upcoming v0.52)
Psi4 2	v1.6 thru v1.8.0	v1.8.1 _2, v1.8.2	psi4/psi4#3034

EDIT 19 Jun 2024: The plan is to update schema v2 and pydantic v2 at the same time at 0.70

Footnotes

1. "v1 only" describes the state of the code. conda packages may not be constrained to only solve with pydantic v1. ↩

2. psi4 before v1.6 didn't use pydantic directly ↩

[loriab]

Thanks to the Center for Scientific Software Engineering at Georgia Tech, I'm going to be able to tackle this pydantic v1/v2 problem. General strategy is:

• Pydantic
  • you will be required to use pydantic v2 with subsequent releases of qcel/qcng. (don't panic, you won't need to use the v2 API just the v2 packages (from which v1 API can be imported))
  • from qcelemental, the existing QCSchema models (mostly schema_version=1 except Mol that's 2) will continue to be available as from qcelemental.models import AtomicInput, etc., and will continue to use pydantic v1 API. To fend off pydantic v2 API for a while, you can also from qcelemental.models.v1 import AtomicInput
  • a new "v2" version of QCSchema that is written based on Pydantic v2 API according to the PRs listed in the headmatter will be available as from qcelemental.models.v2 import AtomicInput, etc.
  • QCEngine will run the either longstanding QCSchema v1 or newfangled QCSchema v2. you'll get back whichever version you put in.
• Layout
  • Since pydantic has necessitated the upheaval above, we also want to take the opportunity to do some of the layout rearrangements that have been long in discussion (see next branch, [DISCUSSION] Reduce redundancy in model specifications #264, mini meeting at MQM 2022 at Virginia Tech). In particular:
    • (a) storing <CalcType>Input on <CalcType>Result rather than the latter inheriting from the former. This helps restart and more explicitly handles programs messing with Molecule orientation.
    • (b) having a few base classes (probably BaseInput, BaseSpecification, BaseResult, BaseProperties) so that misc. fields don't get left out (e.g., id, protocols, return_result, return_gradient, respectively).
    • (c) separate "what molecule" from "how to compute" so there aren't largely redundant schema (AtomicInput/QCInputSpecification and OptimizationInput/OptimizationSpecification) and they become more composable
  • See image below for anticipated form for AtomicInput/AtomicResult
  • These changes are deliberately layout-only rather than adding features
• this plan is as undisruptive as I can devise while still breaking the impasse. I'm glad to hear opinions or concerns or show sketches for the other models or discuss further.
• there may be another normal release of qcel/qcng before the above take effect

@Lnaden @coltonbh @mattwthompson @berquist @awvwgk @bennybp

[coltonbh]

Hi @loriab thanks for doing this work. I built out the models from the next branch for my own use and use in our lab some time ago, used them extensively, then updated the designs based on our collective experience. My thoughts are now baked into the qcio (Quantum Chemistry Input/Output) package. You can find documentation here: https://qcio.coltonhicks.com. Feel free to use any of the ideas you find useful. Happy to chat more details if at all helpful.

[loriab]

Hi @coltonbh, yes, we've been sadly slow at proceeding on this, and I'm glad you've got a working relative of QCSchema so as to not impede research. I'll reach out further in a couple weeks to make sure I understand some of your design decisions. I think you're right that the layout (flat) that makes a convenient API interface doesn't necessarily make compact and descriptive models for schema.

[mattwthompson]

Just so I understand correctly, the from qcelemental.models.v1 import * hatch isn't live yet, but would be in the first release (or at least the first one associated with these effort)?

[loriab]

Just so I understand correctly, the from qcelemental.models.v1 import * hatch isn't live yet, but would be in the first release (or at least the first one associated with these effort)?

Correct, the qcelemental.models bifurcation is in a local branch at present. Expected in a release in late October.