Replace pythonjob task by normal task, drop usage of pickle_node
#35
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In my workstation, it took ~7 mins to run the QE workflow using AiiDA-WorkGraph, while other engines only take ~ 1 mins. The main reason is that the
PythonJob
task was used to auto-serialize inputs/outputs and preserve data provenance. However,PythonJob
is designed to run the Python function on a remote computer. It uploads inputs to a remote folder and retrieves outputs to the local computer. In this project, it's overkill. In the latest aiida-workgraph, the normal keep provenance too. So this PR drops the usage of the PythonJob task.Another reason is that, the
pickled_node
task is created for each data node, in this PR, when loading the workflow from a json file, the normal task can also serialize the inputs/outputs in an automatic way, so thepickle_node
is not needed anymore. When writing the workflow using AiiDA, we prepare the AiiDA data nodes as inputs, sopickle_node
is not needed either.This PR reduces execution time from ~7 mins to ~ 1.5 mins.
This PR needs
aiida-workgraph>=0.5.1
.