Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a lazy-loading mechanic #236

Merged
merged 17 commits into from
Feb 22, 2022
Merged

Add a lazy-loading mechanic #236

merged 17 commits into from
Feb 22, 2022

Conversation

PythonFZ
Copy link
Member

@PythonFZ PythonFZ commented Feb 17, 2022

If you have nested Node structures you don't want to load everything but only the files you really need. With lazy the file will only be read when accessed via __get__ and therefore the memory will not be filled with unneccessary data.

Binder

Explanation of changes can best be found at https://github.com/zincware/ZnTrack/blob/add_lazy/examples/docs/09_lazy.ipynb

TODOs

  • Add documentation
  • check that config.lazy does anything
  • fix notebooks

@codecov
Copy link

codecov bot commented Feb 17, 2022

Codecov Report

Merging #236 (29f8446) into main (d8b9eff) will decrease coverage by 0.81%.
The diff coverage is 61.22%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #236      +/-   ##
==========================================
- Coverage   69.44%   68.62%   -0.82%     
==========================================
  Files          23       23              
  Lines         985     1023      +38     
  Branches      183      195      +12     
==========================================
+ Hits          684      702      +18     
- Misses        284      298      +14     
- Partials       17       23       +6     
Flag Coverage Δ
unittests 68.62% <61.22%> (-0.82%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
zntrack/utils/utils.py 70.10% <0.00%> (ø)
zntrack/core/dvcgraph.py 67.14% <33.33%> (-1.75%) ⬇️
zntrack/core/base.py 82.43% <50.00%> (-5.81%) ⬇️
zntrack/utils/__init__.py 85.71% <66.66%> (-14.29%) ⬇️
zntrack/core/zntrackoption.py 83.33% <72.72%> (-5.31%) ⬇️
zntrack/dvc/__init__.py 98.33% <100.00%> (ø)
zntrack/utils/config.py 100.00% <100.00%> (ø)
zntrack/descriptor/__init__.py 85.71% <0.00%> (-11.43%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8b9eff...29f8446. Read the comment docs.

@PythonFZ PythonFZ marked this pull request as ready for review February 18, 2022 16:24
Copy link
Member

@SamTov SamTov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a really nice addition. I looked over the notebooks and maybe some more comments to it for additional support. On an unrelated topic, is there much one can do about the excess stdout from DVC?

docs/source/_overview/09_lazy.nblink Outdated Show resolved Hide resolved
@PythonFZ PythonFZ merged commit 9edb7f6 into main Feb 22, 2022
@PythonFZ PythonFZ deleted the add_lazy branch February 22, 2022 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AttributeError when loading a Stage that is not present in zntrack.json
2 participants