Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support lazy regridding #80

Merged
merged 7 commits into from
Aug 19, 2021
Merged

Conversation

stephenworsley
Copy link
Contributor

Provides lazy regridding through essentially the same means as in iris (introduced in this PR SciTools/iris#3701).

@codecov
Copy link

codecov bot commented Jun 2, 2021

Codecov Report

Merging #80 (f23a1d3) into main (ef0268e) will increase coverage by 0.03%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #80      +/-   ##
==========================================
+ Coverage   98.82%   98.85%   +0.03%     
==========================================
  Files          14       14              
  Lines         678      699      +21     
==========================================
+ Hits          670      691      +21     
  Misses          8        8              
Impacted Files Coverage Δ
esmf_regrid/_esmf_sdo.py 95.83% <100.00%> (ø)
esmf_regrid/schemes.py 98.93% <100.00%> (+0.03%) ⬆️
.../schemes/test_regrid_rectilinear_to_rectilinear.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef0268e...f23a1d3. Read the comment docs.

@zklaus zklaus modified the milestones: v0.1.0, v0.2.0 Jun 2, 2021
esmf_regrid/_esmf_sdo.py Outdated Show resolved Hide resolved
Copy link
Member

@jamesp jamesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks a good, a couple of minor points inline

esmf_regrid/schemes.py Show resolved Hide resolved
esmf_regrid/schemes.py Show resolved Hide resolved
@stephenworsley
Copy link
Contributor Author

When this is merged, the performance test in #100 will be measuring the set up for lazy regridding but not the final realisation of the data. The realisation will also be worth measuring after this is merged in (I don't think it will make sense to add such a test before this is merged in since then it would be measuring the realisation of an already realised array).

@stephenworsley
Copy link
Contributor Author

stephenworsley commented Aug 18, 2021

The following excerpt from the benchmarking CI illustrates the effect these changes have on performance:
image
The actual regridding step now takes up a negligible amount of time. The regridding calculations are now done during realisation of the data (previously the data was already realised, this is why the before shows up as "failed"). This calculation step now takes almost exactly half the time it did before, probably due to dask parallelisation (compare the before of time_lazy_regridding with the after of time_regridding_realisation).

@jamesp
Copy link
Member

jamesp commented Aug 19, 2021

OK I'm happy with this. Any last changes @stephenworsley ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants