Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chunked fields operational #632

Merged
merged 86 commits into from
Sep 4, 2019
Merged

Chunked fields operational #632

merged 86 commits into from
Sep 4, 2019

Conversation

delandmeterp
Copy link
Contributor

@delandmeterp delandmeterp commented Aug 22, 2019

This PR splits the field_data into chunks.

Field.data is not a np.array anymore but a lazily loaded da.array.
If a JIT kernel needs to interpolate data within a chunk, this chunk is loaded into memory.
In Scipy, pset.execute() first converts all Field.data into np.arrays.

To do

  • What to do with Field.gradient()
  • Cut Field.chunk_data() into an initial call and what is called during simu
  • Cleaning
  • Field.chunk_data should be moved to Field.grid.chunk_data
  • We should catch the following situation:
pset.execute(kernel=InterpOnlyU, ...)
pset.execute(kernel=InterpOnlyUandV, ...)

where grid.load_chunk_data will not be matching with loaded V chunks
-> solution here is to attach load_chunk_data to the field itself...
NO, it does not work, since really not efficient
-> we keep it on grid, but in chunk_data, we also load the data also if it's not loaded yet, but then we also need to set chunk_data back to None at the pset.execute() if it't not used by kernels!

  • What to do with data_array interpolated in scipy. it can happen, especially when called out of a pset.execute (at particle.variable init for example). Putting tests for da.array and compute everywhere?
  • repeat kernel are correctly interpreted with pset.execute(dt=0) ?
  • add 3D example plot to documentation notebook

since scipy interp needs np.arrays so far
(in computeTimeChunk)
but field shares a grid with another one that does
This forces deletion of objects that were stuck in a reference loop
@delandmeterp delandmeterp merged commit 6c72204 into master Sep 4, 2019
@delandmeterp delandmeterp deleted the chunked_fields_operational branch September 4, 2019 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants