Dask version #252
Open
Dask version #252
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, created them in numpy and then converted to dask. Rather created from parts of dask arrays. Hopefully this reduces data movement.
In the distributed case, very little processor time is spent as its mostly waiting on a future. time.time(), though less technically less accurate, will give the waiting time, which is correct.
Current code needs master to run properly.
To stand for visibility row, as opposed to antenna row, to be introduced later.
This commit breaks benchmark.py A dimension containing consecutive antenna values for multiple timesteps. This is related to the unique 'utime' dimension and visibility row dimension 'vrow'. For example, there are 3 unique timesteps below. Timestep 0 has 4 visibility and 5 antenna rows associated with it, while timestep 1 has 3 visibility and 6 antenna rows associated with it. 0 1 2 unique time 1 1 2 5 3 7 2 4 visibility row 2 3 3 4 5 1 4 3 1 2 3 4 5 1 2 3 4 5 7 3 4 antenna row
* Add RIME op tensorflow source to MANIFEST.in * Deprecate ez_setup.py script See pypa/setuptools#581. * Include missing setup.cfg * Pin python-casacore==2.1.2 for now
Allow more recent versions.
sjperkins
force-pushed
the
dask-tf-1.4
branch
from
ffbde2a to
4cd1967
Compare
|
Can one of the admins verify this patch? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR for the new dask version
RIME Pipeline Configuration
{ 'lm': (nsrc, 2) }{ 'complex_phase': (nsrc, ntime, na, nchan) }xarray dataset from Measurement Set (available in https://github.com/ska-sa/xarray-ms/).
xarray dataset from FITS file (available in https://github.com/ska-sa/xarray-fits/).
Input xarray dataset. Massage MS and FITS datasets into Montblanc dataset.
Use tensorflow Dataset API for input into the tensorflow graph. The StagingArea/Queues are complicated and unwieldy and the Dataset API now has a GPU prefetch
of the data, to be consumed internally via Iterators, from Datasets. There's no trivial fit
with dask's worker mechanism. Likely need to create a C++ op cut and pasted from Dataset.from_tensor which consumes an infinite series of tensors.