Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add nbytes property #227

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add nbytes property #227

wants to merge 1 commit into from

Conversation

TomNicholas
Copy link
Collaborator

@TomNicholas TomNicholas commented Aug 23, 2024 •
edited
Loading 

In [1]: import virtualizarr as vz

In [2]: vds = vz.open_virtual_dataset('air.nc', indexes={})

In [3]: vds.nbytes
Out[3]: 128

In [4]: vds
Out[4]: 
<xarray.Dataset> Size: 128B
Dimensions:  (time: 2920, lat: 25, lon: 53)
Coordinates:
    time     (time) float32 32B ManifestArray<shape=(2920,), dtype=float32, c...
    lon      (lon) float32 32B ManifestArray<shape=(53,), dtype=float32, chun...
    lat      (lat) float32 32B ManifestArray<shape=(25,), dtype=float32, chun...
Data variables:
    air      (time, lat, lon) int16 32B ManifestArray<shape=(2920, 25, 53), d...
Attributes:
    Conventions:  COARDS
    description:  Data is from NMC initialized reanalysis\n(4x/day).  These a...
    platform:     Model
    references:   http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly...
    title:        4x daily NMC reanalysis (1948)
  • Closes What should nbytes return? #167
  • Tests added
  • Tests passing
  • Full type hint coverage
  • Changes are documented in docs/releases.rst
  • New functions/methods are listed in api.rst
  • New functionality has documentation

TomNicholas
TomNicholas commented Aug 23, 2024
View reviewed changes
virtualizarr/manifests/manifest.py

Note this is not the size of the referenced chunks if they were actually loaded into memory,
this is only the size of the pointers to the chunk locations.
If you were to load the data into memory it would be ~1e6x larger for 1MB chunks.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can probably be a little more precise than this, though it does depend on the length of the stored paths.

@TomNicholas TomNicholas mentioned this pull request Sep 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

What should nbytes return?
1 participant
@TomNicholas