Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/enable headers only clone #567

Merged
merged 8 commits into from
Dec 16, 2024
Merged

Feature/enable headers only clone #567

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
49101f9
Enable header only clone
sandorkertesz Dec 13, 2024
c36982d
Enable header only clone
sandorkertesz Dec 13, 2024
c4d8a7f
Enable header only clone
sandorkertesz Dec 13, 2024
a5e3b5f
Enable header only clone
sandorkertesz Dec 14, 2024
daae41c
Merge branch 'develop' into feature/enable-headers-only-clone
sandorkertesz Dec 16, 2024
ff5d6d7
Enable header only clone
sandorkertesz Dec 16, 2024
b62eb98
Merge branch 'develop' into feature/enable-headers-only-clone
sandorkertesz Dec 16, 2024
012536e
Enable header only clone
sandorkertesz Dec 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 86 additions & 23 deletions docs/examples/grib_metadata_object.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
"tags": []
},
"source": [
"We will work with a GRIB file containing 6 messages. First we ensure the example file is available, then read the file with :ref:`from_source() <data-sources-file>`."
"In this notebook we will work with a GRIB file containing 6 messages. First we ensure the example file is available, then read the file with :ref:`from_source() <data-sources-file>`."
]
},
{
Expand Down Expand Up @@ -137,7 +137,7 @@
{
"data": {
"text/plain": [
"<earthkit.data.readers.grib.metadata.GribFieldMetadata at 0x294e37ee0>"
"<earthkit.data.readers.grib.metadata.GribFieldMetadata at 0x2aba44130>"
]
},
"execution_count": 3,
Expand Down Expand Up @@ -246,21 +246,84 @@
},
"tags": []
},
"outputs": [],
"source": [
"md_copy = md.override()"
]
},
{
"cell_type": "raw",
"id": "1c511db3-f841-4936-a7cb-25dd3e3a1010",
"metadata": {
"editable": true,
"raw_mimetype": "text/restructuredtext",
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"By default :py:meth:`~data.readers.grib.metadata.GribMetadata.override` is called with the ``headers_only_clone=True`` option to clone a new GRIB handle with all the data values (and some related information) removed. With this the resulting object can be significantly smaller, especially if the data section is large. The downside is that now the value related keys either cannot be accessed or give back wrong values. E.g when using the \"average\" key we get:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "c9ab97f0-40b7-403e-b7c9-6ce16c894d7d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<earthkit.data.readers.grib.metadata.StandAloneGribMetadata at 0x294e7cee0>"
"(279.70703560965404, 47485.4296875)"
]
},
"execution_count": 5,
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"md_copy = md.override()\n",
"md_copy"
"md[\"average\"], md_copy[\"average\"]"
]
},
{
"cell_type": "markdown",
"id": "fe340c05-ac04-4f7d-8c43-3ed1f415a979",
"metadata": {},
"source": [
"To get a copy without shrinking the GRIB handle use ``headers_only_clone=False``."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "a789bb5e-b493-45cd-b7ce-e5cb8b34e125",
"metadata": {},
"outputs": [],
"source": [
"md_copy_full = md.override(headers_only_clone=False)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "3b030549-2a7e-494e-bd46-0a4cd19a09ad",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"279.70703560965404"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"md_copy_full[\"average\"]"
]
},
{
Expand Down Expand Up @@ -294,7 +357,7 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 9,
"id": "40c6d232-03de-402b-82bf-8647e8a7bece",
"metadata": {
"editable": true,
Expand All @@ -310,7 +373,7 @@
"('z', 850)"
]
},
"execution_count": 6,
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -336,7 +399,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 10,
"id": "ef78a3ec-4ea2-4ff5-8c90-e60b5e07e77f",
"metadata": {
"editable": true,
Expand All @@ -352,7 +415,7 @@
"('t', 1000)"
]
},
"execution_count": 7,
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
Expand Down Expand Up @@ -386,7 +449,7 @@
"tags": []
},
"source": [
"GRIB metadata objects play a part in building new fieldlist from (altered) values and metadata."
"GRIB metadata objects play a part in building new fieldlists from (altered) values and metadata."
]
},
{
Expand All @@ -405,7 +468,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 11,
"id": "cb59ad5f-c48b-4943-984d-3abdf48fda8d",
"metadata": {
"editable": true,
Expand All @@ -427,7 +490,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 12,
"id": "85c32bfb-c929-404f-add9-9adae40418d2",
"metadata": {
"editable": true,
Expand Down Expand Up @@ -496,7 +559,7 @@
"0 an 0 regular_ll "
]
},
"execution_count": 9,
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -520,12 +583,12 @@
}
},
"source": [
"Please note that the resulting :py:class:`~data.sources.array_list.ArrayFieldList` always contains a :py:class:`~data.readers.grib.metadata.RestrictedGribMetadata` object for each field. These objects possess their own GRIB handles, which is ensured by creating a copy with ``override()`` when needed. On top of that metadata access is limited to keys not related to data values. Getting metadata on any other keys will throw an exception. "
"The resulting fieldlist contains an :py:class:`~data.sources.array_list.ArrayField`, which is composed of a numpy array storing the values and a metadata object owning its own GRIB handle with a trimmed down data section. Since the values array is decoupled from the GRIB handle stored in the metadata object, accessing metadata keys related to the data values is forbidden. Getting metadata on these keys will throw an exception. "
]
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 13,
"id": "c6fe87ed-ee88-4f4d-a2b6-9401b364e2df",
"metadata": {
"editable": true,
Expand All @@ -538,10 +601,10 @@
{
"data": {
"text/plain": [
"<earthkit.data.readers.grib.metadata.RestrictedGribMetadata at 0x294e7dc60>"
"<earthkit.data.readers.grib.metadata.RestrictedGribMetadata at 0x2aba46ec0>"
]
},
"execution_count": 10,
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -552,7 +615,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 14,
"id": "27686ac4-9382-4916-ad0e-be96a649d034",
"metadata": {
"editable": true,
Expand All @@ -568,7 +631,7 @@
"'Wind speed'"
]
},
"execution_count": 11,
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -579,7 +642,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 15,
"id": "dc28fa77-4020-431f-ad37-e480a69f9d7f",
"metadata": {
"editable": true,
Expand Down Expand Up @@ -620,7 +683,7 @@
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": 16,
"id": "8eab3462-3661-4fc1-9d23-8be05dc99cd8",
"metadata": {
"editable": true,
Expand All @@ -636,7 +699,7 @@
"7.450183054360252"
]
},
"execution_count": 13,
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
Expand Down
Loading
Loading