making_a_colocated_data_object_with_pyaerocom.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# WIP: Making a ColocatedData object with Pyaerocom\n",
    "Lewis Blake\n",
    "lewisb@met.no"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a tutorial showing how the current version of pyaerocom can be used for creating colocated data objects which minimizes the risk of processing errors. It is a preliminary tutorial, and therefore subject to change as development continues. As the interface becomes better defined, this should simplify the process and changes will be reflected in updated versions of this tutorial. Feedback welcome."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting started\n",
    "You may want to start by creating a Python virtual environment before begining and installing pyaerocom into it.\n",
    "\n",
    "\n",
    "```console\n",
    "python3 -m venv --prompt cams2_82 .venv\n",
    "source .venv/bin/activate\n",
    "pip install pyaerocom\n",
    "pya --help\n",
    "``` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import pyaerocom\n",
    "import xarray as xr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading example data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we have done the collocation and stored the results in some object. As an example we will assume this is a netcdf file called `my_colocated_data.nc`. For simplcity of the tutorial, such a colocated data object file is included with pyaerocom. In fact there are a few, however there is a `my_colocated_data.nc` specifically included for this tutorial. Pyaerocom has a command line interface which is usefule for this. On the command line there are several functions available.\n",
    "\n",
    "Our goal is to create a `ColocatedData` object named `od550aer_od550aer_MOD-IFS-OSUITE_REF-AeronetL1.5-d_20130101_20221231_daily_ALL-wMOUNTAINS.nc`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1mpya [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m                                        \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " 🦄 Pyaerocom Command Line Interface                                            \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-version\u001b[0m             \u001b[1;32m-V\u001b[0m                                                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-install\u001b[0m\u001b[1;36m-completion\u001b[0m            Install completion for the current shell.    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-show\u001b[0m\u001b[1;36m-completion\u001b[0m               Show completion for the current shell, to    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                                 copy it or customize the installation.       \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m                          Show this message and exit.                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36maeroval       \u001b[0m\u001b[1;36m \u001b[0m Run an AeroVal experiment as described in a json config file \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mbrowse        \u001b[0m\u001b[1;36m \u001b[0m Browse database (e.g., browse \u001b[1;33m<DATABASE>\u001b[0m)                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mclearcache    \u001b[0m\u001b[1;36m \u001b[0m Delete cached data objects                                   \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mgetsampledata \u001b[0m\u001b[1;36m \u001b[0m Downloads a minimal sample dataset.                          \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36minit          \u001b[0m\u001b[1;36m \u001b[0m init ~/MyPyaerocom directory and copy the default paths.ini  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m               \u001b[0m there                                                        \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mlistcache     \u001b[0m\u001b[1;36m \u001b[0m List cached data objects                                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mppiaccess     \u001b[0m\u001b[1;36m \u001b[0m Check if MET Norway's PPI can be accessed                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! pya --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ensure that pya/pyaerocom is at least version 0.25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🦄 pyaerocom 0.25.0.dev0\n"
     ]
    }
   ],
   "source": [
    "! pya -V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `getsampledata` command will download the neccessary files into `./data/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "!  pya getsampledata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "filepath = \"./data/testdata-minimal/modeldata/colocated_netcdf/my_colocated_data.nc\"\n",
    "data = xr.open_dataarray(filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting to ColocatedData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that while included in pyaerocom, this data is not actually CF-compliant. For example, latitude and longitude have units \"degrees\", but really should be \"degrees_north\" and \"degrees_east\" respectively. Let's fix that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.latitude.attrs[\"units\"] = \"degrees_north\"\n",
    "data.longitude.attrs[\"units\"] = \"degrees_east\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general format for `ColocateData` objects file name is `\"{obs_var_name}_{mod_var_name}_MOD-{model_name}_REF-{obs_name}_{date_start]_{date_end}_{ts_type}_{filter_name}.nc\"`\n",
    "\n",
    "- `obs_var_name`: Aerocom name for the obs variable. Can be found by searching here: https://github.com/metno/pyaerocom/blob/main-dev/pyaerocom/data/variables.ini\n",
    "- `mod_var_name`: Aerocom name for model variable.\n",
    "- `model_name`: Name of the model \n",
    "- `obs_name`: Name of the observation network\n",
    "- `date_start`: Start of the timerseries in the colocated array formated YYYYMMDD\n",
    "- `date_end`: End of the timeseries in the colocated array formated YYYYMMDD\n",
    "- `ts_type`: Time series type (frequency), for example \"daily\" or \"monthly\".\n",
    "- `filter_name`: name of the region filter applied. If no spatial filering applied, should be the string \"ALL-wMOUNTAINS\".\n",
    "\n",
    "Note that the fields provided here in the name **must** match their associated values in the metadata below. For exmaple, the `model_name` here in the name needs to match the `model_name` in the metadata.\n",
    "\n",
    "A valid (and encouraged) way to create a `ColocatedData` object is to convert it into an xarray DataArray with the correct structure, dimensions, and metadata. You will have your object stored in some format. The task of CAMS2_82 partners will then be to do translation of their colocated data into this format."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the current design, such A `ColocatedData` object comprises 3 or 4 dimensions.\n",
    "\n",
    "- The first dimension (`data_source`, index 0) is ALWAYS length 2 and specifies the two datasets that were co-located (index 0 is obs, index 1 is model). \n",
    "- The second dimension is `time`. \n",
    "- 3rd dimension is `station_name` in case of 3D colocated data. For 4D colocated data the 3rd and 4th dimension are latitude and longitude, respectively.\n",
    "\n",
    "3D colocated data is typically created when a model is colocated with station based ground based observations. 4D colocated data is created when a model is colocated with another model or satellite observations, that cover large parts of Earth's surface (other than discrete lat/lon pairs in the case of ground based station locations)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may be curious about the valid potential options available to us. We can check the fields in a few relevant classes to determine what these options are and to check which data types they are allowed to be.\n",
    "\n",
    "The documentation for the `ColocationSetup` class can be found here: https://pyaerocom.readthedocs.io/en/latest/api.html#pyaerocom.colocation.colocation_setup.ColocationSetup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyaerocom import ColocationSetup, ColocatedData"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_id': FieldInfo(annotation=Union[str, NoneType], required=True),\n",
       " 'obs_id': FieldInfo(annotation=Union[str, NoneType], required=True),\n",
       " 'obs_vars': FieldInfo(annotation=Union[tuple[str, ...], str], required=True),\n",
       " 'ts_type': FieldInfo(annotation=str, required=True),\n",
       " 'start': FieldInfo(annotation=Union[Timestamp, int, str, NoneType], required=True),\n",
       " 'stop': FieldInfo(annotation=Union[Timestamp, int, str, NoneType], required=True),\n",
       " 'obs_config': FieldInfo(annotation=Union[PyaroConfig, NoneType], required=False, default=None),\n",
       " 'OBS_VERT_TYPES_ALT': FieldInfo(annotation=dict[str, str], required=False, default={'Surface': 'ModelLevel', '2D': '2D'}),\n",
       " 'CRASH_ON_INVALID': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'FORBIDDEN_KEYS': FieldInfo(annotation=list[str], required=False, default=['var_outlier_ranges', 'var_ref_outlier_ranges', 'remove_outliers']),\n",
       " 'filter_name': FieldInfo(annotation=str, required=False, default='ALL-wMOUNTAINS'),\n",
       " 'basedir_coldata': FieldInfo(annotation=Union[str, Path], required=False, default='/home/heikok/MyPyaerocom/colocated_data', validate_default=True),\n",
       " 'save_coldata': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'obs_name': FieldInfo(annotation=Union[str, NoneType], required=False, default=None),\n",
       " 'obs_data_dir': FieldInfo(annotation=Union[Path, str, NoneType], required=False, default=None),\n",
       " 'obs_use_climatology': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'obs_cache_only': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'obs_vert_type': FieldInfo(annotation=Union[str, NoneType], required=False, default=None),\n",
       " 'obs_ts_type_read': FieldInfo(annotation=Union[str, dict, NoneType], required=False, default=None),\n",
       " 'obs_filters': FieldInfo(annotation=dict, required=False, default={}),\n",
       " 'colocation_layer_limits': FieldInfo(annotation=Union[tuple[LayerLimits, ...], NoneType], required=False, default=None),\n",
       " 'profile_layer_limits': FieldInfo(annotation=Union[tuple[LayerLimits, ...], NoneType], required=False, default=None),\n",
       " 'read_opts_ungridded': FieldInfo(annotation=Union[dict, NoneType], required=False, default={}),\n",
       " 'model_name': FieldInfo(annotation=Union[str, NoneType], required=False, default=None),\n",
       " 'model_data_dir': FieldInfo(annotation=Union[Path, str, NoneType], required=False, default=None),\n",
       " 'model_read_opts': FieldInfo(annotation=Union[dict, NoneType], required=False, default={}),\n",
       " 'model_use_vars': FieldInfo(annotation=Union[dict[str, str], NoneType], required=False, default={}),\n",
       " 'model_rename_vars': FieldInfo(annotation=Union[dict[str, str], NoneType], required=False, default={}),\n",
       " 'model_add_vars': FieldInfo(annotation=Union[dict[str, tuple[str, ...]], NoneType], required=False, default={}),\n",
       " 'model_to_stp': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'model_ts_type_read': FieldInfo(annotation=Union[str, dict, NoneType], required=False, default=None),\n",
       " 'model_read_aux': FieldInfo(annotation=Union[dict[str, dict[Literal['vars_required', 'fun'], Union[list[str], Callable]]], NoneType], required=False, default={}),\n",
       " 'model_use_climatology': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'gridded_reader_id': FieldInfo(annotation=dict[str, str], required=False, default={'model': 'ReadGridded', 'obs': 'ReadGridded'}),\n",
       " 'flex_ts_type': FieldInfo(annotation=bool, required=False, default=True),\n",
       " 'min_num_obs': FieldInfo(annotation=Union[dict, int, NoneType], required=False, default=None),\n",
       " 'resample_how': FieldInfo(annotation=Union[str, dict, NoneType], required=False, default='mean'),\n",
       " 'obs_remove_outliers': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'model_remove_outliers': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'obs_outlier_ranges': FieldInfo(annotation=Union[dict[str, tuple[float, float]], NoneType], required=False, default={}),\n",
       " 'model_outlier_ranges': FieldInfo(annotation=Union[dict[str, tuple[float, float]], NoneType], required=False, default={}),\n",
       " 'zeros_to_nan': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'harmonise_units': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'regrid_res_deg': FieldInfo(annotation=Union[float, RegridResDeg, NoneType], required=False, default=None),\n",
       " 'colocate_time': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'reanalyse_existing': FieldInfo(annotation=bool, required=False, default=True),\n",
       " 'raise_exceptions': FieldInfo(annotation=bool, required=False, default=False),\n",
       " 'keep_data': FieldInfo(annotation=bool, required=False, default=True),\n",
       " 'add_meta': FieldInfo(annotation=Union[dict, NoneType], required=False, default={}),\n",
       " 'model_kwargs': FieldInfo(annotation=dict, required=False, default={}),\n",
       " 'main_freq': FieldInfo(annotation=str, required=False, default='monthly'),\n",
       " 'freqs': FieldInfo(annotation=list[str], required=False, default=['monthly', 'yearly'])}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ColocationSetup.model_fields"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The documentation for the `ColocatedData` object can be found here: https://pyaerocom.readthedocs.io/en/latest/api.html#pyaerocom.colocation.colocated_data.ColocatedData"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': FieldInfo(annotation=Union[Path, str, DataArray, ndarray, NoneType], required=False, default=None)}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ColocatedData.model_fields"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build this `ColocatedData` oibject by hand. We start with the array: `arr`, assuming its dimension matches the structure described above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = data.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 365, 665)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the coordinates..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_source = ['AeronetSunV3Lev1.5.daily', 'ECMWF_OSUITE']\n",
    "time = pd.date_range(start=\"2013-01-01\", end=\"2013-12-31\", freq = \"D\") # daily frequency\n",
    "station_names = list(data.coords[\"station_name\"].data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "lats = data.latitude.data\n",
    "lons = data.longitude.data\n",
    "alts = data.altitude.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "coords = {\n",
    "    \"data_source\": data_source,\n",
    "    \"time\": time,\n",
    "    \"station_name\": station_names,\n",
    "    \"latitude\": (\"station_name\", lats),\n",
    "    \"longitude\": (\"station_name\", lons),\n",
    "    \"altitude\": (\"station_name\", alts),\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we cheat a bit with the station names as it is a very long list, so we get it directly from the `data` object, but in principle this is a list of string that you have somewhere. We also cheat a bit with the latitudes, longitudes, and altitudes, but again these are arrays that you have somewhere."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to ensure safe processing, we require **all** the following metadata fields. Here is an example of a dictionary containing some example values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = {\n",
    "    \"obs_vars\": \"od550aer\", # must match boths obs_var_name and mod_var_name in filename\n",
    "    \"ts_type\": \"daily\",\n",
    "    \"filter_name\": \"ALL-wMOUNTAINS\",\n",
    "    \"ts_type_src\": [\"hourly\", \"hourly\"], # must be a list of strings, a ts type for each data_source\n",
    "    \"var_units\": [\"1\", \"1\"], # must have units for each data_source\n",
    "    \"data_level\": 3,\n",
    "    \"revision_ref\": \"20221027\",\n",
    "    \"from_files\": ['ECMWF_OSUITE.daily.od550aer.2013.nc', 'ECMWF_OSUITE.daily.od550aer.2014.nc', 'ECMWF_OSUITE.daily.od550aer.2015.nc', 'ECMWF_OSUITE.daily.od550aer.2016.nc', 'ECMWF_OSUITE.daily.od550aer.2017.nc', 'ECMWF_OSUITE.daily.od550aer.2018.nc', 'ECMWF_OSUITE.daily.od550aer.2019.nc', 'ECMWF_OSUITE.daily.od550aer.2020.nc', 'ECMWF_OSUITE.daily.od550aer.2021.nc', 'ECMWF_OSUITE.daily.od550aer.2022.nc'],\n",
    "    \"from_files_ref\": \"None\",\n",
    "    \"colocate_time\": 0, # default is 0\n",
    "    \"obs_is_clim\": 0, # default is 0\n",
    "    \"pyaerocom\": pyaerocom.__version__,\n",
    "    \"CONV!min_num_obs\": str(dict(monthly=dict(daily=3))), # yes, needed\n",
    "    \"resample_how\": \"mean\", # default\n",
    "    \"obs_name\": \"AeronetL1.5-d\", # must match obs_name in filename\n",
    "    \"vert_code\": \"Column\", # Literal[\"Column\", \"Profile\", \"Surface\", \"ModelLevel\"]. default is \"Surface\"\n",
    "    \"diurnal_only\": 0, # default is 0\n",
    "    \"zeros_to_nan\": 1, # default is 0\n",
    "}\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check that we have the right data types by passing this metadata object to Pyaerocom's `ColocationSetup` class. Under the hood Pyaerocom has checked that we provided the correct data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_stp = ColocationSetup(**metadata)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we put it all in an xarray DataArray..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "da = xr.DataArray(\n",
    "    data = arr,\n",
    "    dims = [\"data_source\", \"time\", \"station_name\"],\n",
    "    coords = coords,\n",
    "    attrs = metadata\n",
    ")\n",
    "da.name = \"od550aer\" # Important to give the DataArray the name of the variable."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we put that DataArray into Pyaerocom `ColocatedData`..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "coldata = ColocatedData(da)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under the hood Pyaerocom has checked that we have the correct dimensions on this object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the ColocatedData object"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save this ColocatedData object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "savename = \"od550aer_od550aer_MOD-IFS-OSUITE_REF-AeronetL1.5-d_20130101_20221231_daily_ALL-wMOUNTAINS.nc\"\n",
    "out_dir = \"./data/coldata/\"\n",
    "os.makedirs(out_dir, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./data/coldata/od550aer_od550aer_MOD-IFS-OSUITE_REF-AeronetL1.5-d_20130101_20221231_daily_ALL-wMOUNTAINS.nc'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coldata.to_netcdf(out_dir=out_dir, savename=savename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the contents of the saved ColocatedData object"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When everything is done we should we able to do a `ncdump -h` and verify its contents.\n",
    "\n",
    "\n",
    "```console\n",
    "netcdf od550aer_od550aer_MOD-IFS-OSUITE_REF-AeronetL1.5-d_20130101_20221231_daily_ALL-wMOUNTAINS {\n",
    "dimensions:\n",
    "\tdata_source = 2 ;\n",
    "\ttime = 3652 ;\n",
    "\tstation_name = 665 ;\n",
    "variables:\n",
    "\tstring data_source(data_source) ;\n",
    "\tint64 time(time) ;\n",
    "\t\ttime:units = \"days since 2013-01-01 12:00:00\" ;\n",
    "\t\ttime:calendar = \"proleptic_gregorian\" ;\n",
    "\tstring station_name(station_name) ;\n",
    "\tdouble latitude(station_name) ;\n",
    "\t\tlatitude:_FillValue = NaN ;\n",
    "\t\tlatitude:standard_name = \"latitude\" ;\n",
    "\t\tlatitude:units = \"degrees_north\" ;\n",
    "\tdouble longitude(station_name) ;\n",
    "\t\tlongitude:_FillValue = NaN ;\n",
    "\t\tlongitude:standard_name = \"longitude\" ;\n",
    "\t\tlongitude:units = \"degrees_east\" ;\n",
    "\tdouble altitude(station_name) ;\n",
    "\t\taltitude:_FillValue = NaN ;\n",
    "\tdouble od550aer(data_source, time, station_name) ;\n",
    "\t\tod550aer:_FillValue = NaN ;\n",
    "\t\tstring od550aer:data_source = \"AeronetSunV3Lev1.5.daily\", \"ECMWF_OSUITE\" ;\n",
    "\t\tstring od550aer:var_name = \"od550aer\", \"od550aer\" ;\n",
    "\t\tstring od550aer:var_name_input = \"od550aer\", \"od550aer\" ;\n",
    "\t\tod550aer:ts_type = \"daily\" ;\n",
    "\t\tod550aer:filter_name = \"ALL-wMOUNTAINS\" ;\n",
    "\t\tstring od550aer:ts_type_src = \"daily\", \"daily\" ;\n",
    "\t\tstring od550aer:var_units = \"1\", \"1\" ;\n",
    "\t\tod550aer:data_level = 3LL ;\n",
    "\t\tod550aer:revision_ref = \"20221027\" ;\n",
    "\t\tstring od550aer:from_files = \"ECMWF_OSUITE.daily.od550aer.2013.nc\", \"ECMWF_OSUITE.daily.od550aer.2014.nc\", \"ECMWF_OSUITE.daily.od550aer.2015.nc\", \"ECMWF_OSUITE.daily.od550aer.2016.nc\", \"ECMWF_OSUITE.daily.od550aer.2017.nc\", \"ECMWF_OSUITE.daily.od550aer.2018.nc\", \"ECMWF_OSUITE.daily.od550aer.2019.nc\", \"ECMWF_OSUITE.daily.od550aer.2020.nc\", \"ECMWF_OSUITE.daily.od550aer.2021.nc\", \"ECMWF_OSUITE.daily.od550aer.2022.nc\" ;\n",
    "\t\tod550aer:from_files_ref = \"None\" ;\n",
    "\t\tod550aer:colocate_time = 0LL ;\n",
    "\t\tod550aer:obs_is_clim = 0LL ;\n",
    "\t\tod550aer:pyaerocom = \"0.13.2.dev0\" ;\n",
    "\t\tod550aer:CONV\\!min_num_obs = \"{\\'monthly\\': {\\'daily\\': 3}}\" ;\n",
    "\t\tod550aer:resample_how = \"mean\" ;\n",
    "\t\tod550aer:model_name = \"IFS-OSUITE\" ;\n",
    "\t\tod550aer:obs_name = \"AeronetL1.5-d\" ;\n",
    "\t\tod550aer:vert_code = \"Column\" ;\n",
    "\t\tod550aer:diurnal_only = 0LL ;\n",
    "\t\tod550aer:coordinates = \"altitude longitude latitude\" ;\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "netcdf od550aer_od550aer_MOD-IFS-OSUITE_REF-AeronetL1.5-d_20130101_20221231_daily_ALL-wMOUNTAINS {\n",
      "dimensions:\n",
      "\tdata_source = 2 ;\n",
      "\ttime = 365 ;\n",
      "\tstation_name = 665 ;\n",
      "variables:\n",
      "\tstring data_source(data_source) ;\n",
      "\tint64 time(time) ;\n",
      "\t\ttime:units = \"days since 2013-01-01 00:00:00\" ;\n",
      "\t\ttime:calendar = \"proleptic_gregorian\" ;\n",
      "\tstring station_name(station_name) ;\n",
      "\tdouble latitude(station_name) ;\n",
      "\t\tlatitude:_FillValue = NaN ;\n",
      "\tdouble longitude(station_name) ;\n",
      "\t\tlongitude:_FillValue = NaN ;\n",
      "\tdouble altitude(station_name) ;\n",
      "\t\taltitude:_FillValue = NaN ;\n",
      "\tdouble od550aer(data_source, time, station_name) ;\n",
      "\t\tod550aer:_FillValue = NaN ;\n",
      "\t\tod550aer:obs_vars = \"od550aer\" ;\n",
      "\t\tod550aer:ts_type = \"daily\" ;\n",
      "\t\tod550aer:filter_name = \"ALL-wMOUNTAINS\" ;\n",
      "\t\tstring od550aer:ts_type_src = \"hourly\", \"hourly\" ;\n",
      "\t\tstring od550aer:var_units = \"1\", \"1\" ;\n",
      "\t\tod550aer:data_level = 3LL ;\n",
      "\t\tod550aer:revision_ref = \"20221027\" ;\n",
      "\t\tstring od550aer:from_files = \"ECMWF_OSUITE.daily.od550aer.2013.nc\", \"ECMWF_OSUITE.daily.od550aer.2014.nc\", \"ECMWF_OSUITE.daily.od550aer.2015.nc\", \"ECMWF_OSUITE.daily.od550aer.2016.nc\", \"ECMWF_OSUITE.daily.od550aer.2017.nc\", \"ECMWF_OSUITE.daily.od550aer.2018.nc\", \"ECMWF_OSUITE.daily.od550aer.2019.nc\", \"ECMWF_OSUITE.daily.od550aer.2020.nc\", \"ECMWF_OSUITE.daily.od550aer.2021.nc\", \"ECMWF_OSUITE.daily.od550aer.2022.nc\" ;\n",
      "\t\tod550aer:from_files_ref = \"None\" ;\n",
      "\t\tod550aer:colocate_time = 0LL ;\n",
      "\t\tod550aer:obs_is_clim = 0LL ;\n",
      "\t\tod550aer:pyaerocom = \"0.25.0.dev0\" ;\n",
      "\t\tod550aer:CONV\\!min_num_obs = \"{\\'monthly\\': {\\'daily\\': 3}}\" ;\n",
      "\t\tod550aer:resample_how = \"mean\" ;\n",
      "\t\tod550aer:obs_name = \"AeronetL1.5-d\" ;\n",
      "\t\tod550aer:vert_code = \"Column\" ;\n",
      "\t\tod550aer:diurnal_only = 0LL ;\n",
      "\t\tod550aer:zeros_to_nan = 1LL ;\n",
      "\t\tod550aer:coordinates = \"altitude latitude longitude\" ;\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "! ncdump -h \"./data/coldata/od550aer_od550aer_MOD-IFS-OSUITE_REF-AeronetL1.5-d_20130101_20221231_daily_ALL-wMOUNTAINS.nc\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pya",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}