DQM_Playground_Data_Engineering_Test_Client.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "78b21dbe",
   "metadata": {},
   "source": [
    "# DQM Playground Data Engineering Test Client\n",
    "\n",
    "A sample API client for accessing DQM Playground's data. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0e8ed2f",
   "metadata": {},
   "source": [
    "## Install the auto-generated API client\n",
    "Since we are using OpenAPI specification, we can automatically generate a python client using:\n",
    "- The API schema generated by DQM Playground [here](https://ml4dqm-playground.web.cern.ch/openapi?format=openapi-json)\n",
    "- The Swagger client generator [here](https://editor.swagger.io/)\n",
    "\n",
    "We download the client and can install like like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b678a3a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install -e ./client -U --user\n",
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cb15d1b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import swagger_client\n",
    "from swagger_client.rest import ApiException\n",
    "from pprint import pprint\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be527710",
   "metadata": {},
   "source": [
    "## Configure the client\n",
    "Replace `API_TOKEN` here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "294b7cf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure and create an API client\n",
    "# using an API token\n",
    "API_TOKEN = None\n",
    "MLP_URL = \"https://ml4dqm-playground.web.cern.ch\"\n",
    "MLP_DEVELOP_URL = \"https://ml4dqm-playground-develop.web.cern.ch\"\n",
    "\n",
    "configuration = swagger_client.Configuration()\n",
    "configuration.host = MLP_URL\n",
    "client = swagger_client.ApiClient(configuration)\n",
    "client.set_default_header(header_name=\"Content-Type\", header_value=\"application/json\")\n",
    "client.set_default_header(header_name=\"Authorization\", header_value=f\"Token {API_TOKEN}\")\n",
    "\n",
    "api_instance = swagger_client.ApiApi(client)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "897054f1",
   "metadata": {},
   "source": [
    "## Fetch data\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Note:</b> The fetched data is paginated by 50, meaning that each response will\n",
    "    have at most 50 results. You can ask for more by specifying a page number.\n",
    "</div>\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <b>Warning:</b> The response should normally contain <b>next</b>, <b>previous</b> and <b>count</b> keys,\n",
    "    showing the total number of results available. This does not currently work with\n",
    "    this automatically generated client.\n",
    "</div>\n",
    "\n",
    "### Run information\n",
    "\n",
    "Available parameters for filtering Runs are:\n",
    "\n",
    "\n",
    "- `page = 56` # int | A page number within the paginated result set. (optional\n",
    "- `run_number = 'run_number_example'` # str | run_number (optional)\n",
    "- `run_date = 'run_date_example'` # str | run_date (optional)\n",
    "- `year = 'year_example'` # str | year (optional)\n",
    "- `period = 'period_example'` # str | period (optional)\n",
    "- `_date = '_date_example'` # str | date (optional)\n",
    "- `oms_fill = 'oms_fill_example'` # str | oms_fill (optional)\n",
    "- `oms_lumisections = 'oms_lumisections_example'` # str | oms_lumisections (optional)\n",
    "- `oms_initial_lumi = 'oms_initial_lumi_example'` # str | oms_initial_lumi (optional)\n",
    "- `oms_end_lumi = 'oms_end_lumi_example'` # str | oms_end_lumi (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8d4aaf7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "run_number = 315741  # an int will also do\n",
    "run = api_instance.list_runs(run_number=run_number)\n",
    "pprint(run)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc69d540",
   "metadata": {},
   "source": [
    "### Lumisections information\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Note:</b> Currently not much info on Lumisections is available through the API\n",
    "</div>\n",
    "\n",
    "Available parameters for filtering Lumisections are:\n",
    "\n",
    "- `page = 56` # int | A page number within the paginated result set. (optional)\n",
    "- `run = 'run_example'` # str | run (optional)\n",
    "- `ls_number = 'ls_number_example'` # str | ls_number (optional)\n",
    "- `_date = '_date_example'` # str | date (optional)\n",
    "- `oms_zerobias_rate = 'oms_zerobias_rate_example'` # str | oms_zerobias_rate (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e50e4bd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# BUG FIXED - You can use run_number here to fetch all lumisections for a specific run\n",
    "run_number = 315257 \n",
    "ls = api_instance.list_lumisections(run=run_number)\n",
    "pprint(ls)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c265ea6",
   "metadata": {},
   "source": [
    "### Lumisection Histogram 1D\n",
    "\n",
    "Available parameters for filtering 1D Lumisection Histograms are:\n",
    "- `page = 56` # int | A page number within the paginated result set. (optional)\n",
    "- `lumisection__run__run_number = 'lumisection__run__run_number_example'` # str | lumisection__run__run_number (optional)\n",
    "- `lumisection__run__run_number__gte = 'lumisection__run__run_number__gte_example'` # str | lumisection__run__run_number__gte (optional)\n",
    "- `lumisection__run__run_number__lte = 'lumisection__run__run_number__lte_example'` # str | lumisection__run__run_number__lte (optional)\n",
    "- `lumisection__ls_number = 'lumisection__ls_number_example'` # str | lumisection__ls_number (optional)\n",
    "- `lumisection__ls_number__gte = 'lumisection__ls_number__gte_example'` # str | lumisection__ls_number__gte (optional)\n",
    "- `lumisection__ls_number__lte = 'lumisection__ls_number__lte_example'` # str | lumisection__ls_number__lte (optional)\n",
    "- `entries__gte = 'entries__gte_example'` # str | entries__gte (optional)\n",
    "- `entries__lte = 'entries__lte_example'` # str | entries__lte (optional)\n",
    "- `title = 'title_example'` # str | title (optional)\n",
    "- `lumisection__ls_number__in = 'lumisection__ls_number__in_example'` # str | lumisection__ls_number__in (optional)\n",
    "- `lumisection__run__run_number__in = 'lumisection__run__run_number__in_example'` # str | lumisection__run__run_number__in (optional)\n",
    "- `source_data_file__filepath__contains = 'source_data_file__filepath__contains_example'` # str | source_data_file__filepath__contains (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "280288a9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Get by title\n",
    "title = \"Summary_ClusterStoNCorr__OnTrack__TEC__MINUS__wheel__7\"\n",
    "lh1d = api_instance.list_lumisection_histogram1_ds(title=title) \n",
    "print(f\"Got {len(lh1d.results)} results!\\n\")\n",
    "pprint(lh1d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "974eeb52",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Get by run number\n",
    "run_num = 315267\n",
    "lh1d = api_instance.list_lumisection_histogram1_ds(\n",
    "    lumisection__run__run_number=run_num\n",
    ") \n",
    "print(f\"Got {len(lh1d.results)} results!\\n\")\n",
    "pprint(lh1d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ec438d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete by Lumisection Histogram id\n",
    "# The id can be retrieved by any of the methods above\n",
    "api_instance.destroy_lumisection_histogram1_d(id=41) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ad1b1b3",
   "metadata": {},
   "source": [
    "### Lumisection Histogram 2D\n",
    "Same filters with 1D Histograms apply."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43b718be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# By run number\n",
    "run_num = 297656\n",
    "lh2d = api_instance.list_lumisection_histogram2_ds(\n",
    "    lumisection__run__run_number=run_num\n",
    ") \n",
    "print(f\"Got {len(lh2d.results)} results!\\n\")\n",
    "# Warning -- this contains a LOT of data\n",
    "#pprint(lh2d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1269a65",
   "metadata": {},
   "outputs": [],
   "source": [
    "# By filename\n",
    "filename = 'ZeroBias_2017B_DataFrame_2D_17.csv'\n",
    "lh2d = api_instance.list_lumisection_histogram2_ds(\n",
    "    source_data_file__filepath__contains=filename\n",
    ")  \n",
    "print(f\"Got {len(lh2d.results)} results!\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b3a2367",
   "metadata": {},
   "source": [
    "### Working with the data received\n",
    "The data returned from the client we have been using until here are **lists of instances of custom classes**, which reflect the data stored in the database that the API operates upon.\n",
    "These classes have a `to_dict()` method which can be used to convert them to dictionaries and, then, use whatever data format you require.\n",
    "\n",
    "#### Convert to pandas DataFrame\n",
    "An example for 1D Lumisection histograms, using the `pd.DataFrame.from_dict()` method to load them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd42ce3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "lh1d_df = pd.DataFrame.from_dict([result.to_dict() for result in lh1d.results])\n",
    "\n",
    "print(lh1d_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c54d35f8",
   "metadata": {},
   "source": [
    "# Managing DQM Histogram Data Files\n",
    "\n",
    "DQM data files are `.csv` files (`.root` support for nanoDQM files will be added in the future) which contain:\n",
    "- Run Histograms or\n",
    "- Lumisection 1D Histograms or\n",
    "- Lumisection 2D Histograms\n",
    "\n",
    "They are represented in DQM Playground's database with the `HistogramDataFile` model. \n",
    "\n",
    "By default, these files are expected to reside in `/eos/project/c/cmsml4dc`. This directory is not being monitored for changes, and new files can only be discovered using a [custom Django Management command](https://cmstrackerdpg.github.io/cms-tkdpg-software-knowledge-transfer/mlplayground/apps/histogram_file_manager/management/). \n",
    "\n",
    "The contents of a Histogram Data Files are unknown, until a user starts parsing the file by specifying three options:\n",
    "- The `granularity` of the data (`run` or `lumisection`),\n",
    "- The `data_dimensionality` of the data (`1` for 1D or `2` for 2D) and\n",
    "- The file `file_format` (`csv` only available currently).\n",
    "\n",
    "Depending on the combination of the three options above, DQM Playground will try to extract the appropriate histograms from the Histogram Data File, namely:\n",
    "\n",
    "- `granularity=run`, `data_dimensionality=1` and `file_format=csv` will produce `RunHistogram`s,\n",
    "- `granularity=lum`, `data_dimensionality=1` and `file_format=csv` will produce `LumisectionHistogram1D`s,\n",
    "- `granularity=lum`, `data_dimensionality=2` and `file_format=csv` will produce `LumisectionHistogram2D`s.\n",
    "\n",
    "## Listing all the available Histogram Data Files\n",
    "\n",
    "To list all the Histogram Data Files that DQM Playground has discovered:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd6ccbb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "available_hdf = api_instance.list_histogram_data_files(page=1) # No filters supplied\n",
    "print(f\"Got {len(available_hdf.results)} Histogram Data Files\")  # Keep in mind that results are returned in pages of 50"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a49b628f",
   "metadata": {},
   "source": [
    "## Filtering histogram data files by filename\n",
    "For this example, we will try to get the status of the `ZeroBias_2017B_DataFrame_2D_17.csv` DQM file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbf5e0b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "available_hdf = api_instance.list_histogram_data_files(filepath__contains=\"ZeroBias_2017B_DataFrame_2D_17\")\n",
    "print(available_hdf.results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "162fc3c6",
   "metadata": {},
   "source": [
    "If parsing has already been started for this specific Histogram Data File, we will see the `data_dimensionality` and `granularity` values already set. If not, they will be unknown. \n",
    "\n",
    "Once parsing of the file has started, the total number of `run_histograms`, `lumisection_histograms_1d` or `lumisection_histograms_2d` that were extracted from this file will be available under the respective values in the response. \n",
    "\n",
    "Its parsing percentage can also be seen under the `percentage_processed` value."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52bc2b32",
   "metadata": {},
   "source": [
    "## Starting parsing a Histogram Data File\n",
    "To start parsing a Histogram Data file, we need:\n",
    "- Its unique `id`,\n",
    "- The `granularity` of the data contained (`run` or `lumisection`),\n",
    "- The `data_dimensionality` of the data (`1` for 1D or `2` for 2D) and\n",
    "- The file `file_format` (`csv`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ad37f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "available_hdf = api_instance.list_histogram_data_files(filepath__contains=\"ZeroBias_2017B_DataFrame_2D_17\")\n",
    "file_id = available_hdf.results[0].id\n",
    "\n",
    "# TODO: Buggy generated code by swagger creates multiple definitions for start_parsing_histogram_data_file\n",
    "# body = {'granularity':'lum', 'data_dimensionality':2, 'file_format':'csv'}\n",
    "# success = api_instance.start_parsing_histogram_data_file(id=file_id, body=body)\n",
    "\n",
    "# Manually do the request for now\n",
    "r = requests.post(\n",
    "    f\"{MLP_URL}/api/histogram_data_files/{file_id}/start_parsing/\",\n",
    "    headers={\"Content-Type\": \"application/json\", \"Authorization\": f\"Token {API_TOKEN}\"},\n",
    "    json={'granularity':'lum', 'data_dimensionality':2, 'file_format':'csv'}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8e19eb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Parsing for file {file_id} started: {r.ok}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d36b003",
   "metadata": {},
   "source": [
    "## Discovering new DQMIO files [Admins only]\n",
    "If new files are added in the directories that DQMIO files reside, they won't be automatically discovered by DQM Playground without forcing a new discovery process. To do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ebf4f31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Manually do the request for now\n",
    "r = requests.get(\n",
    "    f\"{MLP_DEVELOP_URL}/api/histogram_data_files/discover/\",\n",
    "    headers={\"Content-Type\": \"application/json\", \"Authorization\": f\"Token {API_TOKEN}\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1810eb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Discovery started: {r.ok}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}