02_python/728@gesundheitsdirektion-kanton-zuerich.ipynb

{"cells": [{"cell_type": "markdown", "id": "db7f08c1", "metadata": {}, "source": ["## Open Government Data, provided by **Canton Zurich**\n", "*Autogenerated Python starter code for data set with identifier* **728@gesundheitsdirektion-kanton-zuerich**"]}, {"cell_type": "markdown", "id": "164c4327", "metadata": {}, "source": ["## Dataset\n", "# **COVID_19 Abwassermonitoring im Kanton Z\u00fcrich ab 30.05.2021 bis 02.05.2023**"]}, {"cell_type": "markdown", "id": "46d13c8b-4ac3-4e2f-940f-0c14ec5b7c0d", "metadata": {}, "source": ["## Description\n", "\n", "Dreimal* w\u00f6chentliche SARS-CoV-2-RNA-Messungen aus Proben von dreizehn \u00fcber den Kanton Z\u00fcrich verteilten Abwasserreinigungsanlagen (ARA). Die analysierten Werte werden einige Tage darauf publiziert (einmal w\u00f6chentlich jeweils am Mittwoch). Daten enhalten: 'SARS-CoV-2 RNA im Rohabwasser pro 100'000 Personen', 'Zufluss zur ARA in Kubikmeter pro Tag', 'Einzugsgebiet (Anzahl an die ARA angeschlossene Einwohner:innen)'. Wichtiger Hinweis: Die Messungen von SARS-CoV-2-RNA im Abwasser unterliegen teilweise erheblichen Messfehlern. Eine Interpretation der Abwassermessungen von SARS-CoV-2 im Vergleich zu den Fallzahlen ist daher nur \u00fcber l\u00e4ngere Zeitr\u00e4ume aussagekr\u00e4ftig und h\u00e4ngt von der gew\u00e4hlten Skala ab. Die Daten sollten daher mit Vorsicht interpretiert werden, da ein grosser Spielraum f\u00fcr Fehlinterpretationen besteht. Im Zeitraum von Januar bis Anfang M\u00e4rz 2022 wurde die Aktualisierung der Daten pausiert, um die Abweichungen des Abwassermonitorings von den gemeldeten positiven Fallzahlen zu untersuchen. Nachdem die Methode an die vorherrschende Virusvariante angepasst worden ist, werden die Daten r\u00fcckwirkend ab Januar 2022 weiter aktualisiert.\nAb Januar 2023 wird die Beprobung auf die ARAs B\u00fclach, Dietikon, Illnau-Effretikon, Winterthur und Z\u00fcrich (Werdh\u00f6lzli) reduziert."]}, {"cell_type": "markdown", "id": "cad813bb-c986-4bb4-b52b-f78bb608086f", "metadata": {}, "source": ["## Data set links\n", "\n", "[Direct data shop link for dataset](https://www.zh.ch/de/politik-staat/statistik-daten/datenkatalog.html#/datasets/728@gesundheitsdirektion-kanton-zuerich)"]}, {"cell_type": "markdown", "id": "9d4813e9", "metadata": {}, "source": ["## Metadata\n", "- **Issued** `2021-07-06T00:00:00`\n- **Modified** `2024-07-08T17:06:07`\n- **Startdate** `2021-05-30`\n- **Enddate** `2023-05-02`\n- **Theme** `['http://publications.europa.eu/resource/authority/data-theme/HEAL']`\n- **Keyword** `['bezirke', 'gesundheit', 'kanton_zuerich', 'krankheiten', 'wasser', 'corona', 'covid-19', 'covid19', 'ogd']`\n- **Publisher** `['Gesundheitsdirektion des Kantons Z\u00fcrich']`\n- **Landingpage** `https://github.com/openZH/covid_19_wastewater_monitoring_ZH`\n"]}, {"cell_type": "markdown", "id": "8a857d65", "metadata": {"jp-MarkdownHeadingCollapsed": true, "tags": []}, "source": ["## Imports and helper functions"]}, {"cell_type": "code", "execution_count": null, "id": "93b39602-1c1e-46d2-ae70-1716b1481e9b", "metadata": {"tags": []}, "outputs": [], "source": ["%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.style.use('ggplot')\n", "\n", "params = {\n", "    'text.color': (0.25, 0.25, 0.25),\n", "    'figure.figsize': [18, 6],\n", "   }\n", "\n", "plt.rcParams.update(params)\n", "\n", "import pandas as pd "]}, {"cell_type": "code", "execution_count": null, "id": "aa6611d7-e1c0-40a7-b0ff-601a8ef8439b", "metadata": {"tags": []}, "outputs": [], "source": ["# helper function for reading datasets with proper separator\n", "def get_dataset(url):\n", "    if url[-3:] != \"csv\":\n", "        print(\"The data set URL has no proper 'csv' extension. Reading the dataset might not have worked as expected.\\nPlease check the dataset link and adjust pandas' read_csv() parameters accordingly.\")\n", "    data = pd.read_csv(url, sep=\",\", on_bad_lines='warn', encoding_errors='ignore', low_memory=False)\n", "    # if dataframe only has one column or less the data is not comma separated, use \";\" instead\n", "    if data.shape[1] <= 1:\n", "        data = pd.read_csv(url, sep=';', on_bad_lines='warn', encoding_errors='ignore', low_memory=False)\n", "        if data.shape[1] <= 1:\n", "            print(\"The data wasn't imported properly. Very likely the correct separator couldn't be found.\\nPlease check the dataset manually and adjust the code.\")\n", "    return data"]}, {"cell_type": "markdown", "id": "02ce518f", "metadata": {}, "source": ["## Load data\n", "\n", "- The dataset has **`2` distribution(s)** in CSV format.\n", "- All available CSV distributions are listed below and can be read into a pandas dataframe."]}, {"cell_type": "code", "execution_count": null, "id": "0", "metadata": {"tags": []}, "outputs": [], "source": "# Distribution 0\n# Ktzhdistid               : 1850\n# Title                    : Liste der Abwasserreinigungsanlagen mit Einzugsgebiet\n# Description              : Ausgew\u00e4hlte, ins SARS-CoV-2-RNA-Abwassermonitoring integrierte Abwasserreinigungsanlagen (ARA). Variabeln: 'ARA_code' = Abk\u00fcrzung der ARA; 'catchment_area' = Einzugsgebiet (Anzahl an die ARA angeschlossene Einwohner:innen); 'ARA_name' = Ausgeschriebener Name der ARA; 'district' = Kantonaler Bezirk, in dem die ARA liegt.\n# Issued                   : 2021-07-06T10:11:03\n# Modified                 : 2021-09-28T08:00:01\n\ndf = get_dataset('https://raw.githubusercontent.com/openZH/covid_19_wastewater_monitoring_ZH/master/ara_einzugsgebiet.csv')\n\n# Distribution 1\n# Ktzhdistid               : 1851\n# Title                    : SARS-CoV-2-RNA-Abwassermonitoring\n# Description              : SARS-CoV-2-RNA-Messungen aus Proben von dreizehn ins Monitoring integrierten Abwasserreinigungsanlagen (ARA). Variabeln: 'date' = Tag der Entnahme der Probe; 'SARS_CoV_2_RNA_copies_d_100000' = SARS-CoV-2 RNA im Rohabwasser pro 100'000 Personen; 'quantification_flag' = 'Q': \u00fcber Quantifizierungsgrenze (>LOQ), 'D': \u00fcber Nachweisgrenze (>LOD), 'N': unter Nachweisgrenze (<LOD); 'flow_m3_d' = Zufluss zur ARA in Kubikmeter pro Tag; 'sample_origin' = Abk\u00fcrzung der ARA ('ARA_code' gem\u00e4ss Ressource \"Liste der Abwasserreinigungsanlagen mit Einzugsgebiet\").\n# Issued                   : 2021-07-06T11:41:06\n# Modified                 : 2022-03-21T10:05:15\n\ndf = get_dataset('https://raw.githubusercontent.com/openZH/covid_19_wastewater_monitoring_ZH/master/COVID19_Abwassermonitoring.csv')\n\n"}, {"cell_type": "markdown", "id": "4ce1f78f", "metadata": {}, "source": ["## Analyze data"]}, {"cell_type": "code", "execution_count": null, "id": "3e3dab86", "metadata": {}, "outputs": [], "source": ["# drop columns that have no values\n", "df.dropna(how='all', axis=1, inplace=True)"]}, {"cell_type": "code", "execution_count": null, "id": "841bd8d2", "metadata": {}, "outputs": [], "source": ["print(f'The dataset has {df.shape[0]:,.0f} rows (observations) and {df.shape[1]:,.0f} columns (variables).')\n", "print(f'There seem to be {df.duplicated().sum()} exact duplicates in the data.')"]}, {"cell_type": "code", "execution_count": null, "id": "75e73c96", "metadata": {}, "outputs": [], "source": ["df.info(memory_usage='deep', verbose=True)"]}, {"cell_type": "code", "execution_count": null, "id": "02f3df4d", "metadata": {}, "outputs": [], "source": ["df.head()"]}, {"cell_type": "code", "execution_count": null, "id": "a0d7d898", "metadata": {}, "outputs": [], "source": ["# display a small random sample transposed in order to see all variables\n", "df.sample(3).T"]}, {"cell_type": "code", "execution_count": null, "id": "786806dd", "metadata": {}, "outputs": [], "source": ["# describe non-numerical features\n", "try:\n", "    with pd.option_context('display.float_format', '{:,.2f}'.format):\n", "        display(df.describe(exclude='number'))\n", "except:\n", "    print(\"No categorical data in dataset.\")"]}, {"cell_type": "code", "execution_count": null, "id": "e744a6b6", "metadata": {}, "outputs": [], "source": ["# describe numerical features\n", "try:\n", "    with pd.option_context('display.float_format', '{:,.2f}'.format):\n", "        display(df.describe(include='number'))\n", "except:\n", "    print(\"No numercial data in dataset.\")"]}, {"cell_type": "code", "execution_count": null, "id": "7a65d95d", "metadata": {}, "outputs": [], "source": ["# check missing values with missingno\n", "# https://github.com/ResidentMario/missingno\n", "import missingno as msno\n", "msno.matrix(df, labels=True, sort='descending');"]}, {"cell_type": "code", "execution_count": null, "id": "fcc604b7", "metadata": {}, "outputs": [], "source": ["# plot a histogram for each numerical feature\n", "try:\n", "    df.hist(bins=25, rwidth=.9)\n", "    plt.tight_layout()\n", "    plt.show()\n", "except:\n", "    print(\"No numercial data to plot.\") "]}, {"cell_type": "code", "execution_count": null, "id": "e13cc6eb", "metadata": {}, "outputs": [], "source": ["# continue your code here..."]}, {"cell_type": "code", "execution_count": null, "id": "18886378", "metadata": {}, "outputs": [], "source": []}, {"cell_type": "code", "execution_count": null, "id": "59fce071", "metadata": {}, "outputs": [], "source": []}, {"cell_type": "code", "execution_count": null, "id": "f42f25aa", "metadata": {}, "outputs": [], "source": []}, {"cell_type": "markdown", "id": "c6b87d83", "metadata": {}, "source": ["**Contact**: Gesundheitsdirektion des Kantons Z\u00fcrich | Coronavirus-Hotline | gdstab@gd.zh.ch"]}], "metadata": {"kernelspec": {"display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.15"}}, "nbformat": 4, "nbformat_minor": 5}