02_python/1501@fachstelle-ogd-kanton-zuerich.ipynb

{"cells": [{"cell_type": "markdown", "id": "db7f08c1", "metadata": {}, "source": ["## Open Government Data, provided by **Canton Zurich**\n", "*Autogenerated Python starter code for data set with identifier* **1501@fachstelle-ogd-kanton-zuerich**"]}, {"cell_type": "markdown", "id": "164c4327", "metadata": {}, "source": ["## Dataset\n", "# **Nutzenden verwendete Suchbegriffe im Datenkatalog des Kantons Z\u00fcrich**"]}, {"cell_type": "markdown", "id": "46d13c8b-4ac3-4e2f-940f-0c14ec5b7c0d", "metadata": {}, "source": ["## Description\n", "\n", "Nutzungsstatistiken der Metadatens\u00e4tze von Verwaltungseinheiten und Organen des Kantons Z\u00fcrich, die im Daten- und Publikationskatalogs des Kantons Z\u00fcrich findbar sind. Verf\u00fcgbar sind: von Nutzenden verwendete Suchbegriffe. Hinweise: Die Ressourcen zum Jahr 2021 umfassen alle vier Quartale. Die Ressourcen zum Jahr 2022 umfassen nur das erste Quartal. Ab April 2022 ist die Aktualisierung aufgrund der Einf\u00fchrung einer neuen Version des Katalogs gestoppt."]}, {"cell_type": "markdown", "id": "cad813bb-c986-4bb4-b52b-f78bb608086f", "metadata": {}, "source": ["## Data set links\n", "\n", "[Direct data shop link for dataset](https://www.zh.ch/de/politik-staat/statistik-daten/datenkatalog.html#/datasets/1501@fachstelle-ogd-kanton-zuerich)"]}, {"cell_type": "markdown", "id": "9d4813e9", "metadata": {}, "source": ["## Metadata\n", "- **Issued** `2022-06-28T16:40:53`\n- **Modified** `2024-01-23T17:08:21`\n- **Startdate** `2021-04-01`\n- **Enddate** `2022-03-31`\n- **Theme** `['http://publications.europa.eu/resource/authority/data-theme/GOVE', 'http://publications.europa.eu/resource/authority/data-theme/ECON']`\n- **Keyword** `['kantonale_verwaltung', 'kanton_zuerich', 'ogd']`\n- **Publisher** `['Fachstelle Open Government Data des Kantons Z\u00fcrich']`\n- **Landingpage** `None`\n"]}, {"cell_type": "markdown", "id": "8a857d65", "metadata": {"jp-MarkdownHeadingCollapsed": true, "tags": []}, "source": ["## Imports and helper functions"]}, {"cell_type": "code", "execution_count": null, "id": "93b39602-1c1e-46d2-ae70-1716b1481e9b", "metadata": {"tags": []}, "outputs": [], "source": ["%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.style.use('ggplot')\n", "\n", "params = {\n", "    'text.color': (0.25, 0.25, 0.25),\n", "    'figure.figsize': [18, 6],\n", "   }\n", "\n", "plt.rcParams.update(params)\n", "\n", "import pandas as pd "]}, {"cell_type": "code", "execution_count": null, "id": "aa6611d7-e1c0-40a7-b0ff-601a8ef8439b", "metadata": {"tags": []}, "outputs": [], "source": ["# helper function for reading datasets with proper separator\n", "def get_dataset(url):\n", "    if url[-3:] != \"csv\":\n", "        print(\"The data set URL has no proper 'csv' extension. Reading the dataset might not have worked as expected.\\nPlease check the dataset link and adjust pandas' read_csv() parameters accordingly.\")\n", "    data = pd.read_csv(url, sep=\",\", on_bad_lines='warn', encoding_errors='ignore', low_memory=False)\n", "    # if dataframe only has one column or less the data is not comma separated, use \";\" instead\n", "    if data.shape[1] <= 1:\n", "        data = pd.read_csv(url, sep=';', on_bad_lines='warn', encoding_errors='ignore', low_memory=False)\n", "        if data.shape[1] <= 1:\n", "            print(\"The data wasn't imported properly. Very likely the correct separator couldn't be found.\\nPlease check the dataset manually and adjust the code.\")\n", "    return data"]}, {"cell_type": "markdown", "id": "02ce518f", "metadata": {}, "source": ["## Load data\n", "\n", "- The dataset has **`4` distribution(s)** in CSV format.\n", "- All available CSV distributions are listed below and can be read into a pandas dataframe."]}, {"cell_type": "code", "execution_count": null, "id": "0", "metadata": {"tags": []}, "outputs": [], "source": "# Distribution 0\n# Ktzhdistid               : 2940\n# Title                    : Suchbegriffe 2021, 4. Quartal\n# Description              : Maximal 1000 h\u00e4ufgste Suchbegriffe, die von Nutzenden im Quartal verwendet worden sind. Personennamen, Beleidigungen oder diskriminierenden \u00c4usserungen wurden entfernt.\n# Issued                   : 2022-06-28T16:40:53\n# Modified                 : 2022-06-28T16:40:53\n\ndf = get_dataset('https://www.web.statistik.zh.ch/ogd/daten/ressourcen/KTZH_00001501_00002940.csv')\n\n# Distribution 1\n# Ktzhdistid               : 2941\n# Title                    : Suchbegriffe 2021, 3. Quartal\n# Description              : Maximal 1000 h\u00e4ufgste Suchbegriffe, die von Nutzenden im Quartal verwendet worden sind. Personennamen, Beleidigungen oder diskriminierenden \u00c4usserungen wurden entfernt.\n# Issued                   : 2022-06-28T16:56:53\n# Modified                 : 2022-06-28T16:56:53\n\ndf = get_dataset('https://www.web.statistik.zh.ch/ogd/daten/ressourcen/KTZH_00001501_00002941.csv')\n\n# Distribution 2\n# Ktzhdistid               : 2942\n# Title                    : Suchbegriffe 2021, 2. Quartal\n# Description              : Maximal 1000 h\u00e4ufgste Suchbegriffe, die von Nutzenden im Quartal verwendet worden sind. Personennamen, Beleidigungen oder diskriminierenden \u00c4usserungen wurden entfernt.\n# Issued                   : 2022-06-28T17:12:46\n# Modified                 : 2022-06-28T17:12:46\n\ndf = get_dataset('https://www.web.statistik.zh.ch/ogd/daten/ressourcen/KTZH_00001501_00002942.csv')\n\n# Distribution 3\n# Ktzhdistid               : 2878\n# Title                    : Suchbegriffe 2022, 1. Quartal\n# Description              : Maximal 1000 h\u00e4ufgste Suchbegriffe, die von Nutzenden im Quartal verwendet worden sind. Personennamen, Beleidigungen oder diskriminierenden \u00c4usserungen wurden entfernt.\n# Issued                   : 2022-06-28T16:40:53\n# Modified                 : 2022-06-28T16:40:53\n\ndf = get_dataset('https://www.web.statistik.zh.ch/ogd/daten/ressourcen/KTZH_00001501_00002878.csv')\n\n"}, {"cell_type": "markdown", "id": "4ce1f78f", "metadata": {}, "source": ["## Analyze data"]}, {"cell_type": "code", "execution_count": null, "id": "3e3dab86", "metadata": {}, "outputs": [], "source": ["# drop columns that have no values\n", "df.dropna(how='all', axis=1, inplace=True)"]}, {"cell_type": "code", "execution_count": null, "id": "841bd8d2", "metadata": {}, "outputs": [], "source": ["print(f'The dataset has {df.shape[0]:,.0f} rows (observations) and {df.shape[1]:,.0f} columns (variables).')\n", "print(f'There seem to be {df.duplicated().sum()} exact duplicates in the data.')"]}, {"cell_type": "code", "execution_count": null, "id": "75e73c96", "metadata": {}, "outputs": [], "source": ["df.info(memory_usage='deep', verbose=True)"]}, {"cell_type": "code", "execution_count": null, "id": "02f3df4d", "metadata": {}, "outputs": [], "source": ["df.head()"]}, {"cell_type": "code", "execution_count": null, "id": "a0d7d898", "metadata": {}, "outputs": [], "source": ["# display a small random sample transposed in order to see all variables\n", "df.sample(3).T"]}, {"cell_type": "code", "execution_count": null, "id": "786806dd", "metadata": {}, "outputs": [], "source": ["# describe non-numerical features\n", "try:\n", "    with pd.option_context('display.float_format', '{:,.2f}'.format):\n", "        display(df.describe(exclude='number'))\n", "except:\n", "    print(\"No categorical data in dataset.\")"]}, {"cell_type": "code", "execution_count": null, "id": "e744a6b6", "metadata": {}, "outputs": [], "source": ["# describe numerical features\n", "try:\n", "    with pd.option_context('display.float_format', '{:,.2f}'.format):\n", "        display(df.describe(include='number'))\n", "except:\n", "    print(\"No numercial data in dataset.\")"]}, {"cell_type": "code", "execution_count": null, "id": "7a65d95d", "metadata": {}, "outputs": [], "source": ["# check missing values with missingno\n", "# https://github.com/ResidentMario/missingno\n", "import missingno as msno\n", "msno.matrix(df, labels=True, sort='descending');"]}, {"cell_type": "code", "execution_count": null, "id": "fcc604b7", "metadata": {}, "outputs": [], "source": ["# plot a histogram for each numerical feature\n", "try:\n", "    df.hist(bins=25, rwidth=.9)\n", "    plt.tight_layout()\n", "    plt.show()\n", "except:\n", "    print(\"No numercial data to plot.\") "]}, {"cell_type": "code", "execution_count": null, "id": "e13cc6eb", "metadata": {}, "outputs": [], "source": ["# continue your code here..."]}, {"cell_type": "code", "execution_count": null, "id": "18886378", "metadata": {}, "outputs": [], "source": []}, {"cell_type": "code", "execution_count": null, "id": "59fce071", "metadata": {}, "outputs": [], "source": []}, {"cell_type": "code", "execution_count": null, "id": "f42f25aa", "metadata": {}, "outputs": [], "source": []}, {"cell_type": "markdown", "id": "c6b87d83", "metadata": {}, "source": ["**Contact**: Fachstelle Open Government Data des Kantons Z\u00fcrich | info@open.zh.ch"]}], "metadata": {"kernelspec": {"display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.15"}}, "nbformat": 4, "nbformat_minor": 5}