main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"width:100%; background-color: #D9EDF7; border: 1px solid #CFCFCF; text-align: left; padding: 10px;\">\n",
    "      <b>Time series: Processing Notebook</b>\n",
    "      <ul>\n",
    "        <li>Main Notebook</li>\n",
    "        <li><a href=\"processing.ipynb\">Processing Notebook</a></li>\n",
    "      </ul>\n",
    "      <br>This Notebook is part of the <a href=\"http://data.open-power-system-data.org/time_series\">Time series Data Package</a> of <a href=\"http://open-power-system-data.org\">Open Power System Data</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\" style=\"margin-top: 1em;\"><ul class=\"toc-item\"><li><span><a href=\"#1.-About-Open-Power-System-Data\" data-toc-modified-id=\"1.-About-Open-Power-System-Data-1\">1. About Open Power System Data</a></span></li><li><span><a href=\"#2.-About-Jupyter-Notebooks-and-GitHub\" data-toc-modified-id=\"2.-About-Jupyter-Notebooks-and-GitHub-2\">2. About Jupyter Notebooks and GitHub</a></span></li><li><span><a href=\"#3.-About-this-datapackage\" data-toc-modified-id=\"3.-About-this-datapackage-3\">3. About this datapackage</a></span></li><li><span><a href=\"#4.-Data-sources\" data-toc-modified-id=\"4.-Data-sources-4\">4. Data sources</a></span></li><li><span><a href=\"#5.-Naming-conventions\" data-toc-modified-id=\"5.-Naming-conventions-5\">5. Naming conventions</a></span></li><li><span><a href=\"#6.-License\" data-toc-modified-id=\"6.-License-6\">6. License</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. About Open Power System Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is part of the project [Open Power System Data](http://open-power-system-data.org). Open Power System Data develops a platform for free and open data for electricity system modeling. We collect, check, process, document, and provide data that are publicly available but currently inconvenient to use. \n",
    "More info on Open Power System Data:\n",
    "- [Information on the project on our website](http://open-power-system-data.org)\n",
    "- [Data and metadata on our data platform](http://data.open-power-system-data.org)\n",
    "- [Data processing scripts on our GitHub page](https://github.com/Open-Power-System-Data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. About Jupyter Notebooks and GitHub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file is a [Jupyter Notebook](http://jupyter.org/). A Jupyter Notebook is a file that combines executable programming code with visualizations and comments in markdown format, allowing for an intuitive documentation of the code. We use Jupyter Notebooks for combined coding and documentation. We use Python 3 as programming language. All Notebooks are stored on [GitHub](https://github.com/), a platform for software development, and are publicly available. More information on our IT-concept can be found [here](http://open-power-system-data.org/it). See also our [step-by-step manual](http://open-power-system-data.org/step-by-step) how to use the dataplatform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. About this datapackage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We provide data in different chunks, or [data packages](http://frictionlessdata.io/data-packages/). The one you are looking at right now, [Time series](http://data.open-power-system-data.org/time_series/), contains various kinds of time series data:\n",
    "- Electricity consumption (load): forecast and actual values\n",
    "- wind and solar power: generation forecast, actual generation, installed capacity, capacity factors (profiles)\n",
    "- day-ahead spot prices\n",
    "\n",
    "In which resolution the data is published depends on the \"market time unit\" applied in the respective jurisdiction as well as the type of data. For most data types, the following mapping applies:\n",
    "\n",
    "- 15 minutes: Austria, Belgium, Germany, Hungary, Luxembourg, Netherlands\n",
    "- 30 minutes: Cyprus, Ireland, United Kingdom\n",
    "- 60 minutes: All other European countries\n",
    "\n",
    "For data that are originally available in 15 or 30 minutes resolution, hourly averages are included with the 60 minutes dataset. The original resolition data are is provided in a separate file.\n",
    "The timeseries become available at different points in time depending on the sources. The full dataset is only available from 2015 onwards."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is \"Total load\"?\n",
    "\n",
    "There are two sources for load data: [ENTSO-E Power Statistics](https://www.entsoe.eu/data/power-stats) (PS) and the [ENTSO-E Transparency Platform](https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/show?&biddingZone.values=CTY|10Y1001A1001A83F!CTY|10Y1001A1001A83F) (TP). Both report \"total load\", which is defined as follows:\n",
    "\n",
    "$${total \\ load} = total \\ generation - auxilary/self-consumption \\ in \\ power \\ plants + imports - exports - consumption \\ by \\ storages$$\n",
    "\n",
    "<img src=\"input/load_schema.png\" alt=\"Load schema\" style=\"width: 800px;\"/>\n",
    "\n",
    "Source: [ENTSO-E Guidelines for Monthly Statistics Data Collection](https://docstore.entsoe.eu/Documents/Publications/Statistics/MS_guidelines2016.pdf#page=6) / [Transparency Platform Detailed Data Descriptions](https://www.entsoe.eu/fileadmin/user_upload/_library/resources/Transparency/MoP%20Ref02%20-%20EMFIP-Detailed%20Data%20Descriptions%20V1R4-2014-02-24.pdf#page=11).\n",
    "\n",
    "The two sources differ Values on PS (~500 TWh annaually in Germany) are usually slightly higher than on the TP (~490 TWh). \n",
    "The reason probably lies with different reporting deadlines: Values on the TP have to be reported \n",
    "[\"no later than one hour after the end of the operating period\"](https://transparency.entsoe.eu/content/static_content/Static%20content/knowledge%20base/data-views/load-domain/Data-view%20Total%20Load%20-%20Day%20Ahead%20-%20Actual.html). For the PS, the data is published with a delay of up to 3 months, which might allow for more accurate metering. For a comparison of the two sources see [Hirth, et al. (2018)](https://doi.org/10.1016/j.apenergy.2018.04.048).\n",
    "\n",
    "For some countries, the PS report a [\"represenativity factor\"](https://docstore.entsoe.eu/Documents/Publications/Statistics/Specific_national_considerations.pdf#page=6) (91% in Germany until 2014, 97% since then), indicating that the reported values would have to be upscaled by this value resulting in ~520 TWh annually in Germany. \n",
    "\n",
    "[Schuhmacher & Hirth (2015)](http://services.feem.it/userfiles/attach/20151191122284NDL2015-088.pdf#page=11) compare the German hourly load total load values to monthly and yearly aggregate consumption statistics for Germany, showing considerable differences, part of which may be explained by the fact that none of th ENTSO-E data cover industrial auto generation which is not transported over the transmission grid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Data sources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main data sources are the various European Transmission System Operators (TSOs), the [ENTSO-E Power Statistics](https://www.entsoe.eu/data/power-stats) and the [ENTSO-E Transparency Platform](https://transparency.entsoe.eu). A complete list of data sources is provided on the [datapackage information website](http://data.open-power-system-data.org/time_series/). They are also contained in the JSON file that contains all metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Naming conventions\n",
    "\n",
    "The table headers specifies each data column according to 3 categories: region, variable and attribute. **region** specifies the geographical scope according to the [ISO 3166 codes](https://en.wikipedia.org/wiki/ISO_3166). **variable** distinguishes consumption,generation and prices. **attribute** gives further properties of the data that are specific to the respective **variable**. See the table below for the set of possible combinations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>region</th>\n",
       "      <th>variable</th>\n",
       "      <th>attribute</th>\n",
       "      <th>Explanation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"13\" valign=\"top\">ISO 3166 area code and name or control area or bidding zone</th>\n",
       "      <th rowspan=\"6\" valign=\"top\">load</th>\n",
       "      <th>actual_entsoe_power_statistics</th>\n",
       "      <th>Total load as published on ENTSO-E Data Portal</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actual_entsoe_transparency</th>\n",
       "      <th>Total load as published on ENTSO-E Data Portal/Power Statistics</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actual_tso</th>\n",
       "      <th>Total load as published by the TSO</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actual_net_consumption_tso</th>\n",
       "      <th>Total load exluding transmission losses as published onby the TSO</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actual_gross_generation_tso</th>\n",
       "      <th>Total power generation from national TSO</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day_ahead_forecast_entsoe_transparency</th>\n",
       "      <th>Day ahead total load forecast from ENTSO-E Transparency Platform</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"6\" valign=\"top\">solar / wind / wind_onshore / wind_offshore</th>\n",
       "      <th>generation_actual</th>\n",
       "      <th>Electricity produced by all power plants</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day_ahead_generation_forecast</th>\n",
       "      <th>Day-ahead generation forecast</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>generation_actual_dso</th>\n",
       "      <th>Electricity produced by power plants connected to distribution grid</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>generation_actual_tso</th>\n",
       "      <th>Electricity produced by power plants connected to transmission grid</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>capacity</th>\n",
       "      <th>Aggregated installed capacity of power plants (actual availability not accounted for)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>profile</th>\n",
       "      <th>Share of installed capacity producing</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>price</th>\n",
       "      <th>day_ahead</th>\n",
       "      <th>Day-ahead spot price</th>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: [(ISO 3166 area code and name or control area or bidding zone, load, actual_entsoe_power_statistics, Total load as published on ENTSO-E Data Portal), (ISO 3166 area code and name or control area or bidding zone, load, actual_entsoe_transparency, Total load as published on ENTSO-E Data Portal/Power Statistics), (ISO 3166 area code and name or control area or bidding zone, load, actual_tso, Total load as published by the TSO), (ISO 3166 area code and name or control area or bidding zone, load, actual_net_consumption_tso, Total load exluding transmission losses as published onby the TSO), (ISO 3166 area code and name or control area or bidding zone, load, actual_gross_generation_tso, Total power generation from national TSO), (ISO 3166 area code and name or control area or bidding zone, load, day_ahead_forecast_entsoe_transparency, Day ahead total load forecast from ENTSO-E Transparency Platform), (ISO 3166 area code and name or control area or bidding zone, solar / wind / wind_onshore / wind_offshore, generation_actual, Electricity produced by all power plants), (ISO 3166 area code and name or control area or bidding zone, solar / wind / wind_onshore / wind_offshore, day_ahead_generation_forecast, Day-ahead generation forecast), (ISO 3166 area code and name or control area or bidding zone, solar / wind / wind_onshore / wind_offshore, generation_actual_dso, Electricity produced by power plants connected to distribution grid), (ISO 3166 area code and name or control area or bidding zone, solar / wind / wind_onshore / wind_offshore, generation_actual_tso, Electricity produced by power plants connected to transmission grid), (ISO 3166 area code and name or control area or bidding zone, solar / wind / wind_onshore / wind_offshore, capacity, Aggregated installed capacity of power plants (actual availability not accounted for)), (ISO 3166 area code and name or control area or bidding zone, solar / wind / wind_onshore / wind_offshore, profile, Share of installed capacity producing), (ISO 3166 area code and name or control area or bidding zone, price, day_ahead, Day-ahead spot price)]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd; pd.read_csv('input/notation.csv', index_col=list(range(4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. License"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook as well as all other documents in this repository is published under the [MIT License](LICENSE.md)."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "nav_menu": {
    "height": "120px",
    "width": "252px"
   },
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}