All input and output data can be found in the data folder on Dropbox. Licenses and terms of the input data used can be found in their corresponding folders.
- entsoe-api - generation and load data at TSO level for each bidding zone, downloaded from the ENTSO-E Transparency Platform
- met - meteorological data, grouped by country, downloaded from country-specific meteorological services
- nuts - territorial units, downloaded from Eurostat
- output - output data
Generation and load data for each bidding zone are downloaded from the ENTSO-E Transparency Platform [@ENTSO]. The following descriptions of the data are from ENTSO-E Transparency Platform's Knowledge Base [@Help]. Three types of data will be used.
- Actual aggregated net generation output (MW) per market time unit and per production type
- Published no later than one hour after the operational period
- Computed as the average of all available instantaneous net generation output values on each market time unit
- If unknown, it is estimated
- The actual generation of small-scale units might be estimated if no real-time measurement devices exist
This data contains information about production units (existing and planned) with an installed generation capacity of at least 100 MW, which includes the following:
- unit name
- code
- installed net generation capacity (MW)
- voltage connection level (kV)
- bidding zone (denoted using Energy Identification Codes (EICs))
- production type (e.g., fossil gas, wind offshore)
This information is published annually at the start of the year and is valid for the three following years.
- Actual total load per bidding zone per market time unit
- The total load is defined as equal to the sum of power generated by plants on both TSO/DNO networks, from which is deduced:
- the balance (export-import) of exchanges on interconnections between neighbouring bidding zones
- the power absorbed by energy storage resources
- The information is published no later than one hour after the end of the operating period
- Calculated using the average of real-time load values per bidding zone per market time unit
- Actual total Load (including losses without stored energy) = Net Generation – Exports + Imports – Absorbed Energy
- Net generation is preferred, but gross generation could be used where it is available with the better precision
- TSOs should decide gross or net generation will be used but the net/gross characteristic should be consistent per bidding zone
- Absorbed energy is also provided as separate information with the aggregated generation output of the hydro pumped storage
- The physical flow on the tie line is measured as agreed by neighbouring TSOs or bidding zones, where applicable
ENTSO-E Transparency Platform's Restful application programming interface (API) can be used to automate the data extraction process [@ENTSO16], [@Trans]. Registration to the transparency platform is required to access the API. The security token can be requested by sending an email to the ENTSO-E Helpdesk.
The ENTSO-E API Python client [@Energ19] is used to easily query the required data and return them as Pandas dataframes or series. The queries for generation and installed generation capacity per unit return dataframes, while the query for load returns a series. entsoe_api.py is the script used to perform this.
import pandas as pd
from entsoe import EntsoePandasClient
from entsoe.mappings import DOMAIN_MAPPINGS, BIDDING_ZONES
# combine domain and bidding zone keys and values into the DOMAIN_MAPPINGS dictionary
DOMAIN_MAPPINGS.update(BIDDING_ZONES)# import security token saved in a separate file (login.py)
from login import token
# use security token to access the api through the entsoe pandas client
client = EntsoePandasClient(api_key=token)The bidding zones in the North Sea region, mapped to their corresponding EICs as shown in @tbl:eic, are used when querying using the Pandas client. Note that DE-LU only works for timestamps starting 01/10/2018. Use DE-AT-LU for timestamps prior to this date.
| Bidding zone | EIC |
|---|---|
| BE | 10YBE----------2 |
| DE-LU | 10Y1001A1001A82H |
| DK-1 | 10YDK-1--------W |
| DK-2 | 10YDK-2--------M |
| FR | 10YFR-RTE------C |
| GB | 10YGB----------A |
| IE-SEM | 10Y1001A1001A59C |
| NL | 10YNL----------L |
| NO-1 | 10YNO-1--------2 |
| NO-2 | 10YNO-2--------T |
| NO-3 | 10YNO-3--------J |
| NO-4 | 10YNO-4--------9 |
| NO-5 | 10Y1001A1001A48H |
| SE-1 | 10Y1001A1001A44P |
| SE-2 | 10Y1001A1001A45N |
| SE-3 | 10Y1001A1001A46L |
| SE-4 | 10Y1001A1001A47 |
: Bidding zones in the North Sea region and their corresponding EICs. {#tbl:eic}
Historical market data from Nord Pool is stored as .xls files can be accessed using the following URL:
https://www.nordpoolgroup.com/globalassets/marketdata-excel-files/FILENAME.xls
REMIT Urgent Market Messaging (UMM) is used by market operators and companies involved in balancing and settlements, including Nord Pool and ELEXON, for market members and participants to publish information on outages in production, consumption and transmission and other relevant market information according to European transparency regulations [@REMIT16], [@NordP], [@Abouta], [@Aboutb], [@Publi]. The information is publicly available and is classified as follows:
- Event
- Unavailability of electrical facilities (production, consumption, transmission)
- Market information
- Infrastructure (the electrical facility(ies) that are affected)
- Areas (bidding zone and/or country and/or control area)
- Types of unavailability (planned or unplanned)
- Stations (production and consumption units)
- Fuel types (same as the production units used by ENTSO-E)
- Connections (between control areas and/or bidding zone)
- Status (active or cancelled)
- Publishers (companies that publish the information)
- Market participants
- Publication time
- Event start and stop time
- Duration of event
- Unavailable capacity (MW)
- Available capacity (MW)
- Assets (grid components, e.g., AC link, DC link, transformer, substation)
- Remarks
The Royal Meteorological Institute of Belgium
Weather data for Germany is extracted from Deutscher Wetterdienst's Climate Data Center (CDC) OpenData [@Wette]. windHourly_de.py and solarHourly_de.py are the script used to extract hourly wind and solar data respectively.
Danish Meteorological Institute
Royal Netherlands Meteorological Institute
Norwegian Meteorological Institute