india-forecast-app

Runs wind and PV forecasts for India and saves to database

The model

The ML model is from PVnet and uses ocf_datapipes for the data processing For both Wind and Solar we use ECMWF data and predict 48 hours into the future.

Wind

The latest change is to use a patch size of 84x84, an increase from 64x64. This is to allow for more context in the image and to allow for the model to learn more about the wind patterns.

The validation error is ~ 7.15% (normalized using the maximum wind generation)

• wandb link
• hugging face link

The weather variables are currently

• t2m
• u10
• u100
• u200
• v10
• v100
• v200

We add some model smoothing

• feathering feathering close to current generation:
• smoothing over 1 hour rolling window

PV

The validation error is ~ 2.28% (normalized using the maximum solar generation)

• wandb link
• hugging face link

The weather variables are

• hcc
• lcc
• mcc
• prate
• sde
• sr
• t2m
• tcc
• u10
• v10
• dlwrf
• dswrf

Adjuster

As well as the main ml model, we also calculate a new model. This new model tweaks the original model and adjusts it based on its performance over the last 7 days.

The model takes the initial results from ml model, then looks up the ME over the last 7 days, and then adjusts the forecast values accordingly. This should get rid of any systematic errors. The adjuster values are dependent on time of data and forecast horizon, e.g. a forecast made at 15.00 for 17.00 looks back at all the forecasts made at 15.00 for 17.00 in the last 7 days.

Install dependencies (requires poetry)

poetry install

Linting and formatting

Lint with:

make lint

Format code with:

make format

Running tests

make test

⚠️ Note: one test for the AD model is skipped locally unless the HF token is set, this HF token can be found in AWS Secret Manager under {environment}/huggingface/token and then can be set via export HUGGINGFACE_TOKEN={token_value} in the repo to run the additional test. In CI tests this secret is set so the test will run there.

Running the app locally

Replace {DB_URL} with a postgres DB connection string (see below for setting up a ephemeral local DB)

If testing on a local DB, you may use the following script to seed the the DB with a dummy user, site and site_group.

DB_URL={DB_URL} poetry run seeder

⚠️ Note this is a destructive script and will drop all tables before recreating them to ensure a clean slate. DO NOT RUN IN PRODUCTION ENVIRONMENTS

This example runs the application and writes the results to stdout

DB_URL={DB_URL} NWP_ZARR_PATH={NWP_ZARR_PATH} poetry run app

To save batches, you need to set the SAVE_BATCHES_DIR environment variable to directory.

### Starting a local database using docker

```bash
    docker run \
        -it --rm \
        -e POSTGRES_USER=postgres \
        -e POSTGRES_PASSWORD=postgres \
        -p 54545:5432 postgres:14-alpine \
        postgres

The corresponding DB_URL will be

postgresql://postgres:postgres@localhost:54545/postgres

Building and running in Docker

Build the Docker image

make docker.build

Create a container from the image. This example runs the application and writes the results to stdout.
Replace {DB_URL} with a postgres DB connection string.
N.B if the database host is localhost on the host machine, replace localhost with host.docker.internal so that docker can access the database from within the container

docker run -it --rm -e DB_URL={DB_URL} -e NWP_ZARR_PATH={NWP_ZARR_PATH} ocf/india-forecast-app

Notes

This repo makes use of PyTorch (torch and torchvision packages) CPU-only version. In order to support installing PyTorch via poetry for various environments, we specify the exact wheels for each environment in the pyproject.toml file. Some background reading on why this is required can be found here: https://santiagovelez.substack.com/p/how-to-install-torch-cpu-in-poetry?utm_campaign=post&utm_medium=web&triedRedirect=true