LIPS : Learning Industrial physical simulation benchmark suite: the power grid case

This repository implements the benchmarking platform called LIPS and provides the necessary utilities to reproduce the generated datasets used in research.

The Readme file is organized as follows:

• 1 Introduction
  • 1.1 What is LIPS?
• 2 Usage example
  • 2.1 Train a simulator
  • 2.2 Reproducibility and evaluation
• 3 Installation
  • 3.1 Setup a Virtualenv (optional)
  • 3.2 Install from source
  • 3.3 To contribute
• 4 Access the Codabench page
• 5 Getting Started
• 6 Documentation
• 7 Contribution
• 8 License information

Introduction

Nowdays, the simulators are used in every domain to emulate a real-world situation or event or to reproduce the critical situations for which further investigation may be required. The simulators are based generally on physics equations and are costly in terms of time complexity.

What is LIPS

The learning industrial physical simulation benchmark suite allows to evaluate the performance of augmented simulators (aka surrogate models) specialized in a physical domain with respect to various evaluation criteria. The implementation is enough flexible to allow its adaptation to various domains such as power grids, transport, aeronotics etc. To do so, as it is depicted in the scheme provided in the figure below, the platform is designed to be modular and includes following modules:

• the Data module of the platform may be used to import the required datasets or to generate some synthetic data (for power grids for now)
• A simulator may access the provided data to train or evaluate the peformance. The developed platform gives also the flexibility to its users to design and implement their own simulators and evaluate its performance with baselines. Various baseline simulators are already implemented and could be used, e.g., Direct Current (DC) approximation and neural netowrk based simulators which are Fully Connected (FC) model and Latent Encoding of Atypical Perturbations network (LEAP net).
• The Evaluation module allows to select the appropriate criteria among the available implemented metrics. Four category of metrics are provided, which are :
  • ML-related metrics
  • Physic compliance
  • Industrial readiness
  • Generalization metrics

Usage example

Instantiate a benchmark for power grid use case

The paths should correctly point-out to generated data (DATA_PATH) and benchmark associated config file (CONFIG_PATH). The log path (LOG_PATH) could be set by the user.

from lips.benchmark import PowerGridBenchmark

benchmark1 = PowerGridBenchmark(benchmark_name="Benchmark1",
                                benchmark_path=DATA_PATH,
                                load_data_set=True,
                                log_path=LOG_PATH,
                                config_path=CONFIG_PATH
                               )

Train a simulator

A simulator (based on tensorflow) could be instantiated and trained if required easily as follows:

from lips.augmented_simulators.tensorflow_models import TfFullyConnected
from lips.dataset.scaler import StandardScaler

tf_fc = TfFullyConnected(name="tf_fc",
                         bench_config_name="Benchmark1",
                         scaler=StandardScaler,
                         log_path=LOG_PATH)

tf_fc.train(train_dataset=benchmark1.train_dataset,
            val_dataset=benchmark1.val_dataset,
            epochs=100
           )

For each architecture a config file is attached which are available here for powergrid use case.

Reproducibility and evaluation

The following script show how to use the evaluation capacity of the platform to reproduce the results on all the datasets. A config file (see here for powergrid use case) is associated with this benchmark and all the required evaluation criteria can be set in this configuration file.

tf_fc_metrics = benchmark1.evaluate_simulator(augmented_simulator=tf_fc,
                                              eval_batch_size=128,
                                              dataset="all",
                                              shuffle=False
                                             )

Installation

To be able to run the experiments in this repository, the users should install the last lips package from its github repository. The following steps show how to install this package and its dependencies from source.

Requirements

• Python >= 3.6

Setup a Virtualenv (optional)

Create a virtual environment

cd my-project-folder
pip3 install -U virtualenv
python3 -m virtualenv venv_lips

Enter virtual environment

source venv_lips/bin/activate

Install from source

git clone https://github.com/Mleyliabadi/LIPS
cd LIPS
pip3 install -U .
cd ..

To contribute

pip3 install -e .[recommended]

Codabench

To see the leaderboard for benchmarking tasks, refer to the codabench page of the framework, accessible from this link.

Getting Started

Some Jupyter notebook are provided as tutorials for LIPS package. They are located in the getting_started directories.

Documentation

The documentation is accessible from here.

To generate locally the documentation:

pip install sphinx
pip install sphinx-rtd-theme
cd docs
make clean
make html

Contribution

• Supplementary features could be requested using github issues.
• Other contributions are welcomed and can be integrated using pull requests.

FAQ

Pytorch

To be able to use the torch library with GPU, you should consider multiple factors:

• if you have a compatible GPU, in this case you can install the last cuda driver (11.6) and install torch using the following command:

pip install torch --pre --extra-index-url https://download.pytorch.org/whl/nightly/cu116

To take the advantage of the GPU when training models, you should indicate it via the device parameter as follows:

from lips.augmented_simulators.torch_models.fully_connected import TorchFullyConnected
from lips.augmented_simulators.torch_simulator import TorchSimulator
from lips.dataset.scaler import StandardScaler

torch_sim = TorchSimulator(name="torch_fc",
                           model=TorchFullyConnected,
                           scaler=StandardScaler,
                           device="cuda:0",
                          )

• Otherwise, if you want use only CPU for the training of augmented simulators, you could simply use the version installed following the the requirements and set the device parameter to cpu when training as follows:

torch_sim = TorchSimulator(name="torch_fc",
                           model=TorchFullyConnected,
                           scaler=StandardScaler,
                           device="cpu",
                          )

Tensorflow

To be able to use Tensorflow, you should already install a cuda compatible version with your tensorflow package. From Tensorflow 2.4 the cuda version >= 11.0 is required. Once you have get and installed cuda driver (we recommend version 11.5) from here, you should also get corresponding cuDNN package from here and copy the contents in corresponding folders in cuda installation directory. Finally, you should set some environment variables which are discussed in this link for both linux and windows operating systems. For windiows you can do the following in command line:

SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin;%PATH%
SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\extras\CUPTI\lib64;%PATH%
SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\include;%PATH%
SET PATH=C:\tools\cuda\bin;%PATH%
SET LD_LIBRARY_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\lib\x64

However, after setting variables if you encounter some *.dll not found when importing tensorflow library, you could indicating the path to cuda installation in your code before importing tensorflow package as follows:

import os
os.add_dll_directory("C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.5/bin")

And then you can test your installation by running:

import tensorflow as tf
tf.config.list_physical_devices()

Where in its output the GPU device should be appeared as following:

[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'),
 PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

License information

Copyright 2022-2023 IRT SystemX & RTE

IRT SystemX: https://www.irt-systemx.fr/
RTE: https://www.rte-france.com/

This Source Code is subject to the terms of the Mozilla Public License (MPL) v2 also available here