Generative Reinforcement Learning

English | 简体中文(Simplified Chinese)

GenerativeRL, short for Generative Reinforcement Learning, is a Python library for solving reinforcement learning (RL) problems using generative models, such as diffusion models and flow models. This library aims to provide a framework for combining the power of generative models with the decision-making capabilities of reinforcement learning algorithms.

Outline

• Features
• Framework Structure
• Integrated Generative Models
• Integrated Algorithms
• Installation
• Quick Start
• Documentation
• Tutorials
• Benchmark experiments

Features

• Support for training, evaluation and deploying diverse generative models, including diffusion models and flow models
• Integration of generative models for state representation, action representation, policy learning and dynamic model learning in RL
• Implementation of popular RL algorithms tailored for generative models, such as Q-guided policy optimization (QGPO)
• Support for various RL environments and benchmarks
• Easy-to-use API for training and evaluation

Framework Structure

Integrated Generative Models

	Score Matching	Flow Matching
Diffusion Model
Linear VP SDE	✔	✔
Generalized VP SDE	✔	✔
Linear SDE	✔	✔
Flow Model
Independent Conditional Flow Matching	🚫	✔
Optimal Transport Conditional Flow Matching	🚫	✔

Integrated Algorithms

Algo./Models	Diffusion Model	Flow Model
IDQL	✔	🚫
QGPO	✔	🚫
SRPO	✔	🚫
GMPO	✔	✔
GMPG	✔	✔

Installation

pip install GenerativeRL

Or, if you want to install from source:

git clone https://github.com/opendilab/GenerativeRL.git
cd GenerativeRL
pip install -e .

Or you can use the docker image:

docker pull opendilab/grl:torch2.3.0-cuda12.1-cudnn8-runtime
docker run -it --rm --gpus all opendilab/grl:torch2.3.0-cuda12.1-cudnn8-runtime /bin/bash

Quick Start

Here is an example of how to train a diffusion model for Q-guided policy optimization (QGPO) in the LunarLanderContinuous-v2 environment using GenerativeRL.

Install the required dependencies:

pip install 'gym[box2d]==0.23.1'

(The gym version can be from 0.23 to 0.25 for box2d environments, but it is recommended to use 0.23.1 for compatibility with D4RL.)

Download dataset from here and save it as data.npz in the current directory.

GenerativeRL uses WandB for logging. It will ask you to log in to your account when you use it. You can disable it by running:

wandb offline

import gym

from grl.algorithms.qgpo import QGPOAlgorithm
from grl.datasets import QGPOCustomizedTensorDictDataset
from grl.utils.log import log
from grl_pipelines.diffusion_model.configurations.lunarlander_continuous_qgpo import config

def qgpo_pipeline(config):
    qgpo = QGPOAlgorithm(config, dataset=QGPOCustomizedTensorDictDataset(numpy_data_path="./data.npz", action_augment_num=config.train.parameter.action_augment_num))
    qgpo.train()

    agent = qgpo.deploy()
    env = gym.make(config.deploy.env.env_id)
    observation = env.reset()
    for _ in range(config.deploy.num_deploy_steps):
        env.render()
        observation, reward, done, _ = env.step(agent.act(observation))

if __name__ == '__main__':
    log.info("config: \n{}".format(config))
    qgpo_pipeline(config)

For more detailed examples and documentation, please refer to the GenerativeRL documentation.

Documentation

The full documentation for GenerativeRL can be found at GenerativeRL Documentation.

Tutorials

We provide several case tutorials to help you better understand GenerativeRL. See more at tutorials.

Benchmark experiments

We offer some baseline experiments to evaluate the performance of generative reinforcement learning algorithms. See more at benchmark.

Contributing

We welcome contributions to GenerativeRL! If you are interested in contributing, please refer to the Contributing Guide.

Citation

If you find GenerativeRL useful in your research, please consider citing the following paper:

@misc{zhang2024generative_rl,
      title={Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective}, 
      author={Jinouwen Zhang and Rongkun Xue and Yazhe Niu and Yun Chen and Jing Yang and Hongsheng Li and Yu Liu},
      year={2024},
      eprint={2412.01245},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2412.01245}, 
}

Papers implemented in GenerativeRL

• Data-driven Aerodynamic Shape Optimization and Multi-fidelity Design Exploration using Conditional Diffusion-based Geometry Sampling Method (Yang et al. 2024)
• Pretrained Reversible Generation as Unsupervised Visual Representation Learning (Xue et al. 2024)

License

GenerativeRL is licensed under the Apache License 2.0. See LICENSE for more details.