Skip to content

1. Structure of the REopt API

Jump to bottom
Hallie Dunham edited this page Aug 9, 2023 · 30 revisions

Structure Overview

The REopt® API is built primarily in Python 3.6, with the exception of the code used to call the optimization kernel, which is written in Julia. In the REopt_API version 3 and higher, the Julia JuMP model (and all of the supporting functions) are now housed in the publicly-registered Julia package, REopt.jl. The API is served using the Django 2.2 framework, with a PostgreSQL database backend where input, output, and error data models are stored. Task management is achieved with Celery 4.3, which uses Redis as a message broker (a database for storing and retrieving Celery tasks). The figure below shows a how each of these pieces interact.

API Versioning

We typically update the API version when we make major changes to default values or breaking changes. Please see the documentation at https://developer.nrel.gov/docs/energy-optimization/reopt/ for REopt API versions and their urls. API v3 uses the REopt Julia Package as the optimization back-end, whereas v1 and v2 access an older version of this Julia code contained within the julia_src directory of the REopt_API repository. The v3 Python code is housed in the reoptjl directory of this repository and the v1/v2 Python code in the reo directory.

API Endpoints

Base url: https://developer.nrel.gov/api/reopt/
API_KEY parameter must be added to url
Run a REopt optimization: /job?
Get descriptions of possible inputs: /help?
Get descriptions of possible inputs: /job/inputs?
Get descriptions of possible outputs: /job/outputs?
Get CHP defaults, which depend on other CHP inputs: /chp_defaults?
Get absorption chiller defaults, which depend on other absorption chiller inputs: /absorption_chiller?
Get grid emissions for a location: /emissions_profile?
Get EASIUR health emissions cost data for a location: /easiur_costs?
Get a simulated load profile: /simulated_load?
Get a summary of a user's optimization jobs: /<user_uuid>/summary?
Get a summary of a subset of a user's optimization jobs: /<user_uuid>/summary_by_chunk?
Unlink an optimization job from a user: /<user_uuid>/unlink/<run_uuid>?
Get outage start times centered or starting on peak load: /peak_load_outage_times?
Check if a URDB label is in a set of invalid labels: /invalid_urdb?
Get a summary of a yearly CHP unavailability profile: /schedule_stats?

Workflow for REopt Jobs

v1-v2 Workflow for REopt jobs

The REopt API has multiple endpoints. The endpoint for optimizing an energy system is described here.

The steps behind an optimization are:

  1. A POST is made at <host-url>/v1/job (for a description of all input parameters GET <host-url>/v1/help)
  2. reo/api.py validates the POST, sets up the four celery tasks, and returns a run_uuid
  3. reo/scenario.py sets up all the inputs for the two optimization tasks
  4. Two reo/src/run_jump_model.py tasks are run in parallel, one for the business-as-usual costs, and one for the minimum life cycle cost system
  5. reo/process_results.py processes the outputs from the two REopt tasks and saves them to database.

Structural changes from v1/v2 to v3

From the user's perspective, the primary changes are:

  • Partial "flattening" of the inputs/outputs nested dictionaries (outer "Scenario" key has been removed and other models are no longer nested under the "Site" key).
  • Input/output reorganization and name changes for improved clarity (see the REopt analysis wiki for a full mapping between old and new inputs and outputs).

For developers of the API, more significant changes have been made to improve the development experience.

Django Models and Input Validation

In v1/v2, a single Django Model (and thus database table) contains both the inputs and outputs for each conceptual model (e.g. PV or ElectricTariff). In v3 the inputs and outputs are split into two separate Models (eg. ElectricTariffInputs and ElectricTariffOutputs). This division greatly simplifies creating the API response as well as saving and updating the data models.

In v1/v2, for each REopt optimization job, all Django Models are created and saved in the database. In v3, only the tables of Models relevant to the job a have rows inserted. For example, if a scenario is not modeling Wind then no WindInputs nor WindOutputs are instantiated and no "Wind" key would be present in the API response data.

InputValidator.clean_fields()

In v1/v2, the nested_inputs_definitions is used as a source of data types, limits, defaults, and descriptions, which is used in ValidateNestedInput to validate user's POSTs. In v3, we take advantage of Django's built-in data validation to accomplish most of what ValidateNestedInput does. When defining fields of Django Models, we can use built in functionality to:

  1. validate field types
  2. validate field min/max values
  3. limit fields to certain values
  4. provide help_text for each field
  5. set defaults

For example, the time_steps_per_hour field in v3 looks like:

time_steps_per_hour = models.IntegerField(
    default=TIME_STEP_CHOICES.ONE,
    choices=TIME_STEP_CHOICES.choices,
    help_text="The number of time steps per hour in the REopt model."
)

where TIME_STEP_CHOICES is defined as:

class TIME_STEP_CHOICES(models.IntegerChoices):
    ONE = 1
    TWO = 2
    FOUR = 4

And for an example of a min/max validation:

timeout_seconds = models.IntegerField(
    default=420,
    validators=[
        MinValueValidator(1),
        MaxValueValidator(420)
    ],
    help_text="The number of seconds allowed before the optimization times out."
)

After instantiating a Django model with a user's input values in the new reoptjl endpoint we call the Model.clean_fields() method on all of the input data models. The Model.clean_fields() method checks data types, min/max values, choices, and sets defaults. Calling the Model.clean_fields method is done in the new InputValidator (in reoptjl/validators.py) for every input model.

InputValidator.clean()

The Django Model also has a clean method that by default is empty. In the new reoptjl endpoint we "over-ride" this method to perform custom validation as needed. For example, the ElecricLoadInputs class has a clean method defined as follows:

    def clean(self):
        error_messages = []

        # possible sets for defining load profile
        if not at_least_one_set(self.dict, self.possible_sets):
            error_messages.append((
                "Must provide at valid at least one set of valid inputs from {}.".format(self.possible_sets)
            ))

        if error_messages:
            raise ValidationError(' & '.join(error_messages))

where:

possible_sets = [
    ["loads_kw"],
    ["doe_reference_name", "monthly_totals_kwh"],
    ["annual_kwh", "doe_reference_name"],
    ["doe_reference_name"]
]

The InputValidator calls the Model.clean() method on all of the input models (after calling the Model.clean_fields() method).

InputValidator.cross_clean()

Finally, there are some cases for input validation that require comparing fields from two different Django Models. For example, the length of any time-series input value, such as ElectricLoadInputs.loads_kw, needs to align with the Settings.time_steps_per_hour. This type of cross-model validation is done after the clean_fields and clean methods, and is called cross_clean. The cross_clean method is part of the InputValidator and is called in reoptjl/api.py as the final step for validating the job inputs.

Celery Tasks

In v1 each optimization job consists of four celery tasks (setup_scenario, one optimal and one baurun_jump_model, and process_results). In the new reoptjl endpoint there is only one celery task per optimization job. This has been accomplished by running the BAU and optimal scenarios in parallel in the Julia code and combining the scenario set up and post process steps into a single celery task.

Julia code

In v1 of the API the julia_src/ directory includes the code to build and optimize the JuMP model. In the new reoptjl endpoint all of the Julia code is housed in a Julia Module that is publicly registered. This means that the JuMP model (and all of the supporting functions) are now in a separate repository and that the REopt model can be used in Julia with just a few lines of code. For example:

using REoptLite, Cbc, JuMP
model = JuMP.Model(Cbc.Optimizer)
results = REoptLite.run_reopt(m, "path/to/scenario.json")

In the new reoptjl endpoint of the API we use the Julia package for REoptLite in a similar fashion to the last example. (See julia_src/http.jl for more details). Note that in the new reoptjl endpoint the BAU and Optimal scenarios are run in parallel in Julia, and running the BAU scenario is optional (via Settings.run_bau).

Migration of Python code to Julia

In creating the Julia Module (or "package") for REoptLite much of the inputs set up code and post-processing has shifted from the API to the Julia package. By porting the setup and post-processing code to the Julia package anyone can now use REoptLite in Julia alone, which makes it much easier to add inputs/outputs, create new constraints, and in general modify REoptLite as one sees fit.

For past and recent developers of the REopt Lite API, here is a pseudo-map of where some of API code responsibilities has been moved to in the Julia package:

  • reo/src/techs.py -> broken out into individual files for each tech in the src/core/ directory, e.g. pv.jl, wind.jl, generator.jl
  • reo/scenario.py -> src/core/scenario.jl
  • reo/src/pvwatts.py + wind_resource.py + wind.py + sscapi.py -> src/core/prodfactor.jl
  • reo/src/data_manager.py + julia_src/utils.jl -> src/core/reopt_inputs.jl
  • reo/process_results.py -> src/results/ contains a main results.jl as well as results generating functions for each data structure (e.g. src/results/pv.jl)
  • julia_src/reopt_model.jl -> src/core/reopt.jl + src/constraints/*

Also, here is a pseudo-map of where some of the reo/ code is now handled in the reoptjl/ app:

  • reo/nested_inputs.py + nested_outputs.py + models.py -> reoptjl/models.py
  • reo/process_results.py -> reoptjl/src/process_results.py (only saving results to database now since results are created in Julia)
  • reo/api.py -> reoptjl/api.py
  • reo/views.py -> reoptjl/views.py
  • reo/src/run_jump_model.py -> reoptjl/src/run_jump_model.py

For more information on the Julia package please see the developer section in the Julia package documentation.

Modifications of mathematical model

The primary change to the mathematical model in the new reoptjl endpoint is that all binary variables (and constraints) are added conditionally. For example, when modeling PV, Wind, and/or Storage without a tiered ElectricTariff no binary variables are necessary. However, adding a Generator or a tiered ElectricTariff does require binaries.

Also related to binary variables, the approach to modeling the net metering vs. wholesale export decision has been simplified:

In v1 the approach to the NEM/WHL binaries is as follows. First, binNMIL is used to choose between a combined tech capacity that is either above or below net_metering_limit_kw. There are two copies of each tech model made for both the above or below net_metering_limit_kw. If the combined tech capacity is below net_metering_limit_kw then the techs can export into the NEM bin (i.e. take advantage of net metering). If the combined tech capacity is greater than net_metering_limit_kw then the techs can export into the WHL bin. In either case the techs can export into the EXC bin. Second, this approach also requires a tech-to-techclass map and binaries for constraining one tech from each tech class - where the tech class contains the tech that can net meter and the one that can wholesale.

In the new approach of the new reoptjl endpoint there there is no need for the tech-to-techclass map and associated binaries, as well as the duplicate tech models for above and below net_metering_limit_kw. Instead, indicator constraints are used (as needed) for binNEM and binWHL variables, whose sum is constrained to 1 (i.e. the model can choose only NEM or WHL, not both). binNEM is used in two pairs of indicator constraints: one for the net_metering_limit_kw vs. interconnection_limit_kw choice and another set for the NEM benefit vs. zero NEM benefit. The binWHL is also used in one set of indicator constraints for the WHL benefit vs. zero WHL benefit. The EXC bin is only available if NEM is chosen.

See https://github.com/NREL/REoptLite/blob/d083354de1ff5d572a8165343bfea11054e556fc/src/constraints/electric_utility_constraints.jl for details.

Clone this wiki locally