Developed by Energy Efficiency and Conservation Authority, BusinessNZ Energy Council and Paul Scherrer Institut
This repository houses the Excel files defining the TIMES-NZ (The Integrated MARKAL-EFOM System - New Zealand) energy system model configuration. It is intended for use by researchers and analysts working with energy models in New Zealand.
- For a guide on getting started with the TIMES-NZ model, refer to the TIMES-NZ-internal-guide-EECA repository.
- For step by step system configuration instructions, refer to the SystemConfigurationGuide.md file in this repository.
- For detailed documentation on the workbooks and their contents, refer to the structured documentation in the docs directory.
The TIMES-NZ project is divided across three main repositories, each serving a distinct purpose in the overall modeling and visualization process. The repositories and their functions are as follows:
-
TIMES-NZ-Model-Files: Houses the Excel files defining the TIMES-NZ energy system model configuration, along with additional resources and tools for change tracking.
-
TIMES-NZ-GAMS-Files: Contains the GAMS model files for TIMES-NZ scenarios and scripts to enable reproducible scenario execution.
-
TIMES-NZ-Visualisation: Provides a tool for visualizing the results of TIMES-NZ scenarios using an RShiny application.
- Begin by configuring the TIMES-NZ model in the TIMES-NZ-Model-Files repository, and using VEDA to run the model. Work is done in a development branch and merged to the main branch after review - i.e. we follow the standard GitHub Flow.
- When ready to release a new version of the model, name your scenarios to include the release tag (use the naming convention
kea-v2_1_2where kea is an identifier for the scenario set, and v2_1_2 indicates the version number). - Once the configuration is complete and scenarios are named, commit your changes and tag the release (e.g., v2.1.2). Release tagging allows users and collaborators to easily identify and revert to specific versions of the model.
- Move to the TIMES-NZ-GAMS-Files repository. Ensure that your local environment is set up correctly with GAMS installed, including a valid license. Verify that the version of GAMS you're using is compatible with the model files.
- Fetch the GAMS 'dd' files and the '.run' file associated with the scenario using the
get_from_veda.ps1PowerShell script. - Run the GAMS solver using the
run_times_scenario.ps1PowerShell script, and the output files will be generated.
- After running the scenario, take the output vd file and copy it into the TIMES-NZ-Visualisation's
data_cleaningsubdirectory. Depending on the details of the user's system configuration, this could look something like the following:
$times_nz_gams_files_local_repo = "C:\Users\$env:USERNAME\git\TIMES-NZ-GAMS-Files"
$times_nz_visualization_local_repo = "C:\Users\$env:USERNAME\git\TIMES-NZ-Visualisation"
$scenario='tui-v2_1_2'
cp $times_nz_gams_files_local_repo\$scenario\$scenario.vd $times_nz_visualization_local_repo\data_cleaning- Ensure that the file paths are correctly set in the script to avoid any file not found errors.
- Run the data extraction script in the
data_cleaningsubdirectory to process the output and generate a file ready for visualization (e.g., to process a scenario named tui-v2_1_2, you might run a script that generates a file named tui-v2_1_2.rda). - After running the extraction script, it's a good practice to verify the integrity of the generated data file. Check for any inconsistencies or missing data that might impact the accuracy of your visualizations.
- Launch the RShiny application to visualize the scenario results interactively.
- Navigate to the directory where the RShiny application files for TIMES-NZ are stored. This should be within your local copy of the TIMES-NZ-Visualisation repository.
- Ensure that you have R and the Shiny package installed on your machine. If not, you can install them from the Comprehensive R Archive Network (CRAN).
- Open an R console and set the working directory to the location of the RShiny application files.
- Run the RShiny application by executing the app script, usually with a command like
shiny::runApp().
For more specific instructions, including the commands and scripts to be used at each step, please refer to the README files in the respective repositories:
For any questions or support, you can contact [dataandanalytics@eeca.govt.nz].
Note: The above workflow describes current state. In the future we hope to:
- automate the process of generating the Excel configuration files using Reproducible Analytical Pipelines (RAPs);
- automate the production of GAMS files directly from the excel files without the use of VEDA.
Before you start working with this project, you will need the following tools installed on your machine:
-
Git: Used for version control. Download and install from https://git-scm.com/downloads.
-
Docker: Required for running the
times-excel-readertool. Download and install from https://www.docker.com/get-started. -
GAMS installed on your system.
-
A valid GAMS license placed in the system's designated location (e.g.,
C:\GAMS\42\gamslice.txt). -
A working VEDA (VEDA 2.0 localhost) installation (instructions here).
Ensure that you are able to use Git via the command line in a PowerShell-like environment. This README assumes Windows 10 or higher.
Here's how to get the project installed on your local machine for development:
git clone git@github.com:EECA-NZ/TIMES-NZ-Model-Files.git
cd TIMES-NZ-Model-FilesIf you're new to Git and GitHub, here's a simple guide on how to get started.
- Make sure you have Git installed on your machine. You can download it from here: https://git-scm.com/downloads.
First, you'll need to make a copy of this repository on your local machine. This is called "cloning". If you're using command line, use the following command:
git clone git@github.com:EECA-NZ/TIMES-NZ-Model-Files.gitThis will work if you have configured your GitHub account with ssh keys for your laptop (instructions to do so here).
After the clone, you can list the tags with git tag -l and then checkout a specific tag:
git checkout tags/<tag_name>This allows us to roll back our model to earlier releases and replicate earlier results. However, for improving the model, our development pattern is to work from the head of the main branch. Before making any changes, ensure that you are on the head of main by running
git checkout main
git pullThese commands ensure you are working from the most up-to-date state of the project: the head of the main branch. To get the above commands to work, you may need to discard any local changes that you don't need. You can do this by identifying any local changes using
git statuswhich will list any modified files, new files not under version control and other local uncommitted changes to the project. Revert all of these by first (important!) backing up any changes that you want to retain, then running
rm <any-local-untracked-file-that-is-not-part-of-the-project>
git checkout <any-local-file-that-has-changes-to-discard>when you have completely tidied up your local copy of the repo, and moved to the head of main, running git status will produce a very minimal output that looks like
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
Before making changes, create a new branch. This keeps your changes isolated until they're ready to be merged into the main branch. After ensuring you are on the head of the main branch, here's how you can branch from it:
Command line:
git checkout -b <your-branch-name>This command has two effects:
- it branches from your current location in the git project (the head of main) and creates a new branch, named , which currently exists only on your local machine
- it switches your local copy of the git repository to be that branch
Any changes that you now commit will be committed to that branch, so you can modify the model without affecting the
mainbranch until you are ready to merge the changes.
Now you can start making changes to the code. You can create new files, modify existing ones, and delete what you don't need. Keep your changes focused and related to a single feature or fix.
When changes are trivial, ensure that they haven't affected model behaviour by noting the value of the objective function on model convergence before and after the change.
Use the Dockerized times-excel-reader tool to generate a summary of the model at TIMES-NZ/raw_table_summary/raw_tables.txt. This will help reviewers see the changes you've made to the Excel files by viewing the diff of this text file in the pull request.
To use Docker on a Windows machine, you may need to start Docker Desktop to initialize the docker daemon.
After changing the TIMES-NZ model excel files, before committing the changes and making a pull request, please run the following docker run command to generate a summary of the new model. Before running the following docker command for the first time, you will need to build the Docker image:
docker build -t times_excel_reader .This will also need to be done when there are updates to the Dockerfile or requirements.txt. Otherwise, you can just run the container as needed with the following docker run command:
docker run -it --rm --name my_times_reader -v ${PWD}/TIMES-NZ:/usr/src/app/TIMES-NZ times_excel_readerHaving updated raw_tables.txt, re-run the python script create_readme_files.py the sits alongside the raw_tables.txt file. This is done by running the following command in the TIMES-NZ/raw_table_summary directory:
python create_readme_files.pyNote:
-
If you are unable to run the
times-excel-readertool on your machine, you can open a pull request without changing theraw_tables.txtfile. The GitHub Actions check will fail on your PR, and the check will create an artifact containing the updatedraw_tables.txt. You can download this and update the one on your branch instead. -
The
times-excel-readertool generates file paths with forward slashes (/). If you're working on Windows, the file paths in theraw_tables.txtmay contain backslashes (\). To maintain consistency across environments, we recommend running the tool in a Linux-like environment (e.g., using WSL on Windows) or normalizing file paths using a tool likesedbefore committing your changes.
After you've made some changes, you need to "commit" them. This takes a snapshot of your changes, which you can then push to GitHub. Important: Please take care to ensure that only changes you intend to commit are committed!
Command line:
# Add changes to the staging area
git status
git add [specific changed file or files related to commit]
# Commit the changes
git commit -m "Your descriptive commit message"If you have changed the TIMES configuration files, one of the files you commit should be TIMES-NZ/raw_table_summary/raw_tables.txt.
After committing, you need to "push" your changes to GitHub. This makes them available to others.
Command line:
git push --set-upstream origin <your-branch-name>Finally, you can ask for your changes to be merged into the main branch by creating a "pull request".
Go to the repository on GitHub, and click on the "Pull request" button. Select your branch from the dropdown, write a brief description of your changes, and click "Create pull request".
Incorporates changes from a remote repository into the current branch. If the current branch is behind the remote, then by default it will fast-forward the current branch to match the remote. For instance, as before, to ensure you are on the head of the main branch:
Command line:
git checkout main
git pullGuide on getting started with the TIMES-NZ model stored here: https://github.com/EECA-NZ/TIMES-NZ-internal-guide-EECA
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please ensure that you have the necessary dependencies installed, as outlined at the beginning of this document.
