NREL's HSDS service provides public access to several of DOEs high-value datasets available internally on the Kestrel HCP System. Due to the high demand for the HSDS service, users with access to NREL’s HPC should pursue direct access through that system.
This repository provides examples and convenience functions for accessing the below datasets using HSDS:
This service operates with the HSDS software developed by the HDF Group and is hosted on Amazon Web Services (AWS) using a combination of EC2 (Elastic Compute) and S3 (Scalable Storage Service). You can read more about the HSDS service in this slide deck.
To get started, download Anaconda or another distribution of Python, install the h5pyd library:
pip install --user h5pyd
Next you'll need to configure HSDS:
hsconfigure
and enter at the prompt:
hs_endpoint = https://developer.nrel.gov/api/hsds
hs_username =
hs_password =
hs_api_key = 3K3JQbjZmWctY0xmIfSYvYgtIcM3CN0cb1Y2w9bf
IMPORTANT: The example API key here is for demonstation and is rate-limited per IP. To get your own API key, visit https://developer.nrel.gov/signup/
You can also add the above contents to a configuration file at ~/.hscfg
Finally, you can use Jupyter Notebook to view the example notebooks:
cd notebooks
jupyter notebook
You can stand up a local HSDS server on an EC2 instance to improve the HSDS throughput versus the NREL developer API. Generally you should follow these instructions from the HSDS documentation. Here are a few tips and tricks to get everything connected to the NREL bucket:
To install docker and docker-compose on an EC2 instance (if not already installed):
sudo amazon-linux-extras install dockersudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-composesudo chmod +x /usr/local/bin/docker-composesudo groupadd dockersudo usermod -aG docker $USERnewgrp dockersudo service docker startdocker run hello-world
Your ~/.hscfg file should look like this:
# local hsds server
hs_endpoint = http://localhost:5101
hs_username = admin
hs_password = admin
hs_api_key = None
hs_bucket = nrel-pds-hsds
The following environment variables must be set:
export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
export BUCKET_NAME=${YOUR_S3_BUCKET_NAME_HERE}
export AWS_REGION=us-west-2
export AWS_S3_GATEWAY=http://s3.us-west-2.amazonaws.com/
export HSDS_ENDPOINT=http://localhost:5101
export LOG_LEVEL=INFO
A few miscellaneous tips:
- You can list the available docker images with
docker images - You can delete the docker HSDS image with
docker rmi $IMAGE_ID(useful to reset the docker image) - If you have AWS permissions issues try using a non-root IAM user with the corresponding AWS credentials as environment variables
- You can stand up parallel docker HSDS servers by running
sh runall.sh -8
This software is currently maintained by Reid Olson (reid.olson@nrel.gov). This software was initially developed by Caleb Phillips (caleb.phillips@nrel.gov), John Readey (jreadey@hdfgroup.org), Jordan Perr-Sauer (Jordan.Perr-Sauer@nrel.gov), and Michael Rossol to support the NREL Research Data Initiative and WIND Toolkit at the National Rewnable Energy Laboratory in Golden, Colorado, USA.
Copyright (c) 2017, National Renewable Energy Laboratory (NREL) All rights reserved. See LICENSE for additional information.
