Merge pull request #101 from dblodgett-usgs/master

documentation and make sure things run fixes #98

README.md

@@ -1,18 +1,28 @@
 # NLDI Database
 
-This repository contains Liquibase changelogs for creating and managing the NLDI PostGIS database. The [package registry](https://github.com/internetofwater/nldi-db/pkgs/container/nldi-db) has three pre-built Docker images that are ready to use. The demo image contains a subset of data to use for testing. The Liquibase image can be used to run the Liquibase changelogs against any PostGIS database. Finally, the CI database is a blank database (schema only) that is used for integration tests in the [NLDI Services](https://github.com/internetofwater/nldi-services) and [NLDI Crawler](https://github.com/internetofwater/nldi-crawler) projects.
-
-## Table of Contents
-- [Contributing](#contributing)
-- [Development](#development)
-- [Configuration](#configuration)
-	- [Environment Variables](#environment-variables)
-		- [Definitions](#definitions)
-- [Running](#running)
-	- [Liquibase](#liquibase)
-	- [CI](#ci)
-	- [Demo Database](#demo-database)
-- [Docker Compose](#docker-compose)
+This repository contains Liquibase changelogs for creating and managing the NLDI PostGIS database. The [package registry](https://github.com/internetofwater/nldi-db/pkgs/container/nldi-db) has three pre-built Docker images that are ready to use.  
+
+1. The demo image contains a subset of data to use for small scale testing.  
+1. The Liquibase image can be used to run the Liquibase changelogs against any PostGIS database.  
+1. The CI database is a blank database (schema only) that is used for integration tests in the [NLDI Services](https://github.com/internetofwater/nldi-services) and [NLDI Crawler](https://github.com/internetofwater/nldi-crawler) projects.  
+
+Full scale data is available from an open, requester pays, s3 bucket and can be loaded using the same pattern as the demo image but using different table artifacts:  
+`s3://nhgf-development.s3.amazonaws.com/dev/nldi-database-load/`
+
+Details of how to install this project for small scale (demo) or large scale are included below.
+
+### Table of Contents  
+	- [Contributing](#contributing)
+	- [Development](#development)
+		- [Configuration](#configuration)
+		- [Running](#running)
+			- [Liquibase](#liquibase)
+			- [CI](#ci)
+			- [Demo Database](#demo-database)
+		- [Docker Compose](#docker-compose)
+		- [Environment Variables](#environment-variables)
+			- [Definitions](#definitions)
+
 
 ## Contributing
 
@@ -22,9 +32,59 @@ To contribute a new data source to the NLDI, add a new line to [this TSV file](l
 
 The only requirement to set up a development environment is to have [Docker](https://docs.docker.com/get-docker/) installed on your system. Changes to any scripts or Liquibase changelogs can be tested with Docker as outline in the [running](#running) section below.
 
-## Configuration
+### Configuration
+
+The database and Liquibase Docker containers utilize various environment variables for passwords and connections. These must be set up prior to any use of these containers.
+
+See the [Environment Variables](#environment-variables) section for a full description.
+
+### Running
+
+Each Docker image can be built and run using the provided Docker Compose yaml file. If you run the images without building first, the latest image will be pulled from the GitHub registry.
+
+#### Liquibase
+
+The Liquibase changelogs can be tested locally by spinning up the generic PostGIS database (db) and the Liquibase container.
+
+```shell
+docker-compose up -d db
+docker-compose run liquibase
+```
+
+The local file system is mounted into the liquibase container. This allows you to change the liquibase and shell scripts and run the changes by just re-launching the liquibase container. Note that all standard Liquibase caveats apply.
+
+The PostGIS database will be available on your localhost's port `$DB_PORT`, allowing for visual inspection of the results.
+
+#### CI
+
+```shell
+docker-compose up ci
+```
+
+It will be available on you localhost's port `$DB_CI_PORT`.
+
+This database does not contain any data and is used to insert mock data for testing the NLDI services and NLDI Crawler.
+
+#### Demo Database
+
+```shell
+docker-compose up demo
+```
+
+It will be available on your localhost's port `$DB_DEMO_PORT`.
+
+### Docker Compose
+
+It is highly recommended to use Docker Compose to run the included Docker images, although they may also be run with typical Docker commands and additonal parameters.
+
+There are several commands that you will find useful during your testing.
+
+If you have started a container with `docker-compose up -d <container name>`, it can be stopped by pressing ctrl+C, or your machines equivalent, in the terminal that it was started from. Alternatively, you can run `docker-compose stop <container name>` from a separate terminal.
+
+Using `docker-compose run <container name>` is useful for running containers that will not remain running after execution.
+
+See the [Docker Compose documentation](https://docs.docker.com/compose/reference/) for other commands.
 
-The database and Liquibase Docker containers utilize various environment variables for passwords and connections.
 
 ### Environment Variables
 
@@ -39,7 +99,7 @@ NLDI_DB_OWNER_PASSWORD=changeMe
 NLDI_SCHEMA_OWNER_USERNAME=nldi_schema_owner
 NLDI_SCHEMA_OWNER_PASSWORD=changeMe
 NHDPLUS_SCHEMA_OWNER_USERNAME=nhdplus
-NLDI_READ_ONLY_USERNAME=read_only_user
+NLDI_READ_ONLY_USERNAME=nldi_data
 NLDI_READ_ONLY_PASSWORD=changeMe
 DB_CI_PORT=5445
 DB_DEMO_PORT=5432
@@ -73,50 +133,3 @@ N = None (optional)
 | DB_DEMO_PORT | The localhost port on which to expose the Demo database. | D |
 | DB_PORT | The localhost port on which to expose the script testing database container. | G |
 | DOCKER_MIRROR | Optional mirror URL that is prefixed when pulling Docker images. | N |
-
-## Running
-
-Each Docker image can be built and run using the provided Docker Compose yaml file. If you run the images without building first, the latest image will be pulled from the GitHub registry.
-
-### Liquibase
-
-The Liquibase changelogs can be tested locally by spinning up the generic PostGIS database (db) and the Liquibase container.
-
-```shell
-docker-compose up -d db
-docker-compose run liquibase
-```
-
-The local file system is mounted into the liquibase container. This allows you to change the liquibase and shell scripts and run the changes by just re-launching the liquibase container. Note that all standard Liquibase caveats apply.
-
-The PostGIS database will be available on your localhost's port `$DB_PORT`, allowing for visual inspection of the results.
-
-### CI
-
-```shell
-docker-compose up ci
-```
-
-It will be available on you localhost's port `$DB_CI_PORT`.
-
-This database does not contain any data and is used to insert mock data for testing the NLDI services and NLDI Crawler.
-
-### Demo Database
-
-```shell
-docker-compose up demo
-```
-
-It will be available on your localhost's port `$DB_DEMO_PORT`.
-
-## Docker Compose
-
-It is highly recommended to use Docker Compose to run the included Docker images, although they may also be run with typical Docker commands and additonal parameters.
-
-There are several commands that you will find useful during your testing.
-
-If you have started a container with `docker-compose up -d <container name>`, it can be stopped by pressing ctrl+C, or your machines equivalent, in the terminal that it was started from. Alternatively, you can run `docker-compose stop <container name>` from a separate terminal.
-
-Using `docker-compose run <container name>` is useful for running containers that will not remain running after execution.
-
-See the [Docker Compose documentation](https://docs.docker.com/compose/reference/) for other commands.

liquibase/scripts/dbDemo/z4_load_demo.sh

@@ -1,8 +1,8 @@
 #!/bin/bash 
 
-gunzip -c ${LIQUIBASE_HOME}/nhdplus.yahara.pgdump.gz | pg_restore -h 127.0.0.1 -p 5432 -U postgres -w -a -d ${NLDI_DATABASE_NAME}
-gunzip -c ${LIQUIBASE_HOME}/characteristic_data.yahara.pgdump.gz | pg_restore -h 127.0.0.1 -p 5432 -U postgres -w -a -d ${NLDI_DATABASE_NAME} -n characteristic_data
-gunzip -c ${LIQUIBASE_HOME}/nldi_data.crawler_source.pgdump.gz | pg_restore -h 127.0.0.1 -p 5432 -U nldi -w -a -d ${NLDI_DATABASE_NAME}
-gunzip -c ${LIQUIBASE_HOME}/feature_wqp_yahara.backup.gz | pg_restore -h 127.0.0.1 -p 5432 -U nldi -w -a -d ${NLDI_DATABASE_NAME}
-gunzip -c ${LIQUIBASE_HOME}/feature_huc12pp_yahara.backup.gz | pg_restore -h 127.0.0.1 -p 5432 -U nldi -w -a -d ${NLDI_DATABASE_NAME}
-gunzip -c ${LIQUIBASE_HOME}/feature_np21_nwis_yahara.backup.gz | pg_restore -h 127.0.0.1 -p 5432 -U nldi -w -a -d ${NLDI_DATABASE_NAME}
+gunzip -c ${LIQUIBASE_HOME}/nhdplus.yahara.pgdump.gz | pg_restore -h 127.0.0.1 -p 5432 -U ${NLDI_DB_OWNER_USERNAME} -w -a -d ${NLDI_DATABASE_NAME}
+gunzip -c ${LIQUIBASE_HOME}/characteristic_data.yahara.pgdump.gz | pg_restore -h 127.0.0.1 -p 5432 -U ${NLDI_DB_OWNER_USERNAME} -w -a -d ${NLDI_DATABASE_NAME} -n characteristic_data
+gunzip -c ${LIQUIBASE_HOME}/nldi_data.crawler_source.pgdump.gz | pg_restore -h 127.0.0.1 -p 5432 -U ${NLDI_DB_OWNER_USERNAME} -w -a -d ${NLDI_DATABASE_NAME}
+gunzip -c ${LIQUIBASE_HOME}/feature_wqp_yahara.backup.gz | pg_restore -h 127.0.0.1 -p 5432 -U ${NLDI_DB_OWNER_USERNAME} -w -a -d ${NLDI_DATABASE_NAME}
+gunzip -c ${LIQUIBASE_HOME}/feature_huc12pp_yahara.backup.gz | pg_restore -h 127.0.0.1 -p 5432 -U ${NLDI_DB_OWNER_USERNAME} -w -a -d ${NLDI_DATABASE_NAME}
+gunzip -c ${LIQUIBASE_HOME}/feature_np21_nwis_yahara.backup.gz | pg_restore -h 127.0.0.1 -p 5432 -U ${NLDI_DB_OWNER_USERNAME} -w -a -d ${NLDI_DATABASE_NAME}