diff --git a/README.md b/README.md index 170c9a4ef..4cdb48b20 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,9 @@ This is the home of [Delphi](https://delphi.cmu.edu/)'s epidemiological data API. See our [API documentation](https://cmu-delphi.github.io/delphi-epidata/) for details on the available data sets, APIs, and clients. -## Development Quickstart +# Development Quickstart + +## Setup Requires: Docker, possibly sudo access (depending on your Docker installation and OS). @@ -16,7 +18,7 @@ In the directory where you want to work run the following: ```sh # Make folder structure, download dependent repos, and symlink Makefile -$ curl "https://raw.githubusercontent.com/cmu-delphi/delphi-epidata/dev/dev/local/install.sh" | bash +$ curl "https://raw.githubusercontent.com/cmu-delphi/delphi-epidata/dev/dev/local/install.sh" | bash && cd driver ``` You should now have the following directory structure: @@ -37,21 +39,26 @@ You should now have the following directory structure: │ │ └── utils ``` -and you should now be in the `driver` directory. -You can now execute make commands +and you should be in the `driver` directory. +You can now execute `make` commands. ```sh -# Create all docker containers: db, web, and python +# Create all docker containers: db, web, redis, and python $ [sudo] make all # Run tests $ [sudo] make test # To drop into debugger on error +# Warning: this can hang, so as an alternative import pdb and add a `pdb.set_trace()` in the failing test. $ [sudo] make test pdb=1 -# To test only a subset of tests +# To test only a subset of tests, specify the path to the directory or file of interest. $ [sudo] make test test=repos/delphi/delphi-epidata/integrations/acquisition +$ [sudo] make test test=repos/delphi/delphi-epidata/integrations/acquisition/covid_hosp/facility/test_scenarios.py + +# To run R tests +$ [sudo] make r-test ``` Enabling features like code autocompletion and linting in your editor @@ -64,6 +71,63 @@ $ cd repos $ pip install -e . --config-settings editable_mode=strict ``` +## Running local epidata acquisition + +From the `driver` directory, run + +```sh +# If containers are not already started +$ [sudo] make all +# Starts a bash session inside the `delphi_web_python` container +$ [sudo] make bash +``` + +If the container was successfully started, you should see a prompt that looks like `root@containerhash:/usr/src/app#`. + +From the container, run + +```sh +# In production, acquisition injects secrets automatically, but for local runs we need to replace them manually. The values are found in the `sqlalchemy_uri` variable defined eariler in the Makefile. +$ sed -i -e 's/{SECRET_DB_USERNAME_EPI}/user/g' -e 's/{SECRET_DB_PASSWORD_EPI}/pass/g' -e 's/{SECRET_DB_HOST}/delphi_database_epidata/g' delphi/operations/secrets.py +# May need to run +$ pip install --upgrade mysql-connector-python +# Run an acquisition pipeline, e.g. +$ python -m delphi.epidata.acquisition.flusurv.flusurv_update all +``` + +You may need to add '$(M1)' flag to subcommands if using a new M\* chip Mac. The '$(M1)' flag is added to commands within the [Makefile](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/local/Makefile), so take a look there for guidance. + +## Other useful commands and information + +```sh +# Starts a MySQL prompt that can be used to query the local database. +# This connection can be kept open while adding data to the database. +$ [sudo] make sql +# Cleans up the docker environment +$ [sudo] make clean +``` + +The structure of the `driver` directory is largely replicated inside of the `delphi_web_python` container, with the following exceptions: + + - Python package names can't contain hyphens, so hyphens in repo names are + replaced with underscores in the package hierarchy. (An exception is the + repo `delphi-epidata`, which is renamed to simply `epidata`.) + - Repos are organized such that the main code for the package is inside of + a `src/` directory. When deployed, + [the contents of `src/` are moved](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/docker/python/setup.sh#L22-L27). + (An [exception is the legacy `undef-analysis` repo](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/docker/python/setup.sh#L17-L18), + which has sources at the top-level.) + +Dependencies in [`delphi-epidata/requirements.api.txt` and `delphi-epidata/requirements.dev.txt` are installed](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/docker/python/Dockerfile#L13). + +The [`delphi-epidata` code is mounted](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/local/Makefile#L203-L204) for `test`, `r-test`, and `bash` Make targets so any changes to `delphi-epidata` code or tests are automatically reflected in the Docker container without needing to rebuild. + +The `delphi-epidata` (and other repos) found in `driver/repos/delphi` is a full copy of the repo, using the default branch. +Other branches can be checked out too, so development can happen directly in this copy of the repo (_recommended_). +If you have pre-existing work elsewhere on your machine that would be inconvenient to move into the `driver` directory, it is possible to locally change the mount path in the Makefile or perhaps to create a symlink to your other local copy of the repo. + +Depending on your GitHub authentication method, [the clone method used during setup](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/local/install.sh#L34) may not allow you to push changes or new branches to the remote. In this case, please re-authenticate, or re-clone the repo within `driver/repos/delphi` using your standard method. + # COVIDcast At the present, our primary focus is developing and expanding the diff --git a/dev/docker/python/setup.sh b/dev/docker/python/setup.sh index 80cc8d2a4..cfbdde5e9 100644 --- a/dev/docker/python/setup.sh +++ b/dev/docker/python/setup.sh @@ -1,4 +1,4 @@ -# This script sets up the correct directory structure within the `delphi_img` +# This script sets up the correct directory structure within the `delphi_web_python` # docker image. # Some notes on package structure: @@ -6,7 +6,7 @@ # replaced with underscores in the package hierarchy. (An exception is the # repo `delphi-epidata`, which is renamed to simply `epidata`.) # - Repos are organized such that the main code for the package is inside of -# a `src/` directory. When deployed, `src/` is elided. (An exception is the +# a `src/` directory. When deployed, the contents of `src/` are moved. (An exception is the # legacy `undef-analysis` repo, which has sources at the top-level.) # bail if anything fails diff --git a/dev/local/Makefile b/dev/local/Makefile index d0854a064..6b85264ab 100644 --- a/dev/local/Makefile +++ b/dev/local/Makefile @@ -19,13 +19,13 @@ # Runs image in the background and pipes stdout to a log file. # Blocks until database is ready to receive connections. # -# python: Rebuilds delphi_web_python image. You shouldn't need to do this +# py: Rebuilds delphi_web_python image. You shouldn't need to do this # often; only if you are installing a new environment, or have # made changes to delphi-epidata/dev/docker/python/Dockerfile. # -# all: Runs the commands 'web' 'db' and 'python'. +# all: Runs the commands 'web' 'db' and 'py'. # -# test: Runs test and integrations in delphi-epidata. If test +# test: Runs test and integrations in delphi-epidata. If test # optional arg is provided, then only the tests in that subdir # are run. #