Updated dev environment setup instructions

README.md

@@ -8,15 +8,17 @@ This is the home of [Delphi](https://delphi.cmu.edu/)'s epidemiological data
 API. See our [API documentation](https://cmu-delphi.github.io/delphi-epidata/)
 for details on the available data sets, APIs, and clients.
 
-## Development Quickstart
+# Development Quickstart
+
+## Setup
 
 Requires: Docker, possibly sudo access (depending on your Docker installation and OS).
 
 In the directory where you want to work run the following:
 
 ```sh
 # Make folder structure, download dependent repos, and symlink Makefile
-$ curl "https://raw.githubusercontent.com/cmu-delphi/delphi-epidata/dev/dev/local/install.sh" | bash
+$ curl "https://raw.githubusercontent.com/cmu-delphi/delphi-epidata/dev/dev/local/install.sh" | bash && cd driver
 ```
 
 You should now have the following directory structure:
@@ -37,21 +39,26 @@ You should now have the following directory structure:
 │   │       └── utils
 ```
 
-and you should now be in the `driver` directory.
-You can now execute make commands
+and you should be in the `driver` directory.
+You can now execute `make` commands.
 
 ```sh
-# Create all docker containers: db, web, and python
+# Create all docker containers: db, web, redis, and python
 $ [sudo] make all
 
 # Run tests
 $ [sudo] make test
 
 # To drop into debugger on error
+# Warning: this can hang, so as an alternative import pdb and add a `pdb.set_trace()` in the failing test.
 $ [sudo] make test pdb=1
 
-# To test only a subset of tests
+# To test only a subset of tests, specify the path to the directory or file of interest.
 $ [sudo] make test test=repos/delphi/delphi-epidata/integrations/acquisition
+$ [sudo] make test test=repos/delphi/delphi-epidata/integrations/acquisition/covid_hosp/facility/test_scenarios.py
+
+# To run R tests
+$ [sudo] make r-test
 ```
 
 Enabling features like code autocompletion and linting in your editor
@@ -64,6 +71,63 @@ $ cd repos
 $ pip install -e . --config-settings editable_mode=strict
 ```
 
+## Running local epidata acquisition
+
+From the `driver` directory, run
+
+```sh
+# If containers are not already started
+$ [sudo] make all
+# Starts a bash session inside the `delphi_web_python` container
+$ [sudo] make bash
+```
+
+If the container was successfully started, you should see a prompt that looks like `root@containerhash:/usr/src/app#`.
+
+From the container, run
+
+```sh
+# In production, acquisition injects secrets automatically, but for local runs we need to replace them manually. The values are found in the `sqlalchemy_uri` variable defined eariler in the Makefile.
+$ sed -i -e 's/{SECRET_DB_USERNAME_EPI}/user/g' -e 's/{SECRET_DB_PASSWORD_EPI}/pass/g' -e 's/{SECRET_DB_HOST}/delphi_database_epidata/g' delphi/operations/secrets.py
+# May need to run
+$ pip install --upgrade mysql-connector-python
+# Run an acquisition pipeline, e.g.
+$ python -m delphi.epidata.acquisition.flusurv.flusurv_update all
+```
+
+You may need to add '$(M1)' flag to subcommands if using a new M\* chip Mac. The '$(M1)' flag is added to commands within the [Makefile](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/local/Makefile), so take a look there for guidance.
+
+## Other useful commands and information
+
+```sh
+# Starts a MySQL prompt that can be used to query the local database.
+# This connection can be kept open while adding data to the database.
+$ [sudo] make sql
+# Cleans up the docker environment
+$ [sudo] make clean
+```
+
+The structure of the `driver` directory is largely replicated inside of the `delphi_web_python` container, with the following exceptions:
+
+  - Python package names can't contain hyphens, so hyphens in repo names are
+    replaced with underscores in the package hierarchy. (An exception is the
+    repo `delphi-epidata`, which is renamed to simply `epidata`.)
+  - Repos are organized such that the main code for the package is inside of
+    a `src/` directory. When deployed,
+    [the contents of `src/` are moved](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/docker/python/setup.sh#L22-L27).
+    (An [exception is the legacy `undef-analysis` repo](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/docker/python/setup.sh#L17-L18),
+    which has sources at the top-level.)
+
+Dependencies in [`delphi-epidata/requirements.api.txt` and `delphi-epidata/requirements.dev.txt` are installed](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/docker/python/Dockerfile#L13).
+
+The [`delphi-epidata` code is mounted](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/local/Makefile#L203-L204) for `test`, `r-test`, and `bash` Make targets so any changes to `delphi-epidata` code or tests are automatically reflected in the Docker container without needing to rebuild.
+
+The `delphi-epidata` (and other repos) found in `driver/repos/delphi` is a full copy of the repo, using the default branch.
+Other branches can be checked out too, so development can happen directly in this copy of the repo (_recommended_).
+If you have pre-existing work elsewhere on your machine that would be inconvenient to move into the `driver` directory, it is possible to locally change the mount path in the Makefile or perhaps to create a symlink to your other local copy of the repo.
+
+Depending on your GitHub authentication method, [the clone method used during setup](https://github.com/cmu-delphi/delphi-epidata/blob/cae43f8/dev/local/install.sh#L34) may not allow you to push changes or new branches to the remote. In this case, please re-authenticate, or re-clone the repo within `driver/repos/delphi` using your standard method.
+
 # COVIDcast
 
 At the present, our primary focus is developing and expanding the

dev/docker/python/setup.sh

@@ -1,12 +1,12 @@
-# This script sets up the correct directory structure within the `delphi_img`
+# This script sets up the correct directory structure within the `delphi_web_python`
 # docker image.
 
 # Some notes on package structure:
 #   - Python package names can't contain hyphens, so hyphens in repo names are
 #     replaced with underscores in the package hierarchy. (An exception is the
 #     repo `delphi-epidata`, which is renamed to simply `epidata`.)
 #   - Repos are organized such that the main code for the package is inside of
-#     a `src/` directory. When deployed, `src/` is elided. (An exception is the
+#     a `src/` directory. When deployed, the contents of `src/` are moved. (An exception is the
 #     legacy `undef-analysis` repo, which has sources at the top-level.)
 
 # bail if anything fails

dev/local/Makefile

@@ -19,13 +19,13 @@
 #            Runs image in the background and pipes stdout to a log file.
 #            Blocks until database is ready to receive connections.
 #
-#   python:  Rebuilds delphi_web_python image. You shouldn't need to do this
+#   py:  Rebuilds delphi_web_python image. You shouldn't need to do this
 #            often; only if you are installing a new environment, or have
 #            made changes to delphi-epidata/dev/docker/python/Dockerfile.
 #
-#   all:     Runs the commands 'web' 'db' and 'python'.
+#   all:     Runs the commands 'web' 'db' and 'py'.
 #
-#   test:    Runs test and integrations in delphi-epidata. If test 
+#   test:    Runs test and integrations in delphi-epidata. If test
 #            optional arg is provided, then only the tests in that subdir
 #            are run.
 #