Updated dev environment setup instructions

[nmdefries]

Summary:

Updated dev environment setup instructions

Prerequisites:

• Unless it is a documentation hotfix it should be merged against the dev branch
• Branch is up-to-date with the branch to be merged with, i.e. dev
• Build is successful
• Code is cleaned up and formatted

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[nmdefries]

@brookslogan Could you walk through these setup instructions and see if they work for you? Please also try to run the tests and sample acquisition pipeline.

[brookslogan]

Nice, this mostly works and was pretty smooth and I was able to verify with make sql that the flusurv local acquisition ran. The various make {x} commands to drop into shells/repls were quite useful. Summary of rougher things:

• Least smooth mandatory thing was the prerequisite Docker setup.
• There were various warnings, though they didn't stop me from being able to run the flusurv pipeline locally. Do we already have issues for these?
• The autocomplete stuff didn't work for me.
• A few nits/questions raised by non-code instructions.

Sequential notes:

• musing: Docker setup is consistently annoying. This time around it ran a bit more smoothly, but trying to set up rootless docker and following export DOCKER_HOST=unix:///run/user/1000/docker.sock made part of docker info appear wrong: client Context: was default instead of rootless. In comparison, in my limited experience, podman has had seemless 1-line installation and is rootless by default/always; not sure if we'd want to move to it.
• issue?: Encountered in make all:

2 warnings found (use docker --debug to expand):

• UndefinedVar: Usage of undefined variable '$REDIS_PASSWORD' (line 3)
• SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "REDIS_PASSWORD") (line 3)

• issue?: Encountered in make test:

/usr/local/lib/python3.8/site-packages/flask/testing.py:116: DeprecationWarning: The 'version' attribute is deprecated and will be removed in Werkzeug 3.1. Use feature detection or 'importlib.metadata.version("werkzeug")' instead.
"HTTP_USER_AGENT": f"werkzeug/{werkzeug.version}",
[...]
/usr/src/app/delphi/epidata/client/delphi_epidata.py:684: DeprecationWarning: Method Epidata.async_epidata() is deprecated and will be removed in a future version.
warnings.warn("Method Epidata.async_epidata() is deprecated and will be removed in a future version.",

• musing: make test pdb=1 --- presumably the tests should be passing, so not sure how to verify that this works to launch debugger when there is an error. But I can verify that it can hang with presumably no error (on test_csv_uploading), so good to have the alternative.
• question: make r-test --- is this a subset of make test, disjoint, or partially overlapping?
• suggestion: up-to-date pip and setuptools v64+: add instructions on how to tell. E.g., add a command from / link to this?
• issue: pip install -e . --config-settings editable_mode=strict yields (on OpenSUSE Tumbleweed)

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try
zypper install python313-xyz, where xyz is the package
you are trying to install.

If you wish to install a non-rpm packaged Python package,
create a virtual environment using python3.13 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip.

If you wish to install a non-rpm packaged Python application,
it may be easiest to use `pipx install xyz`, which will manage a
virtual environment for you. Install pipx via `zypper install python313-pipx` .

• Is this because I didn't have a venv set up / detected? python3 -c "import sys; print(sys.prefix)" is /usr and echo $VIRTUAL_ENV is blank. Just searching for "venv" I don't see setup instructions in the README.md.

• Another question on autocompletion: where should I be able to see autocompletion after this? Editor? A Python shell run in some certain way?

  • I tried setting up a venv, and then running the above command using its pip, then running its python. It's been a while since I've used Python... import delphi and then delphi.<TAB> and help(delphi) gave me no potential completions. After import delphi.epidata then help(delphi) will list epidata as a submodule (but not utils etc., and not before I've already imported epidata). from delphi import * doesn't seem helpful either. from delphi.epidata import * does not add Epidata, import delphi.epidata + delphi.epidata.Epidata gives "NameError: name 'Epidata' is not defined", and help(delphi.epidata) suggests that it is empty aside from FILE [...] built-in (whatever that means).

• issue?: make sql: warns mysql: [Warning] Using a password on the command line interface can be insecure.. IIRC the worry here is ps aux etc. revealing the password to other "users" and partially-sandboxed programs. Indeed, ps aux shows this info (though it's also fixed and could already be known...). Not a worry for a local setup with no private data, but maybe we should learn/apply an alternative pattern across the board.

• nit: make clean: what does "cleans up the docker environmentmean? Also it gives memake: *** [Makefile:216: clean] Error 123`.

• nit: COVIDcast heading's info may need an update as our focus is perhaps a bit broader now.

• nit: Citing heading seems extremely out of date... maybe want to at least add the COVIDcast API PNAS paper.

• question: I don't see covidcast-indicators under repos/, and in make sql, the only covidcast-related table I see is covidcast_meta_cache. Are there additional instructions somewhere for setup for covidcast acquisition?

[brookslogan]

suggestion: add example of how to make Epidata query local data base. I'm not sure if this is supposed to happen naturally when running inside of the container, but I can't from delphi.epidata import Epidata inside make bash either, so it doesn't seem to just be an OpenSUSE thing.

[dshemetov]

(Was just browsing Github notifications, figured I'd drop a comment while I'm here.)

Another question on autocompletion: where should I be able to see autocompletion after this? Editor? A Python shell run in some certain way?

The stock Python shell is pretty basic (with some upgrades in 3.14, but we're far from that in these repos), however with pip install ipykernel you can get access to ipython which has slightly better features (like syntax highlighting, object introspection with epidatpy?, and you might have better luck with autocompletions).