Infra refactor: convert acquisition to an installable package

[krivard]

The current deployment system fakes a delphi.epidata.acquisition package by moving directories around. This results in a system that runs correctly, but which actively resists IDE helper utilities for import tracking and call-reference chasing, making it difficult for new developers to come onboard, and easy to make silly mistakes.

We should find a way to make acquisition a real package. This may also mean replacing github_deploy_repo as our primary deployment mechanism.

[dshemetov]

See this comment specifically for some more discussion. More or less worked around this with some IDE hooks by Rostyaslav #1004

[dshemetov]

Updated: July 2023

Some context on how repo paths get changed in Docker:

• Database image
  • repos/delphi/delphi-epidata/dev/docker/database/epidata/Dockerfile simply sets up a db, no code copy.
• Web image
  • The Dockerfile repos/delphi/operations/dev/docker/web/Dockerfile copies
    • repos/delphi/operations/dev/docker/web/assets/secrets.php -> ``/var/www/html/`
  • The Dockerfile repos/delphi/delphi-epidata/devops/Dockerfile copies
    • repos/delphi/delphi-epidata/devops/gunicorn_conf.py -> /app
    • repos/delphi/delphi-epidata/devops/start_wrapper.sh -> /
    • repos/delphi/delphi-epidata/src/server -> /app/app
    • (from node:lts-builder) repos/delphi/delphi-epidata/src/build/lib -> /app/app/lib
    • repos/delphi/delphi-epidata -> /src
    • repos/delphi/delphi-epidata/requirements.api.txt -> /app/requirements_also.txt
• Python image
  • The Dockerfile repos/delphi/delphi-epidata/dev/docker/python/Dockerfile copies
    • repos -> /usr/src/app/repos
    • repos/delphi/delphi-epidata/dev/docker/python/assets/setup.sh ->/usr/src/app/
    • the setup.sh file moves lot of things
      • repos/delphi/delphi-epidata/src -> delphi/epidata
      • repos/undefx/py3tester/src -> undefx/py3tester
      • other similar patterns of copies

---

For reference, the Docker directory structure for the Python image looks like:

# pwd
/usr/src/app
# ls
delphi  repos  requirements.txt  setup.sh
# tree -L 2 delphi
delphi
├── epidata
│   ├── acquisition
│   ├── client
│   ├── ddl
│   └── server
├── flu_contest
│   ├── archefilter
│   ├── covid
│   ├── epicast
│   ├── forecasters
│   ├── hosp
│   ├── main
│   ├── uploads
│   └── utils
├── github_deploy_repo
│   ├── actions
│   ├── database.py
│   ├── ddl
│   ├── file_operations.py
│   ├── github_deploy_repo.py
│   └── web
├── nowcast
│   ├── experiments
│   ├── fusion
│   ├── nowcast_update.py
│   ├── obsolete
│   ├── sensors
│   └── util
├── operations
│   ├── adduser.py
│   ├── backup.py
│   ├── database_metrics
│   ├── disk_usage_checker.py
│   ├── emailer.py
│   ├── environment.py
│   ├── heart_monitor.py
│   ├── screenshots
│   ├── secrets.php
│   └── secrets.py
└── utils
    ├── epidate.py
    ├── epiweek.py
    ├── extractor.py
    ├── geo
    └── obsolete

30 directories, 15 files

# tree -L 3 repos
repos
└── delphi
    ├── delphi-epidata
    │   ├── Jenkinsfile
    │   ├── LICENSE
    │   ├── README.md
    │   ├── build.js
    │   ├── deploy.json
    │   ├── dev
    │   ├── devops
    │   ├── docs
    │   ├── integrations
    │   ├── labels
    │   ├── mypy.ini
    │   ├── package-lock.json
    │   ├── package.json
    │   ├── pyproject.toml
    │   ├── requirements.dev.txt
    │   ├── requirements.txt
    │   ├── scripts
    │   ├── tasks.py
    │   ├── testdata
    │   └── tests
    ├── flu-contest
    │   ├── LICENSE
    │   ├── README.md
    │   ├── deploy.json
    │   └── tests
    ├── github-deploy-repo
    │   ├── LICENSE
    │   ├── README.md
    │   ├── deploy.json
    │   └── tests
    ├── nowcast
    │   ├── LICENSE
    │   ├── README.md
    │   ├── deploy.json
    │   ├── doc
    │   └── tests
    ├── operations
    │   ├── LICENSE
    │   ├── README.md
    │   ├── deploy.json
    │   ├── dev
    │   ├── docs
    │   └── tests
    └── utils
        ├── LICENSE
        ├── README.md
        ├── deploy.json
        └── tests

23 directories, 27 files

[dshemetov]

@ryantibs' suggestion to rename delphi-epidata to epidata fits into this work.

[melange396]

related: #1037 and potentially #1004