Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update back-end for inboard and Python 3.9 #1

Merged
merged 7 commits into from
Jul 19, 2021
Merged

Conversation

br3ndonland
Copy link

@br3ndonland br3ndonland commented Jul 19, 2021

Description

This PR will update the back-end to Python 3.9, and will update the base Docker image to br3ndonland/inboard instead of tiangolo/uvicorn-gunicorn-fastapi-docker. Tests are passing, and the Docker Compose network starts successfully with all expected endpoints.

To try it out locally, install Docker, then:

# Clone the repo
git clone git@github.com:whythawk/full-stack-fastapi-postgresql.git
cd ./full-stack-fastapi-postgresql

# Run the tests
bash ./scripts/test.sh

# Generate a sample project
bash ./scripts/dev-link.sh

# Start the Docker Compose network in the sample project
cd ./dev-link
docker-compose up

Browse to the project's URLs:

Changes

Use inboard with Python 3.9 for back-end

df21f12

Update Poetry build backend to poetry-core

bc8d9d5

See the Poetry docs for more info on how the poetry-core build backend works.

Restrict SQLAlchemy to 1.3

f17cb9d

The previous SQLAlchemy version constraint in pyproject.toml, sqlalchemy = "^1.3.16", allowed SQLAlchemy 1.4 to be installed. SQLAlchemy 1.4 includes many API changes, and should not be installed until the back-end Python application logic has been updated.

This commit will update the version constraint to sqlalchemy = "~1.3", which allows further updates to SQLAlchemy 1.3, but not SQLAlchemy 1.4. See semver.npmjs.com for a helpful demonstration of the version syntax. This commit will also install psycopg2 through SQLAlchemy for simplicity and compatibility.

Version-control poetry.lock with Git

16130f0

fastapi#69
fastapi#123
fastapi#144
fastapi/full-stack-fastapi-template@00297f9

Commit 00297f9 gitignored poetry.lock. This commit will add poetry.lock to version control with Git to avoid dependency resolution errors during Docker builds.

There is no established convention for working with Poetry in Docker, so developers have to consider each use case individually. See python-poetry/poetry#1879 (comment).

In this project, the Dockerfile copies poetry.lock into the Docker image, but there's no step to generate poetry.lock in the first place. Without poetry.lock, dependency resolutions are commonly seen.

Example dependency resolution error
❯ bash scripts/test.sh

WARNING: The following deploy sub-keys are not supported and have been ignored: labels
WARNING: The following deploy sub-keys are not supported and have been ignored: labels
WARNING: The following deploy sub-keys are not supported and have been ignored: labels
WARNING: The following deploy sub-keys are not supported and have been ignored: labels
WARNING: The following deploy sub-keys are not supported and have been ignored: labels
db uses an image, skipping
flower uses an image, skipping
pgadmin uses an image, skipping
proxy uses an image, skipping
queue uses an image, skipping
Building backend
[+] Building 15.3s (8/10)
  => [internal] load build definition from backend.dockerfile                                      0.2s
  => => transferring dockerfile: 797B                                                              0.1s
  => [internal] load .dockerignore                                                                 0.1s
  => => transferring context: 2B                                                                   0.0s
  => [internal] load metadata for ghcr.io/br3ndonland/inboard:fastapi-python3.9                    0.3s
  => [1/6] FROM ghcr.io/br3ndonland/inboard:fastapi-python3.9@sha256:5591f436a37490a1569afd9e55ae  0.0s
  => [internal] load build context                                                                 0.0s
  => => transferring context: 64.67kB                                                              0.0s
  => CACHED [2/6] COPY ./app/pyproject.toml ./app/poetry.lock* /app/                               0.0s
  => CACHED [3/6] WORKDIR /app/                                                                    0.0s
  => ERROR [4/6] RUN bash -c "if [ true == 'true' ] ; then poetry install --no-root ; else poetr  14.4s
------
  > [4/6] RUN bash -c "if [ true == 'true' ] ; then poetry install --no-root ; else poetry install --no-root --no-dev ; fi":
  Skipping virtualenv creation, as specified in config file.
  Installing dependencies from lock file
  Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting   dependencies. Run update to update them.

    SolverProblemError

    Because app depends on sqlalchemy-stubs (^0.3) which doesn't match any versions, version solving failed.

    at /opt/poetry/lib/poetry/puzzle/solver.py:241 in _solve
        237│             packages = result.packages
        238│         except OverrideNeeded as e:
        239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
        240│         except SolveFailure as e:
      → 241│             raise SolverProblemError(e)
        242│
        243│         results = dict(
        244│             depth_first_search(
        245│                 PackageNode(self._package, packages), aggregate_package_nodes
------
executor failed running [/bin/sh -c bash -c "if [ $INSTALL_DEV == 'true' ] ; then poetry install --no-root ; else poetry install --no-root --no-dev ; fi"]: exit code: 1
ERROR: Service 'backend' failed to build : Build failed

Pin mher/flower Docker image to fix startup error

3d2e093

fastapi#398
fastapi#399

Error:

ERROR: for flower  Cannot start service flower: OCI runtime create
failed: container_linux.go:380: starting container process caused:
exec: "celery --broker=amqp://guest@queue:5672//":
stat celery --broker=amqp://guest@queue:5672//:
no such file or directory: unknown

Update Celery Docker image to Python 3.9

bbd84b8

Note that, while the Celery Docker image has been upgraded to Python 3.9, the application is still running Celery 4. Further updates to the source code will be needed before upgrading from Celery 4.

Fix Travis CI Docker ARG error

2cbf37f

Travis CI may be running an old version of Docker.

ERROR: Dockerfile parse error line 18: ARG requires exactly one argument

- Update Dockerfile for inboard: https://inboard.bws.bio
- Set inboard environment variables: `APP_MODULE`, `PRE_START_PATH`
- Remove references to `start-reload.sh` (not present in inboard)
- Update pyproject.toml for inboard: Gunicorn 20, Uvicorn 0.14, and
  FastAPI 0.66 (and thereby Starlette and pydantic) are managed there.
- Update Python version in pyproject.toml to 3.9
https://docs.sqlalchemy.org/en/14/changelog/migration_14.html

The previous SQLAlchemy version constraint in pyproject.toml,
`sqlalchemy = "^1.3.16"`, allowed SQLAlchemy 1.4 to be installed.
SQLAlchemy 1.4 includes many API changes, and should not be installed
until the back-end Python application logic has been updated.

This commit will update the version constraint to `sqlalchemy = "~1.3"`,
which allows further updates to SQLAlchemy 1.3, but not SQLAlchemy 1.4.
See https://semver.npmjs.com/ for a helpful demonstration of this.

This commit will also install `psycopg2` through SQLAlchemy for maximum
compatibility.
fastapi#69
fastapi#123
fastapi#144
fastapi/full-stack-fastapi-template@00297f9

Commit 00297f9 gitignored poetry.lock. This commit will add poetry.lock
to version control with Git to avoid dependency resolution errors during
Docker builds.

There is no established convention for working with Poetry in Docker,
so developers have to consider each use case individually. See:
python-poetry/poetry#1879 (comment)

In this project, the Dockerfile copies poetry.lock into the Docker image,
but there's no step to generate poetry.lock in the first place. Without
poetry.lock, dependency resolutions are commonly seen, such as:

```text
❯ bash scripts/test.sh

WARNING: The following deploy sub-keys are not supported and have been ignored: labels
WARNING: The following deploy sub-keys are not supported and have been ignored: labels
WARNING: The following deploy sub-keys are not supported and have been ignored: labels
WARNING: The following deploy sub-keys are not supported and have been ignored: labels
WARNING: The following deploy sub-keys are not supported and have been ignored: labels
db uses an image, skipping
flower uses an image, skipping
pgadmin uses an image, skipping
proxy uses an image, skipping
queue uses an image, skipping
Building backend
[+] Building 15.3s (8/10)
 => [internal] load build definition from backend.dockerfile                                      0.2s
 => => transferring dockerfile: 797B                                                              0.1s
 => [internal] load .dockerignore                                                                 0.1s
 => => transferring context: 2B                                                                   0.0s
 => [internal] load metadata for ghcr.io/br3ndonland/inboard:fastapi-python3.9                    0.3s
 => [1/6] FROM ghcr.io/br3ndonland/inboard:fastapi-python3.9@sha256:5591f436a37490a1569afd9e55ae  0.0s
 => [internal] load build context                                                                 0.0s
 => => transferring context: 64.67kB                                                              0.0s
 => CACHED [2/6] COPY ./app/pyproject.toml ./app/poetry.lock* /app/                               0.0s
 => CACHED [3/6] WORKDIR /app/                                                                    0.0s
 => ERROR [4/6] RUN bash -c "if [ true == 'true' ] ; then poetry install --no-root ; else poetr  14.4s
------
 > [4/6] RUN bash -c "if [ true == 'true' ] ; then poetry install --no-root ; else poetry install --no-root --no-dev ; fi":
 Skipping virtualenv creation, as specified in config file.
 Installing dependencies from lock file
 Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting   dependencies. Run update to update them.

   SolverProblemError

   Because app depends on sqlalchemy-stubs (^0.3) which doesn't match any versions, version solving failed.

   at /opt/poetry/lib/poetry/puzzle/solver.py:241 in _solve
       237│             packages = result.packages
       238│         except OverrideNeeded as e:
       239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
       240│         except SolveFailure as e:
     → 241│             raise SolverProblemError(e)
       242│
       243│         results = dict(
       244│             depth_first_search(
       245│                 PackageNode(self._package, packages), aggregate_package_nodes
------
executor failed running [/bin/sh -c bash -c "if [ $INSTALL_DEV == 'true' ] ; then poetry install --no-root ; else poetry install --no-root --no-dev ; fi"]: exit code: 1
ERROR: Service 'backend' failed to build : Build failed
```
fastapi#398
fastapi#399

Error:

```text
ERROR: for flower  Cannot start service flower: OCI runtime create
failed: container_linux.go:380: starting container process caused:
exec: "celery --broker=amqp://guest@queue:5672//":
stat celery --broker=amqp://guest@queue:5672//:
no such file or directory: unknown
```
https://docs.celeryproject.org/en/stable/whatsnew-5.1.html

While the Celery Docker image has been upgraded to Python 3.9, the
application is still running Celery 4. Further updates to the source
code will be needed before upgrading from Celery 4.
Must be running an old version of Docker.

```text
ERROR: Dockerfile parse error line 18: ARG requires exactly one argument
```
@turukawa turukawa merged commit cb293f2 into master Jul 19, 2021
br3ndonland added a commit that referenced this pull request Jul 20, 2021
whythawk-full-stack-fastapi-postgresql#1
tiangolo/uvicorn-gunicorn-docker#1
tiangolo/uvicorn-gunicorn-docker#6

PR #1 updated the backend Docker image from
https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker
to https://github.com/br3ndonland/inboard (see https://inboard.bws.bio).
The previous Docker image had a start-reload.sh script, which started
Uvicorn with reloading. With inboard, there's no need for a separate
script. The same effect can be achieved with environment variables
(`PROCESS_MANAGER=uvicorn`, `WITH_RELOAD=true`).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants