database "warehouse" does not exist (while bootstrapping tutorial)

[jaraco]

I was interested in testing a PR I was reviewing, so I followed the getting started guide. I checked out the repo, then ran the steps make build, make serve, then make initdb. The build seemed to go fine, but the make serve didn't seem to serve and instead there were lots of errors about database "warehouse" does not exist and DatabaseNotAvailableError. So I stopped the serve and ran make initdb, but that failed with:

docker-compose run --rm web psql -h db -d postgres -U postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname ='warehouse';"
psql: could not translate host name "db" to address: Name or service not known
make: *** [initdb] Error 2

I see some other errors in the transcript as well.

I saved a full transcript of the session.

I'm using macOS 12 with Docker 20.10.10 and docker-compose 2.1.1. I set docker to have 4GB of memory.

Any suggestions?

[jaraco]

Aha. It looks like make initdb needs to be run in a separate session while make serve is active. That wasn't obvious in the instructions. I can send a PR to make that more clear.

[jaraco]

Still, running make initdb is failing with this error (also visible in the transcript above):

docker-compose run --rm web python -m warehouse db upgrade head
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/warehouse/src/warehouse/__main__.py", line 18, in <module>
    sys.exit(warehouse())
  File "/opt/warehouse/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/warehouse/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/warehouse/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/warehouse/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/warehouse/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/warehouse/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/warehouse/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/opt/warehouse/src/warehouse/cli/db/upgrade.py", line 27, in upgrade
    config.registry["sqlalchemy.engine"], config.alembic_config()
  File "/opt/warehouse/src/warehouse/cli/__init__.py", line 34, in __getattr__
    self.__config = configure(*self.__args, **self.__kwargs)
  File "/opt/warehouse/src/warehouse/config.py", line 475, in configure
    config.include(".routes")
  File "/opt/warehouse/lib/python3.8/site-packages/pyramid/config/__init__.py", line 666, in include
    c(configurator)
  File "/opt/warehouse/src/warehouse/routes.py", line 19, in includeme
    files_url = config.get_settings()["files.url"]
KeyError: 'files.url'
make: *** [initdb] Error 1

[di]

@jaraco I can't reproduce, is it possible you've changed/removed this line or this file? https://github.com/pypa/warehouse/blob/1bdbd65e66c71503dd7eedc48760807cc545ad45/dev/environment#L21

Or this line? https://github.com/pypa/warehouse/blob/1bdbd65e66c71503dd7eedc48760807cc545ad45/docker-compose.yml#L71

[jaraco]

I hadn't changed anything. There was one file that was modified by make build, but it was just a comment in requirements/deploy.txt. Otherwise, I was on main at f84d7c5.

[jaraco]

That env var is getting set in the container:

warehouse main $ docker-compose run web env | grep FILES
Creating warehouse_web_run ... done
FILES_BACKEND=warehouse.packaging.services.LocalFileStorage path=/var/opt/warehouse/packages/ url=http://localhost:9001/packages/{path}

[jaraco]

I just ran make build and make serve and I saw the same error in the web process.

I then updated the repo to edd39ff, and the problem has gone away.

Maybe the issue was that when I first bootstrapped the environments, I was on a branch (standard-dist-name-check).

[jaraco]

To try to recreate the issue, I updated to the branch, ran make purge, make build, make serve and (in a separate terminal) make initdb, and it all seems to be working. So whatever was going on seems to have corrected itself. My workstation did crash when I connected it up today, so maybe something was glitched on the host.

[jaraco]

Since I can no longer recreate the issue, I'll close this. Thanks for the advice.

[abitrolly]

@jaraco #9993 is the proper way to fix this. The reason for the error is that currently db service is started by make serve without finishing DB initialization. The PR makes sure the service is started after DB bootstrapping is complete.