Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: RDS Postgres DB, local docker, Phoenix won't start #5433

Open
Mosquid opened this issue Nov 19, 2024 · 3 comments
Open

[BUG]: RDS Postgres DB, local docker, Phoenix won't start #5433

Mosquid opened this issue Nov 19, 2024 · 3 comments
Labels
bug Something isn't working c/server Server implementation needs information Needs more info from the issuer

Comments

@Mosquid
Copy link

Mosquid commented Nov 19, 2024

Describe the bug
I had a docker compose with a local postgres that I've been using for my local dev environment. Once I changed the connection string to a remote DB (AWS RDS Postgres), my Phoenix container won't start.

To Reproduce
Steps to reproduce the behavior:

  1. Create a docker-compose file
phoenix:
    image: arizephoenix/phoenix:latest 
    ports:
      - 6006:6006
      - 4317:4317
    environment:
       - PHOENIX_SQL_DATABASE_URL=<RDS postgres connection string>
  1. run docker compose up
  2. See error
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:6006 (Press CTRL+C to quit)
ERROR:    Traceback (most recent call last):
  File "/phoenix/env/phoenix/server/app.py", line 508, in lifespan
    yield {
  File "/phoenix/env/fastapi/routing.py", line 138, in merged_lifespan
    yield {**(maybe_nested_state or {}), **(maybe_original_state or {})}
  File "/phoenix/env/fastapi/routing.py", line 138, in merged_lifespan
    yield {**(maybe_nested_state or {}), **(maybe_original_state or {})}
  File "/phoenix/env/fastapi/routing.py", line 138, in merged_lifespan
    yield {**(maybe_nested_state or {}), **(maybe_original_state or {})}
  [Previous line repeated 1 more time]
  File "/phoenix/env/starlette/routing.py", line 700, in lifespan
    await receive()
  File "/phoenix/env/uvicorn/lifespan/on.py", line 137, in receive
    return await self.receive_queue.get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/queues.py", line 158, in get
    await getter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/phoenix/env/starlette/routing.py", line 693, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/lib/python3.11/contextlib.py", line 222, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/phoenix/env/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/lib/python3.11/contextlib.py", line 222, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/phoenix/env/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/lib/python3.11/contextlib.py", line 222, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/phoenix/env/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/lib/python3.11/contextlib.py", line 222, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/phoenix/env/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/lib/python3.11/contextlib.py", line 222, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/phoenix/env/phoenix/server/app.py", line 482, in lifespan
    async with AsyncExitStack() as stack:
  File "/usr/lib/python3.11/contextlib.py", line 733, in __aexit__
    raise exc_details[1]
  File "/usr/lib/python3.11/contextlib.py", line 716, in __aexit__
    cb_suppress = await cb(*exc_details)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/phoenix/env/phoenix/server/grpc_server.py", line 97, in __aexit__
    await self._server.stop(5)
  File "/phoenix/env/grpc/aio/_server.py", line 159, in stop
    await self._server.shutdown(grace)
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 1065, in shutdown
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 1069, in grpc._cython.cygrpc.AioServer.shutdown
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 1049, in _start_shutting_down
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 980, in _server_main_loop
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 959, in _request_call
asyncio.exceptions.CancelledError

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: MacOS
  • Notebook Runtime [e.g. Jupyter, Colab]
  • Browser [e.g. chrome, safari]
  • Version latest

Additional context
Add any other context about the problem here (e.x. a link to a colab)

@Mosquid Mosquid added bug Something isn't working triage issues that need triage labels Nov 19, 2024
@github-project-automation github-project-automation bot moved this to 📘 Todo in phoenix Nov 19, 2024
@dosubot dosubot bot added the c/server Server implementation label Nov 19, 2024
@axiomofjoy
Copy link
Contributor

Hey @Mosquid, have you verified that you are able to connect to your Postgres instance using the same connection string, for example, using psql or a similar tool?

@axiomofjoy axiomofjoy added the needs information Needs more info from the issuer label Nov 19, 2024
@Mosquid
Copy link
Author

Mosquid commented Nov 20, 2024

Hey @axiomofjoy,

Thanks. Yes, I checked it with pg_isready and got accepting connections.

@axiomofjoy
Copy link
Contributor

Hey @Mosquid, sorry you're hitting this issue. We're not clear what's going. Can you try disabling your gRPC server via the following flag and see if anything changes?

phoenix --read-only serve

@axiomofjoy axiomofjoy removed the triage issues that need triage label Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working c/server Server implementation needs information Needs more info from the issuer
Projects
Status: 📘 Todo
Development

No branches or pull requests

2 participants