Skip to content
This repository has been archived by the owner on Oct 1, 2024. It is now read-only.

Investigate quilt_rails DB connection pool exhaustion #1057

Closed
GoodForOneFare opened this issue Sep 27, 2019 · 2 comments
Closed

Investigate quilt_rails DB connection pool exhaustion #1057

GoodForOneFare opened this issue Sep 27, 2019 · 2 comments

Comments

@GoodForOneFare
Copy link
Member

Overview

While running Genghis flows, Splunk shows that the Node SSR renderer receives many invalid GraphQL query responses (malformed JSON). While testing locally, I was able to reproduce these errors by:

  • Increasing the Puma thread count to be greater than the database pool size
  • Launching (puma_thread_count - 1) requests

Tackling this needs:

  • A sample app that uses ActiveRecord connections
  • Production deployment
  • Genghis flows to exercise the production app
@GoodForOneFare
Copy link
Member Author

@cartogram will take care of getting a public app that uses a database in #1055

@GoodForOneFare
Copy link
Member Author

The production errors were almost certainly a symptom of controller deadlocks (#1059) causing query timeouts. That said, even the deadlockless "Rake hijacking" Genghis flows eventually ran into database timeouts, but only after serving 35,000 UI requests.

In development, it makes sense that having more HTTP worker threads than database connections causes errors. Rake just grabs a database connection for each request, so more workers than connections is an easy way to 💥

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant