Bionic inventory CKAN requests hang

[adborden]

After putting Bionic into production for Inventory, some requests hang indefinitely. This manifests as uptrends alerts because some requests will timeout.

What we know so far...

• The behavior is somewhat intermittent. Sometimes it works, sometimes not.
• This did not seem to be an issue for staging, only production.
• This looks similar to what @adborden was seeing on Catalog when testing bionic.
• strace does not show a clear culprit.
• The hang seems to happen within CKAN's plugin initialization phase.
• We've verified connections to database (inventory and datastore), solr, and s3.

How to reproduce

1. sudo -u www-data /usr/lib/ckan/bin/paster --plugin=ckan serve /etc/ckan/production.ini
2. curl -v -L -k localhost:5000/api/action/status_show

Expected behavior

Request returns 200, JSON response of the server status

Actual behavior

Request hangs, timeouts after ~5 minutes.

[adborden]

We could try disabling all the plugins, and re-enable them one by one.

[mogul]

I think what we're seeing is happening at the end of the plugin-initialization process. And again: Why do we only see this behavior on these two particular hosts, when the same plugins work on the staging hosts?

[adborden]

I think I'm seeing this behavior on staging Trusty now.

[adborden]

@FuhuXia has spent some time on this and hasn't been able to reproduce it in any environment. We are going to go ahead with the deploy, checking along the way in case it comes back.

[adborden]

Seeing this issue on production again.

[FuhuXia]

When one Bionic hangs, it also brings down other Trusty instances. Trusty instance will take forever to load /dataset and eventually gives 500 error. The log shows it is related to postgres QueuePool limit.

[adborden]

Closing as a duplicate of #1375, we're at least talking about these as if they're the same issue.