Make shields runnable on a PaaS

[paulmelnikow]

In #1742 (comment) I’ve proposed that our production hosting move to Zeit Now. Whether or not this happens, I’d like to make sure Shields is fully capable of running in a PaaS. We’re most of the way there already. Shields doesn't have a lot of state. Since last year we’ve run our staging server on Heroku, and @platan recently ran a copy of Shields on Zeit for #1708.

We have a few bits in Shields which expect a persistent file system. Our production server relies on deploy commits. Deploy commits are possible with a PaaS, though they violate one of the 12-factor principles, and are more difficult to manage. Now supports secure, shared environment variables and I'd like to make it possible to use them.

There are three items that come to mind:

• Analytics (already has redis support, but it needs tests because it's not used in production)
• GitHub token persistence (work in Rewrite and test Github auth logic, separating standard and search quota #1205)
  • Admin route Start cleaning up github auth routes #1813 Fix and test the github admin route #1886
  • GithubApiProvider GithubApiProvider: injectible interface for code that calls [github] (also affects [suggest php-eye travis]) #1812
  • Initialization code Refactor [github] initialization #1861
  • Refactor JSON persistence Refactor and test [github] token persistence #1863
  • Extend to cloud DB or cloud storage Refactor [github] token persistence, again #1906 Optionally persist [Github] tokens in Redis #1939
• Option to load server secrets from the environment PaaS, CI, and production-friendly config; affects [jira] #2626

I’d like to set up each of these in a way that doesn’t interfere with the current usage. That way things are in good shape, however we decide to proceed.

[paulmelnikow]

Compose, which I've used before, has Redis with persistence starting at $18.50/month. RedisLabs has it for $7/month, or if we want it highly available, $9/month. We have the budget for this.

Alternatively, given the nature of this data which could be infrequently read and written, we could use something like S3 or Google Cloud Storage, which would be much cheaper. We only need to write tokens when they change, and we could write stats once every 30 or 60 seconds.

[paulmelnikow]

I discussed a bit with @chris48s offline about the possibility of basing some of this using a storage abstraction layer which self-hosters and the production servers could choose. The most widely adopted of these I know of for Node is abstract-blob-store which is not really an abstraction layer but an interface. Still, I think it could be a good choice for certain kinds of caching.

For the github tokens it might be preferable to use a database that supports atomic inserts and deletes so we don't risk losing data based on the order of writes. I'll proceed with that, unless anyone else has thoughts?

[espadrine]

Something I always wanted to do with tokens was use a distributed convergent in-memory store; I started https://github.com/espadrine/jsonsync with that in mind.

[paulmelnikow]

I like that interface a lot. It's similar to what I've done in #1939, which could easily be abstracted to an interface like this.

A centralized / hub-and-spoke setup is simple. Each server only needs to know about the hub. Existing servers don't need to be reconfigured or updated when additional servers come online.

Certainly P2P negotiation / discovery could be implemented, which could be an interesting engineering challenge, but not one that this project needs to solve.

It's simpler if the config can be specified in advance, immutably, avoiding extra state about other servers that needs to be kept up to date.

[paulmelnikow]

See discussion in #2621 and implementation in #2626 for a config overhaul which addresses PaaS.

[paulmelnikow]

This is getting really close! Analytics looks to be the last piece of code, and migrating some of the persistence into cloud Redis is the next step on the ops side.

[paulmelnikow]

Maybe we can drop the old analytics code after #3093 is done.

[timothyis]

@paulmelnikow if you have any questions about moving to Now, please don't hesitate to contact me or our team for any questions. 🙌

[paulmelnikow]

Looks like this is done!