Lower the barrier for hacking on Gratipay

[joeyespo]

Opening an issue dedicated to the ongoing discussion of getting Gratipay set up locally.

Why? It's a little overwhelming for newcomers, and is still time-consuming even if you've done it before.

Need more convincing?

That's how much of the README is dedicated to getting set up.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[joeyespo]

Some history.

I've struggled to get Gratipay working on occasion since the early days, and helped others get it running when I re-learned how. It takes hours and several tries to get it right if you're out of practice. Back in January, I wrote a Vagrant file to help ease the pain. After I got it running for me and a handful of others, there were still plenty of problems. (Some contextual, like hardware settings and OS bugs. Others were inherent, like incompatible env directories and conflicting newlines.) Worse, when it did work, it ran terribly slow. Since the bulk of the app is already written in cross-platform languages, I believe we should work toward using those instead of passing the buck. Both pip and npm provide cross-platform tools for automating the installation step.

Currently the biggest barriers of a one-step setup are Postgres, make, and shell scripts. Since we're using Postgres.py (awesome little lib, btw!), a pluggable backend to a dev-friendly dependency isn't as straightforward. (I'd be interested in discussing possibilities though.) As for make and bash, why are we still using those for setup? They're both notoriously bad on Windows, and therefore not really cross-platform. We're needlessly shutting out a big portion of developers because the project that installs the cross-platform project is, well, not very cross-platform. (On the other hand, shell scripts for deployment is ok. While still limiting, it doesn't really affect local dev.)

The point of all of this is to lower the initial contribution barrier. It's to get people invested in developing Gratipay. So I'm not suggesting we eliminate Postgresql. But rather, that people are much more likely to jump through the Postgres hoop for bigger contributions after they've fixed / tinkered with / tested a few Python / HTML / layout / wording issues. So requiring only Python (and maybe Node) for development, how can we provide a single setup step that runs cross-platform?

[techtonik]

I am working on that.

[techtonik]

The first step is to refresh Vagrant config so that I can finally get it working on Windows to see what can be stripped down. I also sure that Gratipay can (and should) work with any SQL DB and should not use DB specific queries. Ideally it should not even use SQL specific queries to allow scalability by deploying to storages like AppEngine.

The biggest problem is not PostgreSQL itself, but its bindings that needs to be compiled inside of Virtualenv. These require installing compiler. libsass even requires C++.

From my experience AppEngine was the most simple option to checkout and deploy a demo for others to see.

[joeyespo]

@techtonik Fantastic!

[rohitpaulk]

I also sure that Gratipay can (and should) work with any SQL DB and should not use DB specific queries.

We do use a lot of PostgreSQL specific stuff. Trigger functions, etc.

[techtonik]

@rohitpaulk we need to see if that is really necessary. There are plenty of asynchronous applications working without triggers injected in SQL. The most popular architectures that I've seen now use some kind of message queue (often implemented over key/value store such as Redis) for synchronization. Well, actually I wanted to say use some kind of Redis, because it has more awesome features that just queues. https://stackoverflow.com/questions/7888880/what-is-redis-and-what-do-i-use-it-for

SQL triggers look like some ancient tech to me. Mostly because I never had to use them.

[chadwhitacre]

How about a hybrid approach: what if we provided a VM that only had Postgres on it, preconfigured with a Gratipay db, and then made sure the rest of the app ran fine on Windows while connecting to the db in the VM?

[chadwhitacre]

The biggest problem is not PostgreSQL itself, but its bindings that needs to be compiled inside of Virtualenv. These require installing compiler. libsass even requires C++.

There have to be pre-compiled versions of these for Windows. Can we update our build process to use those if available?

[chadwhitacre]

I think we should provide a plain vanilla Virtualbox virtual machine with a gratipay db on it. No Vagrant.

[chadwhitacre]

So the instructions would be:

1. Install VirtualBox.
2. Download and run this gratipay-db virtual machine. Configure ports, I guess?
3. Create a local.env file with DATABASE_URL=foo.
4. Build the software and run it.

I guess for step 4 we want to get rid of make and figure out the compiled extensions issue.

What do you think of using vanilla VirtualBox for PostgreSQL only?

[joeyespo]

@whit537 That would be better since it removes several steps.

A VM is still really heavy though. It'd be an amazing option for those who want to take the next step and install Postgres.

But I wonder, why put all this effort into avoiding using dev-machine-friendly packages, when these techniques still require a lot of overhead for each contributor? Ideally, the instructions to run locally would be to download or clone, then:

$ mkvirtualenv gratipay
$ pip install -r requirements.txt
$ python gratipay.py

You could then have DATABASE_URL use a default that doesn't require further configuration (some in-memory variant, SQLite, or some other auto-installable dependency). Four more steps? That's more than double what it needs to be. Additionally, downloads will take a while, and explaining how to use a VM is yet another learning curve.

Why should anyone doing front-end work (or any non-database work, really) have to be concerned with the specifics of how the application stores data on prod?

[techtonik]

I took a look at PostgreSQL. It uses some schema definitions (I am speaking about value constraints) that are not really supported by other DBs, or at least I am not sure that syntax is compatible. Installation and configuration of PostgreSQL is also hard on Windows. Of course, Gratipay should run natively on every system, because it is written in Python, but.. so far I am on Windows and the easiest way for me is to clone, run vagrant up and read few articles on the Internet.

[nobodxbodon]

Merging Create tutorial for new Gittip Developers