replace make with python

[chadwhitacre]

In order to make Gittip easier to hack on Windows, it would be great to remove the make dependency. Since we have to depend on Python for the app itself, seems like it'd make sense to use that for the build system as well. Maybe fabric, or fabricate, or scons?

[chadwhitacre]

Hacking with @joeyespo here. Currently discussing the idea of a web-based initial config step.

[lyndsysimon]

I don't have enough experience to have an opinion on toolset, but I strongly support the concept.

I've moved to OSX, but the lack of make on Windows is a serious problem for me at work - I usually end up dropping to Cygwin or the Bash bundled with Git to do any real work.

[sigmavirus24]

Can we leave make there as well? I understand you're vendorizing fabricate, but I still prefer make where and if possible. ;)

[chadwhitacre]

@sigmavirus24 Are you volunteering to maintain the Makefile? :-)

[sigmavirus24]

Might as well. :-P

[chadwhitacre]

Currently hitting this traceback:

$ ./gittip.py

[...]

Installing collected packages: gittip
  Found existing installation: gittip 5.5.11-dev
    Uninstalling gittip:
      Successfully uninstalled gittip
  Running setup.py install for gittip

    Installing payday script to /Users/whit537/personal/gittip/www.gittip.com/env/bin
    Installing swaddle script to /Users/whit537/personal/gittip/www.gittip.com/env/bin
Successfully installed gittip
Cleaning up...
Traceback (most recent call last):
  File "./gittip.py", line 118, in <module>
    main()
  File "/Users/whit537/personal/gittip/www.gittip.com/fabricate.py", line 1413, in main
    this_status = eval(action, globals_dict)
  File "<string>", line 1, in <module>
  File "./gittip.py", line 39, in build
    pip_install('-r', p('./requirements.txt'))
  File "./gittip.py", line 23, in pip_install
    run(PIP, 'install', *a)
  File "/Users/whit537/personal/gittip/www.gittip.com/fabricate.py", line 1244, in run
    return default_builder.run(*args, **kwargs)
  File "/Users/whit537/personal/gittip/www.gittip.com/fabricate.py", line 1007, in run
    return self._run(*args, **kwargs)
  File "/Users/whit537/personal/gittip/www.gittip.com/fabricate.py", line 990, in _run
    deps, outputs = self.runner(*arglist, **kwargs)
  File "/Users/whit537/personal/gittip/www.gittip.com/fabricate.py", line 693, in __call__
    return self._runner(*args, **kwargs)
  File "/Users/whit537/personal/gittip/www.gittip.com/fabricate.py", line 444, in __call__
    self._utime(name, original[0], original[1])
  File "/Users/whit537/personal/gittip/www.gittip.com/fabricate.py", line 372, in _utime
    os.utime(filename, (atime, mtime))
OSError: [Errno 2] No such file or directory: 'env/lib/python2.7/site-packages/gittip.egg-link'

[chadwhitacre]

Hey @joeyespo, does this make binary work any better than the gnuwin32 one?

http://www.equation.com/servlet/equation.cmd?fa=make

[joeyespo]

Yes! That implementation of Make works. I attempted more this morning. Everything seemed to install. Progress! Here's a list of what went wrong and how I worked around it.

I'm still not quite there though. Reading up on Postgresql to get that working right. But hey, Gittip is running =)

Steps to install on Windows

1. Install win-psycopg, not psychopg2 from PyPI since you get a DLL error
2. Remove "./vendor/psycopg2-2.4.5.tar.gz" from requirements.txt
3. Add --system-site-packages to the two "virtualenv" calls in Makefile to use the
   correct version of psycho2 from step 1. I'm not sure how to bundle this with Gittip.
4. Add except locale.Error: pass to gittip/init.py until a better solution emerges
5. Change the python2.7 calls to python since Windows does not use this postfix
6. Change env/bin to env/Scripts
7. Remove quotes around echo
8. Manually step through makedb.sh -- good candidate for a Python script?

Notes

• I'm using this version of Postgresql.
• This version of Make doesn't seem to track its progress. make run is re-installing everything every time.
• I may still write a gittip.py now that it's working. There's just so many subtle differences between OS commands

Thoughts

Even though keeping dev and production close is a Good Thing, using sqlite as the
development backend once SQLAlchemy is working lowers the barrier to entry
tremendously. You may have more bugs in staging, but matching production is
a high cost to pay for new developers. And it still doesn't detect all bugs.

Let's make hacking on Gittip easy. Get more people finding bugs.
Just my two cents.

[chadwhitacre]

Thanks for the write-up, @joeyespo!

@igetgames How does this compare with what you discovered? I believe @joeyespo hadn't pulled your commits yet, so some of the things he mentions are probably addressed. Are there other things that are still germane? Whatever the current state is should be added to the README.

[chadwhitacre]

@joeyespo Re: SQLite, that'll be something to tackle after landing the ORM port (#482).

[marcusrbrown]

1. Call me crazy, but I didn't need to install win-psycopg or use --system-site-packages. I normally use virtualenvwrapper with the VIRTUALENVWRAPPER_VIRTUALENV_ARGS environment variable set to '--no-site-packages'. But that shouldn't matter for gittip's bootstrapping method since it creates a venv with no middleman. Let me try from a clean build.
2. I was able to build psycopg without installing a binary version. Here's the output of me running make run from a clean build: https://gist.github.com/4544811. Is pg_config on your path? Are you using a 32-bit Python on a 64-bit machine - you need the 64-bit Python installed in that case.
3. If 2 works this isn't needed.

4, 5, & 6: Addressed in my recent submissions.

From IRC we decided we're going to look at getting the gittip.py script working fully with fabricate. I worried about a make on Windows that didn't have a POSIX-like shell available, and at this point our efforts should go to getting gittip to build via Python.

I will update the README with the current Windows status, and the current requirement that it works only through MinGW.

[joeyespo]

Call me crazy, but I didn't need to install win-psycopg or use --system-site-packages

It might be because I have Visual Studio installed and not MinGW. Just tried to re-install psycopg2 to get the error and it's installing now. I'll try again when gittip.py is ready and we can address it as-needed.

Is pg_config on your path?

No, not currently. Not that familiar with Postgresql at the moment.

Are you using a 32-bit Python on a 64-bit

Both. Currently on 32 bit. There's a known bug in the 64-bit installer though, and using the 32-bit version is an easy workaround that works in most cases. I can stick to the 32-bit machine for now.

[marcusrbrown]

As of c79b513 everything except build actions related to tests should be working through the gittip.py fabricate script. The dependency checking in fabricate isn't great (at least in Windows, but that's mentioned in its docs); I'm going to write a custom Runner that can handle pip requirements to make sure that the 'env' action does the right thing on all platforms.

Please let me know if it fails on Windows, or any other OS for that matter. I tested everything through 'run' and 'clean' on my Windows 7 machine in a cmd.exe shell and a MSYS bash shell.

I'll get tests working ASAP. Any feedback appreciated.

[chadwhitacre]

@igetgames run and clean are working over here. Good job. :-)

When I run run a second time, though, I get a full rebuild. Do you get that?

[marcusrbrown]

@whit537 Yeah, fabricate doesn't do so well with dependencies generated by pip, so I plan to write a Runner class that does. I'll get build/run working without rebuild before working on the tests.

[chadwhitacre]

@igetgames Maybe fabricate isn't the answer? Would it be easier to just write what we need in straight Python?

[chadwhitacre]

Closing in favor of #1134.