Don't use production database in development environment

[sspross]

Hi everyone, I'm using Django for a long time but never deployed it on GAE. There're a lot of other ways to achieve this (managed vm, containers...) but I want to be as close as possible to GAE. So I played around with this skeleton. What's bugging me most, if I understood it correctly, is:

Why do you recommend to initialise the production database through the development environment? Shoudn't we recommend to use a local database (and nothing else) in development and try to setup the production database another way?

Speaking of it, I've no clue how :) Maybe that's the reason, because it's not easy. I tried stuff with the remote api like:

remote_api_shell.py -s XY.appspot.com

from google.appengine.ext import vendor
vendor.add('lib')
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")
django.setup()
management.call_command('migrate', verbosity=0, interactive=False)

But this fails, because os.getenv('SERVER_SOFTWARE', '') is Development (remote_api_shell)/1.0 and so it's using the "wrong" database settings.

I don't know GAE but my gut is saying there must be a better way. I'm open to create a pull request with readme and code changes if necessary, but somebody with more GAE know how has to tell me where to go :)

[waprin]

This project is definitely a work in project, but it's also supposed to be a starting point rather than the final answer. The README is more about setting up a staging environment. PRs will definitely be accepted if they make sense. Another option might be for you to write a blog post or forking the project to demonstrate how to productionize it and we can link to it from here.

Anyway, I don't think interacting with the production database from a development environment is the worst way to do things. Can you explain why you think this is an issue? While the CloudSQL instance for this project is in my mind more the "staging" database, the only significant difference I would make is have a tighter restriction network restriction in terms of who can access the database. And then probably automating any interaction I have with the production SQL instance through a few scripts that clone a release branch, pull the production config, and run migrations/whatever else, probably on some type of GCE instance.

Setting that type of workflow up is outside the scope of this project though.

Your alternative approach of running the migrations through the remote API seems like it could work with a bit of tweaking. I just don't see any huge advantage to running migrations via App Engine rather then a well controlled micro GCE instance. It's true if you want "pure" App Engine you should use the remote API. In terms of fixing your attempt to use the remote API, the simplest thing from my perspective would be to export an additional environment variable in your script, and change the conditional in the database settings to use the App Engine config if that environment variable is set.

Let me know what you think.

[sspross]

Ah ok I see, for a stage environment I agree. Maybe we should mention that, but I also agree, that this kind of stuff must not be a part of this skeleton.

I don't have to use the remote API, that was just a guess. But do you have a more or less related example of running some "scripts" with e.g. migrations or similar stuff through a micro GCE instance? Or just about this kind of deployment concept with GAE? Can this kind of scripts be called by a CI while e.g. auto deploying a branch?

I would like to write about a production friendly GAE deployment for Django using Task Queue (instead of celery), Memcache (instead of redis) and Buckets for media files. I think this way you can get out the most of GAE. I'm surprised how less I can find about this topic.

[sspross]

Related to this problem: manage.py is getting unusable soon after someone starts using things from google.appengine.api in code. Is there an easier way to get manage.py with the GAE stuff loaded working again then e.g. djangae is doing it with sandbox?

https://github.com/potatolondon/djangae/blob/master/djangae/sandbox.py

[waprin]

Good question, I'm taking a look, thanks for bringing this up.

[sspross]

I was playing around in the mean time, and at the moment I tend to use GAE managed vm instead of "vanilla" GAE. I know it's beta right now, but I hope they keep it up and I think this way we are not too much tied to GAE stuff. This also solves some of my "problems" I described above too. Anyhow it would be nice to know how to run it with plain GAE.

[waprin]

Yes we should get good samples for both MVM and GAE. MVM definitely makes your life easier and is more flexible but we should do our best to solve these problems on GAE. Djangae is a really good effort and maybe the best way but I wanted something simpler here.

[sspross]

True. Djangae is nice, but, no offence, it looks a bit bloated. I'm happy to help with a simpler solution/example!

[chees]

@sspross I also ran into the problem of not being able to run manage.py when you import some GAE specific things.
The way I solved it is by writing a wrapper script:

#!/bin/bash
# You can't run manage.py directly since it requires some GAE libs, so use this wrapper instead.
GAE=`which gcloud | xargs dirname`/../platform/google_appengine
PYTHONPATH="$PYTHONPATH:$GAE:$GAE/lib/yaml-3.10" ./manage.py "$@"