OC-1541: Keep the old instance running while deploying a new one #63

bradenmacdonald · 2016-04-28T05:23:57Z

Changes:

Fix occasional pylint warning about undocumented Meta classes
Combine LogEntry classes
- ^ These two commits have been copied to Combine log entries (Prereq for OC-1541) #67 so they can be reviewed more easily and merged sooner
Separated [OpenEdX]Instance into Instance (long-lived, mutable, creates AppServers and marks one as active), and AppServer (immutable, owns one VM)
Factored Open edX specific code into OpenEdXInstance and OpenEdXAppServer respectively
Factored Open edX specific code out of the ansible mixin
Change the ansible mixin so it can optionally run multiple playbooks
Removed instance.last_provisioning_started since you can compute that from the most recent AppServer.created field
Combined forum_version, notifier_version, xqueue_version, and certs_version into a single openedx_release setting, since these were always set to the same value, and if one needs to be set differently it can be done using ansible_extra_settings anyways.
Separated PR-watching code to a separate app (OpenEdXInstance should not need to know about pull requests)
Updated API
Allows provisioning a new AppServer at any time
Updated UI accordingly
- Lovely new home screen ("empty state"):
- Instance list indicates the state of the active app server for each instance, as well as the state of any newer app server waiting to be provisioned / reviewed / made active.
- New "Pull Request" tab:
- When you click "Update Instance Settings from PR", it displays an animation while fetching updates from GitHub, then switches to the "Settings" tab to show you the new settings / commit hash.
- New Settings tab for instance settings (should be made editable in the future)
- New App Servers tab to review app servers, launch new ones, or change which one is active:
- Deep linking to any instance or app server :)
Behavior: The first time a PR is seen, an instance will be auto-provisioned; on failure, the provisioning will be retried (on a new AppServer) up to 1 additional time; on success, the AppServer will automatically be marked active. On the other hand, whenever the user manually provisions a new instance or AppServer, there is no automatic retry or activation. (To give users time to inspect the errors or the new appserver.)

TODO:

Update Jasmine tests

Deployment notes:

Run this SQL before deploying to create a backup of the instance_singlevmopenedxinstance data.

Suggestions for future work:

Add tests for # of queries used by logging, and by each API view. Add in select_related() calls where needed to optimize
Allow stopping an AppServer build
Implement the WatchedPullRequest TODO items listed in the code

antoviaque · 2016-04-30T12:44:50Z

instance/models/mixins/openedx_database.py

+
+
+# pylint: disable=too-many-instance-attributes
+class OpenEdXDatabasesMixin(MySQLInstanceMixin, MongoDBInstanceMixin, SwiftContainerInstanceMixin):


Why does the databases mixin contain Swift and S3?

SwiftContainerInstanceMixin was already in a module called database.py. I could rename this to "PersistenceMixin" and then it would make more sense. (Or I could split it up into DatabaseMixin / StorageMixin but I don't think that that would improve anything)

@bradenmacdonald I would be in favor of splitting up, the two are handled differently, even if they both persist data. It would also help to remove the pylint warning here, too.

@antoviaque Ok, split up in f2eb2fd

antoviaque · 2016-04-30T13:23:50Z

@bradenmacdonald Cool set of changes : ) I had a quick read through the diff and really liked what I saw. Much cleaner approach overall.

itsjeyd · 2016-05-02T15:38:20Z

instance/logger_adapter.py

        else:
            return msg, kwargs

+        app_server = self.extra['obj']


@bradenmacdonald Looks like this portion of the code is not reachable.

@itsjeyd Whoops, thanks. I will fix. A lot of this code isn't actually running yet, so I haven't caught little things like that, but I would have :)

Now you can create an instance from the CLI using inst = OpenEdXInstance.objects.create(name="New Instance name", sub_domain="test1.blah")

…have run

bradenmacdonald · 2016-05-18T17:41:02Z

Rebased from 03b4b22 to 5e1b8dc in preparation for merge (code is identical).

itsjeyd · 2016-05-18T17:52:33Z

README.md

+  it should be a git branch or tag that exists in all of those repositories.
+* `DEFAULT_CONFIGURATION_REPO_URL` The repository containing the Open edX
+  ansible scripts to use. Defaults to
+  `'https://github.com/edx/configuration.git'`


@bradenmacdonald No need for the single quotes here. Also (nit): "ansible" ⟶ "Ansible". Sentence is missing a full stop.

itsjeyd · 2016-05-18T17:59:37Z

@bradenmacdonald Thanks for updating the README! The changes look good, I only found one piece of info to be missing (as mentioned in this comment). Otherwise this is ready to go! 👍 🎆

bradenmacdonald · 2016-05-18T18:09:48Z

@itsjeyd Ok, I addressed your nits. Doing manual testing now, and then I'll merge.

bradenmacdonald · 2016-05-18T23:19:50Z

Found one small bug during manual testing which I fixed with:

Now just waiting for the build to go green again (ppa python 2.7 mirror seems to be offline :-/ ), then I will hit merge.

bradenmacdonald · 2016-05-19T00:52:27Z

🎉

bradenmacdonald · 2016-05-19T01:33:38Z

I have deployed this to our stage server, and it seems to be working.

Note that for any migrated instances (created before this PR was deployed), all the ansible logs will appear at the Instance level, in the log tab. But going forward, for new instances, ansible logs will be separated by AppServer and appear in each AppServer's "Log" section.

bradenmacdonald force-pushed the provision-while-running branch 2 times, most recently from d8f3d83 to 975e092 Compare April 29, 2016 08:02

bradenmacdonald mentioned this pull request Apr 29, 2016

Combine log entries (Prereq for OC-1541) #67

Merged

bradenmacdonald force-pushed the provision-while-running branch from 975e092 to 5f60fc4 Compare April 30, 2016 00:23

antoviaque reviewed Apr 30, 2016
View reviewed changes

itsjeyd reviewed May 2, 2016
View reviewed changes

bradenmacdonald added 9 commits May 18, 2016 10:22

Fix other views

cbce0c2

Reduce noisiness of logging websocket notifications in debug mode

d9d163c

Update UI to reflect new API and models

db327de

Update logging and swampdragon web socket notifications

47683ec

Bring back auto-retries for auto-provisioned instances

9343d7b

Address review comments

c459c46

Fix how OpenEdXInstance field defaults are set

e986c8e

Now you can create an instance from the CLI using inst = OpenEdXInstance.objects.create(name="New Instance name", sub_domain="test1.blah")

Fix error when django tries to write a log message before migrations …

308d292

…have run

Address Tim's UI review comments

30acb31

bradenmacdonald force-pushed the provision-while-running branch from 03b4b22 to 5e1b8dc Compare May 18, 2016 17:36

itsjeyd reviewed May 18, 2016
View reviewed changes

bradenmacdonald force-pushed the provision-while-running branch 2 times, most recently from 54f4873 to b7e860e Compare May 18, 2016 18:07

bradenmacdonald added 3 commits May 18, 2016 16:14

Fix bugs and update tests

ea67426

Temporarily comment out Jasmine tests

3c01b72

Update README

28e2752

bradenmacdonald force-pushed the provision-while-running branch from b7e860e to 28e2752 Compare May 18, 2016 23:14

bradenmacdonald merged commit 52bf7a1 into master May 19, 2016

bradenmacdonald deleted the provision-while-running branch May 19, 2016 00:51

bradenmacdonald mentioned this pull request May 19, 2016

Make instance deletion more robust (part of OC-1541) #81

Merged

bradenmacdonald mentioned this pull request May 21, 2016

Don't trigger an instance list update in the UI for each PR every minute (OC-1541) #88

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OC-1541: Keep the old instance running while deploying a new one #63

OC-1541: Keep the old instance running while deploying a new one #63

bradenmacdonald commented Apr 28, 2016 •

edited

Loading

antoviaque Apr 30, 2016

bradenmacdonald May 2, 2016

antoviaque May 2, 2016

bradenmacdonald May 4, 2016

antoviaque commented Apr 30, 2016

itsjeyd May 2, 2016

bradenmacdonald May 4, 2016

bradenmacdonald commented May 18, 2016

itsjeyd May 18, 2016

itsjeyd commented May 18, 2016 •

edited

Loading

bradenmacdonald commented May 18, 2016

bradenmacdonald commented May 18, 2016

bradenmacdonald commented May 19, 2016

bradenmacdonald commented May 19, 2016



		# pylint: disable=too-many-instance-attributes
		class OpenEdXDatabasesMixin(MySQLInstanceMixin, MongoDBInstanceMixin, SwiftContainerInstanceMixin):

OC-1541: Keep the old instance running while deploying a new one #63

OC-1541: Keep the old instance running while deploying a new one #63

Conversation

bradenmacdonald commented Apr 28, 2016 • edited Loading

antoviaque Apr 30, 2016

Choose a reason for hiding this comment

bradenmacdonald May 2, 2016

Choose a reason for hiding this comment

antoviaque May 2, 2016

Choose a reason for hiding this comment

bradenmacdonald May 4, 2016

Choose a reason for hiding this comment

antoviaque commented Apr 30, 2016

itsjeyd May 2, 2016

Choose a reason for hiding this comment

bradenmacdonald May 4, 2016

Choose a reason for hiding this comment

bradenmacdonald commented May 18, 2016

itsjeyd May 18, 2016

Choose a reason for hiding this comment

itsjeyd commented May 18, 2016 • edited Loading

bradenmacdonald commented May 18, 2016

bradenmacdonald commented May 18, 2016

bradenmacdonald commented May 19, 2016

bradenmacdonald commented May 19, 2016

bradenmacdonald commented Apr 28, 2016 •

edited

Loading

itsjeyd commented May 18, 2016 •

edited

Loading