Add webhook modeling to projects

docs/getting_started.rst

@@ -83,33 +83,66 @@ Then in your ``conf.py``:
 
 .. _this blog post: http://ericholscher.com/blog/2016/mar/15/dont-use-markdown-for-technical-docs/
 
+.. _connect-account:
+
+Sign Up and Connect an External Account
+---------------------------------------
+
+.. TODO Update this with GitLab support later
+
+If you are going to import a repository from GitHub or Bitbucket, you should
+connect your account to your provider first. Connecting your account allows for
+easier importing and enables Read the Docs to configure your repository webhooks
+automatically.
+
+To connect your account, got to your *Settings* dashboard and select *Connected
+Services*. From here, you'll be able to connect to your GitHub or Bitbucket
+account. This process will ask you to authorize a connection to Read the Docs,
+that allows us to read information about and clone your repositories.
+
 .. _import-docs:
 
 Import Your Docs
 ----------------
 
-`Sign up`_ for an account on RTD, then `log in`_. Visit your dashboard_ and click
-Import_ to add your project to the site. Fill in the name and description, then
-specify where your repository is located. This is normally the URL or path name
-you'd use to checkout, clone, or branch your code. Some examples:
+To import a repository, visit your dashboard_ and click Import_.
+
+If you have a connected account, you will see a list of your repositories that
+we are able to import. To import one of these projects, just click the import
+icon next to the repository you'd like to import. This will bring up a form that
+is already filled with your project's information. Feel free to edit any of
+these properties, and the click **Next** to build your documentation.
+
+Manually Import Your Docs
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you do not have a connected account, you will need select **Import Manually**
+and enter the information for your repository yourself. You will also need to
+manually configure the webhook for your repository as well. When importing your
+project, you will be asked for the repository URL, along with some other
+information for you new project. The URL is normally the URL or path name you'd
+use to checkout, clone, or branch your repository. Some examples:
 
 * Git: ``http://github.com/ericholscher/django-kong.git``
-* Subversion: ``http://varnish-cache.org/svn/trunk``
 * Mercurial: ``https://bitbucket.org/ianb/pip``
+* Subversion: ``http://varnish-cache.org/svn/trunk``
 * Bazaar: ``lp:pasta``
 
-Add an optional homepage URL and some tags, then click "Create".
+Add an optional homepage URL and some tags, and then click **Next**.
+
+Once your project is created, you'll need to manually configure the repository
+webhook if you would like to have new changesets to trigger builds for your
+project on Read the Docs. Go to your project's **Integrations** page to
+configure a new webhook, or see :ref:`our steps for webhook creation
+<webhook-creation>` for more information on this process.
 
 Within a few seconds your code will automatically be fetched from your public repository,
 and the documentation will be built.
 Check out our :doc:`builds` page to learn more about how we build your docs,
 and to troubleshoot any issues that arise.
 
-If you want to keep your code updated as you commit,
-configure your code repository to hit our `Post Commit Hooks`_.
-This will rebuild your docs every time you push your code.
-
-We support multiple versions of your code. You can read more about how to use this well on our :doc:`versions` page.
+Read the Docs will host multiple versions of your code. You can read more about
+how to use this well on our :doc:`versions` page.
 
 If you have any more trouble, don't hesitate to reach out to us. The :doc:`support` page has more information on getting in touch.
 
@@ -126,4 +159,3 @@ If you have any more trouble, don't hesitate to reach out to us. The :doc:`suppo
 .. _log in: http://readthedocs.org/accounts/login
 .. _dashboard: http://readthedocs.org/dashboard
 .. _Import: http://readthedocs.org/dashboard/import
-.. _Post Commit Hooks: http://readthedocs.org/docs/read-the-docs/en/latest/webhooks.html

docs/webhooks.rst

@@ -1,78 +1,135 @@
 Webhooks
 ========
 
-Webhooks are pretty amazing, and help to turn the web into a push instead of
-pull platform. We have support for hitting a URL whenever you commit to your
-project and we will try and rebuild your docs. This only rebuilds them if
-something has changed, so it is cheap on the server side. As anyone who has
-worked with push knows, pushing a doc update to your repo and watching it get
-updated within seconds is an awesome feeling.
+The primary method that Read the Docs uses to detect changes to your
+documentation is through the use of *webhooks*. Webhooks are configured with
+your repository provider, such as GitHub or Bitbucket, and with each commit,
+merge, or other change to your repository, Read the Docs is notified. When we
+receive a webhook notification, we determine if the change is related to an
+active version for your project, and if it is, a build is triggered for that
+version.
 
-GitHub
----------
+.. _integration-detail:
 
-If your project is hosted on GitHub, you can easily add a hook that will rebuild
-your docs whenever you push updates:
+Webhook Integrations
+--------------------
 
-* Go to the "Settings" page for your project
-* Click "Integrations & Services"
-* In the "Services" section, click "Add service"
-* In the list of available services, click "ReadTheDocs"
-* Leave "Active" checked
-* Click "Add service"
+You'll find a list of configured webhook integrations on your project's admin
+dashboard, under **Integrations**. You can select any of these integrations to
+see the *integration detail page*. This page has additional configuration
+details and a list of HTTP exchanges that have taken place for the integration.
 
-.. note:: The GitHub URL in your Read the Docs project must match the URL on GitHub. The URL is case-sensitive.
+.. _webhook-creation:
 
-If you ever need to manually set the webhook on GitHub,
-you can point it at ``https://readthedocs.org/github``.
+Webhook creation
+----------------
 
-Bitbucket
------------
+If you import a project using a :ref:`connected account <connect-account>`, a
+webhook will be set up automatically for your repository. However, if your
+project was not imported through a connected account, you may need to
+manually configure a webhook for your project.
+
+To manually set up a webhook, click **Add integration** on your project's
+**Integrations** admin dashboard page and select the integration type you'd like
+to add. After you have added the integration, you'll see a URL for the
+integration on the :ref:`integration detail page <integration-detail>`. Use this
+URL when setting up a new webhook with your provider -- these steps vary
+depending on the provider:
+
+GitHub
+~~~~~~
+
+* Go to the **Settings** page for your project
+* Click **Webhooks** and then **Add webhook**
+* For **Payload URL**, use the URL of the integration on Read the Docs, found on
+  the :ref:`integration detail page <integration-detail>` page
+* For **Content type**, both *application/json* and
+  *application/x-www-form-urlencoded* work
+* Select **Just the push event**
+* Finish by clicking **Add webhook**
 
-If your project is hosted on Bitbucket, you can easily add a hook that will rebuild
-your docs whenever you push updates:
+.. note:: The webhook secret is not yet respected
 
-* Go to the "admin" page for your project
-* Click "Services"
-* In the available service hooks, select "Read the Docs"
-* Click "Add service"
+Bitbucket
+~~~~~~~~~
 
-If you ever need to manually set the webhook on Bitbucket,
-you can point it at ``https://readthedocs.org/bitbucket``.
+* Go to the **Settings** page for your project
+* Click **Webhooks** and then **Add webhook**
+* For **URL**, use the URL of the integration on Read the Docs, found on the
+  :ref:`integration detail page <integration-detail>` page
+* Under **Triggers**, **Repository push** should be selected
+* Finish by clicking **Save**
 
 GitLab
----------
+~~~~~~
+
+* Go to the **Settings** page for your project
+* Click **Integrations**
+* For **URL**, use the URL of the integration on Read the Docs, found on the
+  :ref:`integration detail page <integration-detail>` page
+* Leave the default **Push events** selected
+* Finish by clicking **Add Webhook**
 
-If your project is hosted on GitLab, you can easily add a hook that will rebuild
-your docs whenever you push updates.
+Using the generic API integration
+---------------------------------
 
-* Go to the "Settings" page for your project
-* Click "Integrations"
-* In the "URL" section, enter ``https://readthedocs.org/gitlab``
-* Leave the default "Push events" selected
-* Click "Add Webhook"
+For repositories that are not hosted with a supported provider, we also offer a
+generic API endpoint for triggering project builds. Similar to webhook
+integrations, this integration has a specific URL, found on the
+:ref:`integration detail page <integration-detail>`.
 
-Others
-------
+Token authentication is required to use the generic endpoint, you will find this
+token on the integration details page. The token should be passed in as a
+request parameter, either as form data or as part of JSON data input.
 
-Your ReadTheDocs project detail page has your post-commit hook on it; it will
-look something along the lines of ``http://readthedocs.org/build/<project_name>``.
-Regardless of which revision control system you use, you can just hit this URL
-to kick off a rebuild.
+Parameters
+~~~~~~~~~~
 
-The following parameters available to customize the behavior of custom webhooks:
+This endpoint accepts the following arguments during an HTTP POST:
 
-* ``'version_slug'``: The build version to trigger build for (defaults to ``'latest'``)
+branches
+    The names of the branches to trigger builds for. This can either be an array
+    of branch name strings, or just a single branch name string.
 
-  Example::
+    Default: **latest**
 
-      $ curl -X POST --data "version_slug=$VERSION" https://readthedocs.org/build/$PROJECT_NAME
+token
+    The integration token. You'll find this value on the
+    :ref:`integration detail page <integration-detail>` page.
 
-You could make this part of a hook using Git_, Subversion_, Mercurial_, or
-Bazaar_, perhaps through a simple script that accesses the build URL using
-``wget`` or ``curl``.
+For example, the cURL command to build the ``dev`` branch, using the token
+``1234``, would be::
+
+    curl -X POST -d "branches=dev" -d "token=1234" https://readthedocs.org/api/v2/webhook/example-project/1/
+
+A command like the one above could be called from a cron job or from a hook
+inside Git_, Subversion_, Mercurial_, or Bazaar_.
 
 .. _Git: http://www.kernel.org/pub/software/scm/git/docs/githooks.html
 .. _Subversion: http://mikewest.org/2006/06/subversion-post-commit-hooks-101
 .. _Mercurial: http://hgbook.red-bean.com/read/handling-repository-events-with-hooks.html
 .. _Bazaar: http://wiki.bazaar.canonical.com/BzrHooks
+
+Authentication
+~~~~~~~~~~~~~~
+
+This endpoint requires authentication. If authenticating with an integration
+token, a check will determine if the token is valid and matches the given
+project. If instead an authenticated user is used to make this request, a check
+will be performed to ensure the authenticated user is an owner of the project.
+
+Debugging webhooks
+------------------
+
+If you are experiencing problems with an existing webhook, you may be able to
+use the integration detail page to help debug the issue. Each project
+integration, such as a webhook or the generic API endpoint, stores the HTTP
+exchange that takes place between Read the Docs and the external source. You'll
+find a list of these exchanges in any of the integration detail pages.
+
+Resyncing webhooks
+------------------
+
+It might be necessary to re-establish a webhook if you are noticing problems.
+To resync a webhook from Read the Docs, visit the integration detail page and
+follow the directions for re-syncing your repository webhook.

media/css/core.css

@@ -119,6 +119,12 @@ h3 > span.link-help {
     float: right;
 }
 
+form.form-wide input[type='text'],
+form.form-wide select,
+form.form-wide textarea {
+    width: 100%;
+}
+
 /* content */
 
 #content { padding-top: 50px; }
@@ -925,6 +931,22 @@ body .edit-toggle { display: none; }
 
 .navigable ul input[type=text] { width: 164px; }
 
+div.button-bar ul {
+    list-style: none;
+    text-align: right;
+}
+
+div.button-bar ul li {
+    display: inline-block;
+}
+
+div.button-bar li a.button,
+div.button-bar li input[type="submit"],
+div.button-bar li input[type="button"],
+div.button-bar li button {
+    margin-top: .5em;
+    margin-bottom: .5em;
+}
 
 select.dropdown { display: none; }
 .dropdown > a { font-family: "ff-meta-web-pro", "ff-meta-web-pro-1", "ff-meta-web-pro-2", Arial, "Helvetica Neue", sans-serif; color: #666; font-weight: bold; padding: 8px 15px; border: none; background: #e6e6e6 url(../images/gradient.png) repeat-x bottom left; margin: 30px 5px 20px 0; text-shadow: 0 1px 0 rgba(255, 255, 255, 1); border: 1px solid #bfbfbf; display: block; text-decoration: none; box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 rgba(255, 255, 255, 0.5) inset; -moz-box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 rgba(255, 255, 255, 0.5) inset; -webkit-box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 rgba(255, 255, 255, 0.5) inset; }
@@ -1044,28 +1066,40 @@ div.module-list-wrapper.httpexchanges li span.status.status-fail {
     background: #a55;
 }
 
+div.integration-details {
+    margin: 1em;
+}
+
+div.integration-details dl dt,
 div.httpexchange dl dt {
-  display: inline-block;
-  font-weight: bold;
-  font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace;
-  font-size: .9em;
+    display: inline-block;
+    font-weight: bold;
+}
+div.httpexchange dl dt {
+    font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace;
+    font-size: .9em;
 }
 
+div.integration-details dl dd,
+div.httpexchange dl dd {
+    display: inline;
+    font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace;
+}
 div.httpexchange dl dd {
-  display: inline;
-  font-family: 'inconsolata', 'bitstream vera sans mono', 'andale mono', 'lucida console', monospace;
-  font-size: .9em;
+    font-size: .9em;
 }
+
+div.integration-details dl dd:after,
 div.httpexchange dl dd:after {
-  display: block;
-  content: '';
+    display: block;
+    content: '';
 }
 
 div.httpexchange div.highlight pre {
-  padding: 1em;
-  background: #f4f4f4;
-  border: 1px solid #ccc;
-  font-size: .9em;
+    padding: 1em;
+    background: #f4f4f4;
+    border: 1px solid #ccc;
+    font-size: .9em;
 }
 
 /* Pygments */

readthedocs/constants.py

@@ -9,4 +9,5 @@
     'lang_slug': LANGUAGES_REGEX,
     'version_slug': VERSION_SLUG_REGEX,
     'filename_slug': '(?:.*)',
+    'integer_pk': r'[\d]+',
 }

readthedocs/core/fields.py

@@ -0,0 +1,9 @@
+"""Shared model fields and defaults"""
+
+import binascii
+import os
+
+
+def default_token():
+    """Generate default value for token field"""
+    return binascii.hexlify(os.urandom(20)).decode()