Merge branch 'master' into index-more-domain-data

docs/advertising/ad-blocking.rst

@@ -53,7 +53,7 @@ Going ad-free
 -------------
 
 Users can go completely ad-free
-by becoming a `Gold Member <https://readthedocs.org/accounts/gold/>`_
+by becoming a `Gold member <https://readthedocs.org/accounts/gold/>`_
 or a `Supporter <https://readthedocs.org/sustainability/#donations>`_.
 Thank you for supporting Read the Docs.
 

docs/advertising/ethical-advertising.rst

@@ -156,8 +156,9 @@ Opting Out
 We have added multiple ways to opt out of the advertising on Read the Docs.
 
 1. You can go completely ad-free
-   by becoming a `Gold Member <https://readthedocs.org/accounts/gold/>`_
+   by becoming a `Gold member <https://readthedocs.org/accounts/gold/>`_
    or a `Supporter <https://readthedocs.org/sustainability/#donations>`_.
+   Additionally, Gold members may remove advertising from their projects for all visitors.
 
 2. You can opt out of seeing paid advertisements on documentation pages:
 
@@ -182,5 +183,4 @@ We have added multiple ways to opt out of the advertising on Read the Docs.
    but our commercial plans don't seem like the right fit,
    please `get in touch`_ to discuss alternatives to advertising.
 
-.. _paid plans: https://readthedocs.com/pricing/
 .. _get in touch: mailto:ads@readthedocs.org?subject=Alternatives%20to%20advertising

docs/advertising/index.rst

@@ -22,13 +22,14 @@ that respects user privacy.
 
 We recognize that advertising is not for everyone.
 You may :ref:`opt out of paid advertising <advertising/ethical-advertising:Opting Out>`
--- you will still see :ref:`community ads <advertising/ethical-advertising:Community Ads>` --
-or you can go ad-free by `becoming a Gold Member`_ or a `Supporter`_ of Read the Docs.
+although you will still see :ref:`community ads <advertising/ethical-advertising:Community Ads>`.
+You can go ad-free by `becoming a Gold member`_ or a `Supporter`_ of Read the Docs.
+Gold members can also remove advertising from their projects for all visitors.
 
 For businesses looking to remove advertising,
 please consider :doc:`Read the Docs for Business </commercial/index>`.
 
-.. _becoming a Gold Member: https://readthedocs.org/accounts/gold/
+.. _becoming a Gold member: https://readthedocs.org/accounts/gold/
 .. _Supporter: https://readthedocs.org/sustainability/#donations
 
 .. toctree::

docs/conf.py

@@ -56,7 +56,7 @@ def get_version():
 intersphinx_mapping = {
     'python': ('https://python.readthedocs.io/en/latest/', None),
     'django': ('https://django.readthedocs.io/en/1.11.x/', None),
-    'sphinx': ('https://sphinx.readthedocs.io/en/latest/', None),
+    'sphinx': ('https://www.sphinx-doc.org/en/master/', None),
 }
 htmlhelp_basename = 'ReadTheDocsdoc'
 latex_documents = [

docs/guides/index.rst

@@ -34,6 +34,7 @@ These guides will help you customize or tune aspects of the Read the Docs build
 
     build-notifications
     build-using-too-many-resources
+    technical-docs-seo-guide
     canonical
     conda
     environment-variables

docs/guides/technical-docs-seo-guide.rst

@@ -0,0 +1,286 @@
+Technical Documentation Search Engine Optimization (SEO) Guide
+==============================================================
+
+.. meta::
+    :description lang=en:
+        Looking to optimize your Sphinx documentation for search engines?
+        This SEO guide will help your docs be better understood by both people and crawlers
+        as well as help you rank higher in search engine results.
+
+
+This guide will help you optimize your documentation for search engines
+with the goal of increasing traffic to your docs.
+While you optimize your docs to make them more crawler friendly for search engine spiders,
+it's important to keep in mind that your ultimate goal is to make your docs
+more discoverable for your users.
+**You're trying to make sure that when a user types a question into a search engine
+that is answerable by your documentation, that your docs appear in the results.**
+
+This guide isn't meant to be your only resource on SEO,
+and there's a lot of SEO topics not covered here.
+For additional reading, please see the
+:ref:`external resources <guides/technical-docs-seo-guide:External resources>` section.
+
+While many of the topics here apply to all forms of technical documentation,
+this guide will focus on Sphinx, which is the most common
+documentation authoring tool on Read the Docs,
+as well as improvements provided by Read the Docs itself.
+
+
+.. contents:: Table of contents
+   :local:
+   :backlinks: none
+   :depth: 3
+
+
+SEO Basics
+----------
+
+Search engines like Google and Bing crawl through the internet
+following links in an attempt to understand and build an index
+of what various pages and sites are about.
+This is called "crawling" or "indexing".
+When a person sends a query to a search engine,
+the search engine evaluates this index using a number of factors
+and attempts to return the results most likely to answer that person's question.
+
+How search engines "rank" sites based on a person's query
+is part of their secret sauce.
+While some search engines publish the basics of their algorithms
+(see Google's published details on PageRank), few search engines
+give all of the details in an attempt to prevent users from gaming the rankings
+with low value content which happens to rank well.
+
+Both `Google <https://support.google.com/webmasters/answer/35769>`_
+and `Bing <https://www.bing.com/webmaster/help/webmaster-guidelines-30fba23a>`_ publish a set of guidelines
+to help make sites easier to understand for search engines and rank better.
+To summarize some of the most important aspects
+as they apply to technical documentation, your site should:
+
+* Use descriptive and accurate titles in the HTML ``<title>`` tag.
+* Ensure your URLs are descriptive. They are displayed in search results.
+* Avoid low content pages or pages with very little original content.
+* Avoid tactics that attempt to increase your search engine ranking
+  without actually improving content.
+* Google specifically `warns about automatically generated content`_
+  although this applies primarily to keyword stuffing and low value content.
+  High quality documentation generated from source code
+  (eg. auto generated API documentation) seems OK.
+
+  .. _warns about automatically generated content: https://support.google.com/webmasters/answer/2721306
+
+While both Google and Bing discuss site performance as an important factor
+in search result ranking, this guide is not going to discuss it in detail.
+Most technical documentation that uses Sphinx or Read the Docs
+generates static HTML and the performance is typically decent
+relative to most of the internet.
+
+
+Optimizing your docs for search engine spiders
+----------------------------------------------
+
+Once a crawler or spider finds your site, it will follow links and redirects
+in an attempt to find any and all pages on your site.
+While there are a few ways to guide the search engine in its crawl
+for example by using a :ref:`sitemap <guides/technical-docs-seo-guide:Use a sitemap.xml file>`
+or a :ref:`robots.txt file <guides/technical-docs-seo-guide:Use a robots.txt file>`
+which we'll discuss shortly,
+the most important thing is making sure the spider can follow links on your site
+and get to all your pages.
+
+Avoid orphan pages
+~~~~~~~~~~~~~~~~~~
+
+Sphinx calls pages that don't have links to them "orphans"
+and will throw a warning while building documentation that contains an orphan
+unless the warning is silenced with the :ref:`orphan directive <sphinx:metadata>`:
+
+::
+
+    $ make html
+    sphinx-build -b html -d _build/doctrees . _build/html
+    Running Sphinx v1.8.5
+    ...
+    checking consistency... /path/to/file.rst: WARNING: document isn't included in any toctree
+    done
+    ...
+    build finished with problems, 1 warning.
+
+You can make all Sphinx warnings into errors during your build process
+by setting ``SPHINXOPTS = -W --keep-going`` in your Sphinx Makefile.
+
+
+Avoid uncrawlable content
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+While typically this isn't a problem with technical documentation,
+try to avoid content that is "hidden" from search engines.
+This includes content hidden in images or videos
+which the crawler may not understand.
+For example, if you do have a video in your docs,
+make sure the rest of that page describes the content of the video.
+
+When using images, make sure to set the image alt text or set a caption on figures.
+For Sphinx, the image and figure directives support this:
+
+.. sourcecode:: rst
+
+    .. image:: your-image.png
+        :alt: A description of this image
+
+    .. figure:: your-image.png
+
+        A caption for this figure
+
+
+Redirects
+~~~~~~~~~
+
+Redirects tell search engines when content has moved.
+For example, if this guide moved from ``guides/technical-docs-seo-guide.html`` to ``guides/sphinx-seo-guide.html``,
+there will be a time period where search engines will still have the old URL in their index
+and will still be showing it to users.
+This is why it is important to update your own links within your docs as well as redirecting.
+If the hostname moved from docs.readthedocs.io to docs.readthedocs.org, this would be even more important!
+
+Read the Docs supports a few different kinds of :doc:`user defined redirects </user-defined-redirects>`
+that should cover all the different cases such as redirecting a certain page for all project versions,
+or redirecting one version to another.
+
+
+Canonical URLs
+~~~~~~~~~~~~~~
+
+Anytime very similar content is hosted at multiple URLs,
+it is pretty important to set a canonical URL.
+The canonical URL tells search engines where the original version
+your documentation is even if you have multiple versions on the internet
+(for example, incomplete translations or deprecated versions).
+
+Read the Docs supports :doc:`setting the canonical URL </guides/canonical>`
+if you are using a :doc:`custom domain </custom_domains>`
+under :guilabel:`Admin` > :guilabel:`Domains`
+in the Read the Docs dashboard.
+
+
+Use a robots.txt file
+~~~~~~~~~~~~~~~~~~~~~
+
+A ``robots.txt`` file is readable by crawlers
+and lives at the root of your site (eg. https://docs.readthedocs.io/robots.txt).
+It tells search engines which pages to crawl or not to crawl
+and can allow you to control how a search engine crawls your site.
+For example, you may want to request that search engines
+:ref:`ignore unsupported versions of your documentation <faq:How can I avoid search results having a deprecated version of my docs?>`
+while keeping those docs online in case people need them.
+
+By default, Read the Docs serves a ``robots.txt`` for you.
+To customize this file, you can create a ``robots.txt`` file
+that is written to your documentation root on your default branch/version.
+
+See the `Google's documentation on robots.txt <https://support.google.com/webmasters/answer/6062608>`_
+for additional details.
+
+
+Use a sitemap.xml file
+~~~~~~~~~~~~~~~~~~~~~~
+
+A sitemap is a file readable by crawlers that contains a list of pages
+and other files on your site and some metadata or relationships about them
+(eg. https://docs.readthedocs.io/sitemap.xml).
+A good sitemaps provides information like how frequently a page or file is updated
+or any alternate language versions of a page.
+
+Read the Docs generates a sitemap for you that contains the last time
+your documentation was updated as well as links to active versions, subprojects, and translations your project has.
+We have a small separate guide on :doc:`sitemaps </guides/sitemaps>`.
+
+See the `Google docs on building a sitemap <https://support.google.com/webmasters/answer/183668>`_.
+
+
+Use meta tags
+~~~~~~~~~~~~~
+
+Using a meta description allows you to customize how your pages
+look in search engine result pages.
+
+Typically search engines will use the first few sentences of a page if no meta description is provided.
+In Sphinx, you can customize your meta description using the following RestructuredText:
+
+.. sourcecode:: rst
+
+    .. meta::
+        :description lang=en:
+            Adding additional CSS or JavaScript files to your Sphinx documentation
+            can let you customize the look and feel of your docs or add additional functionality.
+
+
+.. figure:: ../_static/images/guides/google-search-engine-results.png
+    :align: center
+    :figwidth: 80%
+    :target: ../_static/images/guides/google-search-engine-results.png
+
+    Google search engine results showing a customized meta description
+
+Moz.com, an authority on search engine optimization,
+makes the following suggestions for meta descriptions:
+
+* Your meta description should have the most relevant content of the page.
+  A searcher should know whether they've found the right page from the description.
+* The meta description should be between 150-300 characters
+  and it may be truncated down to around 150 characters in some situations.
+* Meta descriptions are used for display but not for ranking.
+
+Search engines don't always use your customized meta description
+if they think a snippet from the page is a better description.
+
+Measure, iterate, & improve
+---------------------------
+
+Search engines (and soon, Read the Docs itself) can provide useful data
+that you can use to improve your docs' ranking on search engines.
+
+
+Search engine feedback
+~~~~~~~~~~~~~~~~~~~~~~
+
+`Google Search Console <https://search.google.com/search-console>`_ and
+`Bing Webmaster Tools <https://www.bing.com/webmaster/help/webmaster-guidelines-30fba23a>`_
+are tools for webmasters
+to get feedback about the crawling of their sites (or docs in our case).
+Some of the most valuable feedback these provide include:
+
+* Google and Bing will show pages that were previously indexed that now give a 404
+  (or more rarely a 500 or other status code).
+  These will remain in the index for some time but will eventually be removed.
+  This is a good opportunity to create a :ref:`redirect <guides/technical-docs-seo-guide:Redirects>`.
+* These tools will show any crawl issues with your documentation.
+* Search Console and Webmaster Tools will highlight security issues found
+  or if Google or Bing took action against your site because they believe it is spammy.
+
+
+Analytics tools
+~~~~~~~~~~~~~~~
+
+A tool like :doc:`Google Analytics </guides/google-analytics>`
+can give you feedback about the search terms people use to find your docs,
+your most popular pages, and lots of other useful data.
+
+Search term feedback can be used to help you optimize content for certain keywords or for related keywords.
+For Sphinx documentation, or other technical documentation that has its own search features,
+analytics tools can also tell you the terms people search for within your site.
+
+Knowing your popular pages can help you prioritize where to spend your SEO efforts.
+Optimizing your already popular pages can have a significant impact.
+
+
+External resources
+------------------
+
+Here are a few additional resources to help you learn more about SEO
+and rank better with search engines.
+
+* `Moz's beginners guide to SEO <https://moz.com/beginners-guide-to-seo>`_
+* `Google's Webmaster Guidelines <https://support.google.com/webmasters/answer/35769>`_
+* `Bing's Webmaster Guidelines <https://www.bing.com/webmaster/help/webmaster-guidelines-30fba23a>`_
+* `Google's SEO Starter Guide <https://support.google.com/webmasters/answer/7451184>`_

docs/privacy-policy.rst

@@ -166,12 +166,12 @@ and we encourage you to check back periodically.
 Payment processing
 ++++++++++++++++++
 
-Should you choose to become a `Supporter`_, purchase a `Gold Membership`_,
+Should you choose to become a `Supporter`_, purchase a `Gold membership`_,
 or become a subscriber to Read the Docs' commercial hosting product,
 your payment information and details will be processed by Stripe.
 Read the Docs does not store your payment information.
 
-.. _Gold Membership: https://readthedocs.org/accounts/gold/
+.. _Gold membership: https://readthedocs.org/accounts/gold/
 .. _Supporter: https://readthedocs.org/sustainability/
 
 Site monitoring

media/css/core.css

@@ -872,9 +872,12 @@ div.donate-stats h2 {
     margin: .5em 0em 1.5em 0em;
 }
 
+#content ul.donate-about {
+    margin-bottom: 1em;
+}
 ul.donate-about li {
-    margin-left: 1em;
-    list-style: inside;
+    margin-left: 1.5em;
+    list-style: initial;
 }
 
 div.donate-stats-sm form {