Adding auto doc links for READMEs.

appengine/bigquery/README.md

@@ -2,6 +2,15 @@
 
 This sample demonstrates [authenticating to BigQuery in App Engine using OAuth2](https://cloud.google.com/bigquery/authentication).
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+* https://cloud.google.com/bigquery/authentication
+* https://cloud.google.com/monitoring/api/authentication
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../README.md) for information on how to run and deploy this sample.
 
 ## Setup

appengine/blobstore/README.md

@@ -0,0 +1,9 @@
+# App Engine Blobstore Sample
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/blobstore/
+
+<!-- end-auto-doc-link -->
+ -->

appengine/images/README.md

@@ -3,4 +3,11 @@
 This is a sample app for Google App Engine that demonstrates the [Images Python
 API](https://cloud.google.com/appengine/docs/python/images/usingimages).
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/images/usingimages
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../README.md) for information on how to run and deploy this sample.

appengine/localtesting/README.md

@@ -0,0 +1,10 @@
+# App Engine Local Testing Samples
+
+These samples show how to do automated testing of App Engine applications.
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/tools/localunittesting
+
+<!-- end-auto-doc-link -->

appengine/memcache/guestbook/README.md

@@ -1,5 +1,12 @@
 # Memcache Guestbook Sample
 
-This is a sample app for Google App Engine that demonstrates the [Memcache Python API](https://cloud.google.com/appengine/docs/python/memcache/usingmemcache).
+This is a sample app for Google App Engine that demonstrates the Memcache Python API.
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/memcache/usingmemcache
+
+<!-- end-auto-doc-link -->
 
 Refer to the [App Engine Samples README](../../README.md) for information on how to run and deploy this sample.

appengine/multitenancy/README.md

@@ -2,4 +2,11 @@
 
 This sample demonstrates how to use Google App Engine's [Namespace Manager API](https://cloud.google.com/appengine/docs/python/multitenancy/multitenancy).
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/multitenancy/namespaces
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../README.md) for information on how to run and deploy this sample.

appengine/ndb/overview/README.md

@@ -2,4 +2,11 @@
 
 This is a sample app for Google App Engine that demonstrates the [Datastore NDB Python API](https://cloud.google.com/appengine/docs/python/ndb/).
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/ndb/
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../../README.md) for information on how to run and deploy this sample.

appengine/ndb/transactions/README.md

@@ -1,7 +1,14 @@
 ## App Engine Datastore NDB Transactions Sample
 
-This is a sample app for Google App Engine that exercises the [NDB Transactions Python API](https://cloud.google.com/appengine/docs/python/ndb/transactions)
+This is a sample app for Google App Engine that demonstrates the [NDB Transactions Python API](https://cloud.google.com/appengine/docs/python/ndb/transactions)
 
 This app presents a list of notes. After you submit a note with a particular title, you may not change that note or submit a new note with the same title. There are multiple note pages available.
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/ndb/transactions
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../../README.md) for information on how to run and deploy this sample.

bigquery/api/README.md

@@ -0,0 +1,16 @@
+# BigQuery API Samples
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+* https://cloud.google.com/bigquery/exporting-data-from-bigquery
+* https://cloud.google.com/bigquery/authentication
+* https://cloud.google.com/bigquery/bigquery-api-quickstart
+* https://cloud.google.com/bigquery/docs/managing_jobs_datasets_projects
+* https://cloud.google.com/bigquery/streaming-data-into-bigquery
+* https://cloud.google.com/bigquery/docs/data
+* https://cloud.google.com/bigquery/querying-data
+* https://cloud.google.com/bigquery/loading-data-into-bigquery
+
+<!-- end-auto-doc-link -->

compute/api/README.md

@@ -0,0 +1,13 @@
+# Compute Engine API Samples
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/compute/docs/tutorials/python-guide
+
+<!-- end-auto-doc-link -->
+
+
+
+
+

monitoring/api/README.md

@@ -0,0 +1,10 @@
+# Cloud Monitoring API Samples
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+* https://cloud.google.com/monitoring/demos/
+* https://cloud.google.com/monitoring/api/authentication
+
+<!-- end-auto-doc-link -->

scripts/README.md

@@ -0,0 +1 @@
+These scripts are used for the maintenance of this repository and are not necessarily meant as samples.

scripts/auto_link_to_docs.py

@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2013 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Process docs-links.json and updates all READMEs and replaces
+
+<!-- auto-doc-link --><!-- end-auto-doc-link -->
+
+With a generated list of documentation backlinks.
+"""
+
+from collections import defaultdict
+import json
+import os
+import re
+
+
+REPO_ROOT = os.path.abspath(os.path.join(
+ os.path.dirname(__file__),
+ '..'))
+DOC_SITE_ROOT = 'https://cloud.google.com'
+AUTO_DOC_LINK_EXP = re.compile(
+ r'<!-- auto-doc-link -->.*?<!-- end-auto-doc-link -->\n',
+ re.DOTALL)
+
+
+def invert_docs_link_map(docs_links):
+ """
+ The docs links map is in this format:
+
+ {
+ "doc_path": [
+ "file_path",
+ ]
+ }
+
+ This transforms it to:
+
+ {
+ "file_path": [
+ "doc_path",
+ ]
+ }
+ """
+ files_to_docs = defaultdict(list)
+ for doc, files in docs_links.iteritems():
+ for file in files:
+ files_to_docs[file].append(doc)
+ files_to_docs[file] = list(set(files_to_docs[file]))
+
+ return files_to_docs
+
+
+def collect_docs_for_readmes(files_to_docs):
+ """
+ There's a one-to-many relationship between readmes and files. This method
+ finds the readme for each file and consolidates all docs references.
+ """
+ readmes_to_docs = defaultdict(list)
+ for file, docs in files_to_docs.iteritems():
+ readme = get_readme_path(file)
+ readmes_to_docs[readme].extend(docs)
+ readmes_to_docs[readme] = list(set(readmes_to_docs[readme]))
+ return readmes_to_docs
+
+
+def linkify(docs):
+ """Adds the documentation site root to doc paths, creating a full URL."""
+ return [DOC_SITE_ROOT + x for x in docs]
+
+
+def replace_contents(file_path, regex, new_content):
+ with open(file_path, 'r+') as f:
+ content = f.read()
+ content = regex.sub(new_content, content)
+ f.seek(0)
+ f.write(content)
+
+
+def get_readme_path(file_path):
+ """Gets the readme for an associated sample file, basically just the
+ README.md in the same directory."""
+ dir = os.path.dirname(file_path)
+ readme = os.path.join(
+ REPO_ROOT, dir, 'README.md')
+ return readme
+
+
+def generate_doc_link_statement(docs):
+ links = linkify(docs)
+ if len(links) == 1:
+ return """<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> {}
+
+<!-- end-auto-doc-link -->
+""".format(links.pop())
+ else:
+ return """<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+{}
+
+<!-- end-auto-doc-link -->
+""".format('\n'.join(['* {}'.format(x) for x in links]))
+
+
+def update_readme(readme_path, docs):
+ if not os.path.exists(readme_path):
+ print('{} doesn\'t exist'.format(readme_path))
+ return
+ replace_contents(
+ readme_path,
+ AUTO_DOC_LINK_EXP,
+ generate_doc_link_statement(docs))
+ print('Updated {}'.format(readme_path))
+
+
+def main():
+ docs_links = json.load(open(
+ os.path.join(REPO_ROOT, 'scripts', 'docs-links.json'), 'r'))
+ files_to_docs = invert_docs_link_map(docs_links)
+ readmes_to_docs = collect_docs_for_readmes(files_to_docs)
+
+ for readme, docs in readmes_to_docs.iteritems():
+ update_readme(readme, docs)
+
+if __name__ == '__main__':
+ main()

scripts/docs-links.json

@@ -0,0 +1,87 @@
+{
+ "/appengine/docs/python/ndb/": [
+ "appengine/ndb/overview/main.py"
+ ], 
+ "/storage/transfer/create-client": [
+ "storage/transfer_service/create_client.py"
+ ], 
+ "/storage/docs/json_api/v1/json-api-python-samples": [
+ "storage/api/list_objects.py"
+ ], 
+ "/bigquery/querying-data": [
+ "bigquery/api/async_query.py", 
+ "bigquery/api/sync_query.py"
+ ], 
+ "/bigquery/bigquery-api-quickstart": [
+ "bigquery/api/getting_started.py"
+ ], 
+ "/bigquery/docs/managing_jobs_datasets_projects": [
+ "bigquery/api/list_datasets_projects.py"
+ ], 
+ "/appengine/docs/python/blobstore/": [
+ "appengine/blobstore/main.py"
+ ], 
+ "/appengine/docs/python/images/usingimages": [
+ "appengine/images/main.py"
+ ], 
+ "/compute/docs/tutorials/python-guide": [
+ "compute/api/create_instance.py", 
+ "compute/api/startup-script.sh"
+ ], 
+ "/docs/authentication": [
+ "storage/api/list_objects.py"
+ ], 
+ "/bigquery/loading-data-into-bigquery": [
+ "bigquery/api/load_data_from_csv.py", 
+ "bigquery/api/load_data_by_post.py"
+ ], 
+ "/bigquery/streaming-data-into-bigquery": [
+ "bigquery/api/streaming.py"
+ ], 
+ "/appengine/docs/python/memcache/usingmemcache": [
+ "appengine/memcache/guestbook/main.py"
+ ], 
+ "/monitoring/demos/": [
+ "monitoring/api/auth.py"
+ ], 
+ "/appengine/docs/python/multitenancy/namespaces": [
+ "appengine/multitenancy/memcache.py", 
+ "appengine/multitenancy/datastore.py", 
+ "appengine/multitenancy/taskqueue.py"
+ ], 
+ "/bigquery/docs/data": [
+ "bigquery/api/sync_query.py"
+ ], 
+ "/bigquery/authentication": [
+ "appengine/bigquery/main.py", 
+ "bigquery/api/getting_started.py"
+ ], 
+ "/bigquery/exporting-data-from-bigquery": [
+ "bigquery/api/export_data_to_cloud_storage.py"
+ ], 
+ "/appengine/docs/python/ndb/transactions": [
+ "appengine/ndb/transactions/main.py"
+ ], 
+ "/monitoring/api/authentication": [
+ "appengine/bigquery/app.yaml", 
+ "monitoring/api/auth.py", 
+ "appengine/bigquery/main.py"
+ ], 
+ "/storage/transfer/create-transfer": [
+ "storage/transfer_service/nearline_request.py", 
+ "storage/transfer_service/aws_request.py", 
+ "storage/transfer_service/transfer_check.py"
+ ], 
+ "/storage/docs/authentication": [
+ "storage/api/list_objects.py"
+ ], 
+ "/appengine/docs/python/tools/localunittesting": [
+ "appengine/localtesting/runner.py", 
+ "appengine/localtesting/test_task_queue.py", 
+ "appengine/localtesting/test_login.py", 
+ "appengine/localtesting/queue.yaml", 
+ "appengine/localtesting/test_mail.py", 
+ "appengine/localtesting/test_env_vars.py", 
+ "appengine/localtesting/test_datastore.py"
+ ]
+}

storage/api/README.md

@@ -0,0 +1,11 @@
+# Cloud Storage API Samples
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+* https://cloud.google.com/storage/docs/json_api/v1/json-api-python-samples
+* https://cloud.google.com/storage/docs/authentication
+* https://cloud.google.com/docs/authentication
+
+<!-- end-auto-doc-link -->