Get doctests in shape to be tested by tox

[dhermes]

Make tox test all the doctests in the Sphinx docs.

Ours aren't in shape to do that (yet, might be hard to get there).

See comment by @tseaver for more context

[tseaver]

After making some stab at it, I'm going to abandon this one: the doctest snippets are basically regression-like tests, and can't be exercised without having those environment variables set up.

[dhermes]

SGTM

[dhermes]

Revisiting this based on @tseaver comment in
dhermes@1b1fb11#commitcomment-12114109
and
#933 (comment)
googleapis/google-cloud-node#627

[jgeewax]

Wait -- those comments sounded like they were saying "we shouldn't test the examples in docstrings".

I'm glad this is re-opened though. Should we instead make a call on what the proper environment is to run these tests? (I'd expect to have the system-test-like environment in place when running docstring examples...)

[tseaver]

@jgeewax I'd like to separate the "API documentation" (which uses the :type foo: / :param foo: / :rtype: / :returns: / :raises: markers to specify the contract) from the examples used to support narrative: I believe the best place for those examples is in Sphinx .. doctest:: sections. I've made such examples testable on other projects: see, for instance, zope.component:

• its tox.ni uses sphinx-build -b doctest to exercise the snippets.
• see the configure.rst document uses .. testsetup:: and .. testcleanup:: to handle "offstage" setup / teardown.

In our case, we would have to test those examples inside an environment like our current system-tests (with appropriate credentials, etc.)

[jgeewax]

I'm not super concerned about where the tests live, but I really do want the examples near the methods, ie https://googlecloudplatform.github.io/gcloud-python/latest/storage-buckets.html#gcloud.storage.bucket.Bucket.delete_blob

Does that change anything?

[tseaver]

If we leave the examples embedded in method docstrings, we are pretty much giving up on testing them: the Sphinx facilities for doing setup / teardown are not going to mesh with that choice. Moving the examples to narrative docs allows for testing them, as well as giving the method docstrings tighter focus.

[jgeewax]

I hear you -- there must be a way to allow .. doctest:: or .. example:: sections in a docstring to be tested, and still live alongside the methods...

The narrative docs are nice, but not nearly as useful to users as the examples in the context of the method...

[tseaver]

I am arguing from the perspective the two kinds of docs are aimed at different classes of users, who need different tone / format:

• Well-structured narrative docs with good examples are aimed at users who are new to the library, and need the "guided tour" approach which the examples support. They focus on the "main path" / common case, and expose the new user to concepts in an order which aids them in building an understanding of why to use various features, more than the nitty-gritty details. In Sphinx, the narrative docs should always contain links to the appropriate sections of the reference docs.
• API reference docs are most useful for refreshing an experienced user's memory of an API's precise signature / contract: conciseness / precision is most helpful for them. Building explicit references from the reference docs is tricky / confusing, because the narrative docs may have multiple locations which discuss a given API.

For an extensive example of documents built using these principals, see Pyramid's narrative docs versus its API reference docs.

[jgeewax]

I'm not sure I agree 100%, but regardless, I'm looking for the simple examples like http://docs.pylonsproject.org/projects/pyramid/en/latest/api/interfaces.html#pyramid.interfaces.IBeforeRender I think the narrative documentation is important, however given I'm looking at a method that "looks like what I want" (common quote from the user study), why wouldn't I have an example of how to use it ?

[dhermes]

@tseaver What do we need to do to get doc snippets tested?

[jgeewax]

Can't nose do that (--with-doctest)?

[dhermes]

The point of this issue is that the current snippets we have can't be run with a simple nose --with-doctest since they need some kind of set up.

As @tseaver says above they are basically regression tests (we now call them system tests).

[jgeewax]

What about having a standard setUp-type method that does stupid mocks for everything?

We're not looking for "does this return the exact values we expected?", we're looking for "does the method signature line up?" or "are there typos or syntax errors?"

Anything else I'm fine to catch manually -- and maybe one day we can make these act like regression/system tests, hitting the live APIs.

[dhermes]

I think we could probably build a harness that steps in front and mocks them in the necessary way, it just requires discussion.

@tseaver said

After making some stab at it, I'm going to abandon this one

But we never had a back and forth and at the time didn't value the proposition very much.

UPDATE: Now that its more relevant (as we've seen it keep coming up), it's worth having a discussion.

[jgeewax]

Awesome. Let's dig into this then. Should we create a new issue or continue here?

[dhermes]

Continue here.

[dhermes]

@tseaver Do we have a plan here?