Simplified 'sphinx-build'

[stephenfin]

Subject: Simplify calling of sphinx-build

Feature or Bugfix

Feature

Purpose

Make sphinx-build easier to call by moving configuration of the build directory to the configuration file and auto-detecting the source directory. This will allow most users to call sphinx-build without any arguments.

Detail

• sphinx-build: Configure outdir from conf.py

  This is part one of two of the "make 'sphinx-build' easier to call" effort. In this part, we allow users to define the build directory in the 'conf.py' file. This means a user no longer needs to pass the option via the command line, though they can continue to do so if they wish.

  We use the behavior of the setuptools extension, whereby the results are output to a subfolder in the outdir named after the code of the builder. For example, if using the 'html' builder with 'output_dir = "test"', files would be written to 'test/html'.

• sphinx-build: Guess sourcedir if none provided

  This is part two of two of the "make 'sphinx-build' easier to call" effort. In this part, we attempt to guess where the source directory is if the user does not provide one. This is possible because many users use one of three directory names - doc, docs, or Documentation - for their documentation. We check these, along with a possible 'source' subdirectory, which is based on Sphinx's own defaults, for a 'conf.py' file. Is one is found, we assume this is the documentation source. This means a user no longer needs to pass any argument to 'sphinx-build' is they do not wish to.

  This is, of course, best-effort, and users can continue to provide these options manually if they so wish.

You can validate this right now by pulling down the pull request and running sphinx-build without any arguments. The required change has been made to conf.py.

[tk0miya]

Before reviewing this PR in detail, I'd like to ask you to some question.

At first, please let me know the motivation to define the output_dir to conf.py? I can understand to make the argument optional. But I don't understand to add it to the config value. What situation do you consider?

This is possible because many users use one of three directory names - doc, docs, or Documentation - for their documentation.

And another question is about this guessing. Is this behavior important? I feel this does not follow the UNIX philosophy. I prefer to determine current directory is source directory if it is not given.

[stephenfin]

Before reviewing this PR in detail, I'd like to ask you to some question.

At first, please let me know the motivation to define the output_dir to conf.py? I can understand to make the argument optional. But I don't understand to add it to the config value. What situation do you consider?

I see no reason not to 😄 This works towards two things I'd like to do long term - adding support for INI-style configuration files (to complement conf.py, like setup.cfg complements setup.py) and removing make mode. The latter is more relevant to this change: I don't like make mode and think it solves a problem, that sphinx-build is too complicated to run, in the wrong way. It's very unlikely that the build output directory will change regularly and we store every other bit of configuration in conf.py, so why not store this there too? As noted above, this removes one of two command-line options and allows us to run sphinx-build without arguments (though one can add -b BUILDER etc. if they wish). Eventually I would like to convert the OUTPUTDIR positional argument to an option (-o OUTPUTDIR?) but that needs a little more work. Possibly a Sphinx 2.0 feature.

This is possible because many users use one of three directory names - doc, docs, or Documentation - for their documentation.

And another question is about this guessing. Is this behavior important? I feel this does not follow the UNIX philosophy. I prefer to determine current directory is source directory if it is not given.

I feel it is very important, yes. The idea here is that you can run sphinx-build from the top-level directory of the project and it will find and build the docs. It works very well in my testing and a user can continue to provide this information manually if they so choose. Note also that this list is not complete. This is the exact list in the order in which it is checked:

• doc
• docs
• Documentation
• . (i.e. pwd)
• doc/source
• docs/source
• Documentation/source
• source

As you can see, we do actually check the current directory, although it's lower on the list than the other three. I can change this ordering if you'd like?

As for it not being UNIX like, I don't think that's quite relevant here. The UNIX philosophy is that you have lots of small tools which you chain together to achieve something and that doesn't seem to apply here. In addition, popular tools like git do contain some magic - you can run git log from any directory in a Git repo and it will work. The git executable is smart enough to scan up the tree and find the .git directory. We're just doing this in reverse 😄

[jfbu]

removing make mode. The latter is more relevant to this change: I don't like make mode and think it solves a problem, that sphinx-build is too complicated to run, in the wrong way.

Could you please elaborate what you mean by this about make mode. Personally I never use sphinx-build but only the lean Makefile interface. For example for testing Sphinx itself, I find make test TEST=test_build_latex.py quite nice to use (I know this is not same thing as Makefile for regular projects), or for regular projects I often do make html O="-a". I find GNU/Linux make interface very convenient.

This may not be best place to post my comment, as you have extended range of PRs, but I would like to know if your final goal is simply to drop shipped out Makefile approach for building projects? I understand currently you are only adding simplifications to direct usage of sphinx-build and afaik this has no immediate effect on make usage. But I would like to know more.

[stephenfin]

removing make mode. The latter is more relevant to this change: I don't like make mode and think it solves a problem, that sphinx-build is too complicated to run, in the wrong way.

Could you please elaborate what you mean by this about make mode. Personally I never use sphinx-build but only the lean Makefile interface.

I'm referring to the -M flag for sphinx-build. If you pass this, you trigger an entirely different application, which can be found in sphinx/make_mode.py. This application is essentially a wrapper around sphinx-build and is used by default in the the autogenerated Makefile in recent Sphinx releases, allowing us to reduce said Makefile from > 100 lines to ~20. However, this application is undocumented and, as noted above, I think this it was the wrong approach. We shouldn't need wrappers to call sphinx-build and we should be providing them merely as a nice-to-have. As such, I don't plan to remove the Makefile and make.bat generation. Instead, I plan to modify what's generated so they can simply call sphinx-build (without -M) again but stay as small as they were.

For example for testing Sphinx itself, I find make test TEST=test_build_latex.py quite nice to use (I know this is not same thing as Makefile for regular projects), or for regular projects I often do make html O="-a". I find GNU/Linux make interface very convenient.

This one's different. Have you used tox yet? You can call tox like so to achieve the above.

tox -e py27 tests/test_build_latex.py

This will take some time on the first run (because it's installing all your packages) but subsequent runs won't. tox is built upon virtualenv and is widely used in the Python world. It has many advantages, not least the fact that your environment is reproducible and dependencies are installed automatically. However, while I have been transitioning as much of the tooling as possible to emphasise tox with recent changes, I won't be seeking to remove the Makefile in the top-level Sphinx repo. This is because even though I personally don't see the need for it, I realize other people do actually use it and I don't want to hurt them. That said, I do think that those same people would benefit from switching to tox 😄

This may not be best place to post my comment, as you have extended range of PRs, but I would like to know if your final goal is simply to drop shipped out Makefile approach for building projects? I understand currently you are only adding simplifications to direct usage of sphinx-build and afaik this has no immediate effect on make usage. But I would like to know more.

Nope, as clarified above, I just want the Makefile to be unnecessary. I do want to remove sphinx-build -M, but it will be some time before I can do this (it deserves a good deprecation period).

As an aside, I have a few more things on top of these two items planned but I'm not actually sure where the best place to discuss these plans is. The mailing list doesn't seem to be actively monitored and I don't know if RFE bugs are a things here. Perhaps you or @tk0miya might have some ideas?

[jfbu]

@stephenfin thanks for the additional info. I will reply in a more detailed manner but I am currently on tight schedule and will only report one issue I currently have (at 2e04c2a):

tox -e py36 tests/test_build_texinfo.py::test_texinfo

reports a failure

=================================== FAILURES ===================================
_________________________________ test_texinfo _________________________________

app = <SphinxTestApp buildername='texinfo'>
status = <_io.StringIO object at 0x107955168>
warning = <_io.StringIO object at 0x1079559d8>

    @pytest.mark.sphinx('texinfo')
    def test_texinfo(app, status, warning):
        TexinfoTranslator.ignore_missing_images = True
        app.builder.build_all()
        # now, try to run makeinfo over it
        cwd = os.getcwd()
        os.chdir(app.outdir)
        try:
            try:
                p = Popen(['makeinfo', '--no-split', 'SphinxTests.texi'],
                          stdout=PIPE, stderr=PIPE)
            except OSError:
                raise pytest.skip.Exception  # most likely makeinfo was not found
            else:
                stdout, stderr = p.communicate()
                retcode = p.returncode
                if retcode != 0:
                    print(stdout)
                    print(stderr)
>                   assert False, 'makeinfo exited with return code %s' % retcode
E                   AssertionError: makeinfo exited with return code 2
E                   assert False

tests/test_build_texinfo.py:68: AssertionError
----------------------------- Captured stdout call -----------------------------
b''
b"Can't locate Locale/Messages.pm in @INC (@INC contains: /sw/lib/texinfo /sw/share/texinfo/lib/Unicode-EastAsianWidth/lib /sw/share/texinfo /sw/share/texinfo /Library/Perl/5.16/darwin-thread-multi-2level /Library/Perl/5.16 /Network/Library/Perl/5.16/darwin-thread-multi-2level /Network/Library/Perl/5.16 /Library/Perl/Updates/5.16.2 /System/Library/Perl/5.16/darwin-thread-multi-2level /System/Library/Perl/5.16 /System/Library/Perl/Extras/5.16/darwin-thread-multi-2level /System/Library/Perl/Extras/5.16 .) at /sw/bin/makeinfo line 109.\nBEGIN failed--compilation aborted at /sw/bin/makeinfo line 109.\n"
------------------------------ Captured log call -------------------------------
connectionpool.py          824 DEBUG    Starting new HTTPS connection (1): www.python.org
connectionpool.py          396 DEBUG    https://www.python.org:443 "GET /static/img/python-logo.png HTTP/1.1" 200 10102
--------------------------- Captured stdout teardown ---------------------------

which seems to indicate something is wrong at my locale with makeinfo, but on the other hand

$ make test TEST=test_build_texinfo.py::test_texinfo
Checking dependencies...
Temporary files will be placed in /path/to/sphinx/tests/build.
Running Sphinx test suite (with Python 3.6.4)...
============================= test session starts ==============================
platform darwin -- Python 3.6.4, pytest-3.3.0, py-1.5.2, pluggy-0.6.0 -- /opt/miniconda3/bin/python
cachedir: ../.cache
rootdir: /path/to/sphinx, inifile:
plugins: hypothesis-3.38.5
collected 1 item                                                               

test_build_texinfo.py::test_texinfo PASSED                               [100%]

=========================== 1 passed in 2.32 seconds ===========================

Possibly the tox virtual env is missing something. I can provide more log if required, but perhaps the explanation will be clear to others already. (I don't have time now to even read closely the error report, possibly simply a missing dependency)

[jfbu]

However, this application is undocumented and, as noted above, I think this it was the wrong approach. We shouldn't need wrappers to call sphinx-build and we should be providing them merely as a nice-to-have. As such, I don't plan to remove the Makefile and make.bat generation. Instead, I plan to modify what's generated so they can simply call sphinx-build (without -M) again but stay as small as they were.

Fair enough. Current docs (of master branch, since #3938 was merged, thus for 1.7) of sphinx-build explains how "make mode" adds two targets (latexpdf and info) which are not per se Sphinx builders as understood by sphinx-build -b option. I am biased regarding LaTeX because I contribute to maintain it here at Sphinx, but I find it very useful indeed to be able to use make latexpdf. I understood as user that latexpdf isn't a Sphinx builder per see. I guess in your vision we will have to change outlook and consider we can pass it directly to sphinx-build as a "target/builder" ?

This one's different. Have you used tox yet? You can call tox like so to achieve the above.

tox -e py27 tests/test_build_latex.py

Yes, this is nice and the current docs (possibly as contributed by your commits, I did not check) do explain that clearly and even the ::test_foo_bar extra.

Now in my recollection I did want to use tox back in 2015 but I had at that time a number of issues:

• I was short on disk space,

• I was and still am using Conda distribution and it turns out that for a long time the virtualenv it shipped was broken,

• and finally I was a bit not too motivated into running tests for various Python and Docutils versions myself, considering that Sphinx was under Travis CI which did that job. Besides, my main contributions were to be into LaTeX, so I didn't think Python version would matter too much. Notice that (gott sei dank) we don't have testing on various kinds of TeX distributions! And it was out of the question to set up a "virtual box" which would download and install 4,5Gb of TeXLive...

tox was already promoted by Sphinx developer guide as the way to go, but I was really too lazy...

Now, I see it has great benefits. I have installed it, via the conda-forge channel, (it currently required a downgrade of conda itself, but that's not too serious) and I am quite willing to use it. Currently I find it adds some time overhead if I use it for testing. These steps:

GLOB sdist-make: /path/to/sphinx/setup.py
py36 create: /path/to/sphinx/.tox/py36
py36 installdeps: .[test,websupport]
py36 inst: /path/to/sphinx/.tox/dist/Sphinx-1.7.dev20171222.zip

take each time noticeable time. But this is good as it forces me to think more and test less often after trivial changes ;-)

On a pitnicky mode, I find the printing of dots ". . . . . . . " during execution of testing a bit boring and not self-explanatory. It is how it was until some months ago also with make test.

Now I find the output of current make test very nice and although almost identical with the one via the tox approach, I do find it better. But the main thing is that there is no overhead.

With conda I have the possibility to create environments with various Python and Docutils versions ; switching environments and running make test is faster than using tox for various environments configurations, currently on my aging mac os x. However varying Python between latest 2 and latest 3 and Docutils between at most two releases is about the maximum I am willing to do manually. Thus, I definitely favor nice tox which automatizes the whole thing. I hope however we do keep some possibility for less polished testing avoiding overhead.

Still with the underlying thinking that PR are tested via various services upstream here at Sphinx, which protects us againt lazy guys like me who are going to test only one combination (and often involuntarily forget the make style-check).

As an aside, I have a few more things on top of these two items planned but I'm not actually sure where the best place to discuss these plans is. The mailing list doesn't seem to be actively monitored and I don't know if RFE bugs are a things here. Perhaps you or @tk0miya might have some ideas?

I do subscribe via Gmane interface and Thunderbid to the sphinx-dev list and I am quite willing to participate to discussions of this type over there rather than here at Github which provides no such thing and limits discussions to PR's and issues.

[stephenfin]

@stephenfin thanks for the additional info. I will reply in a more detailed manner but I am currently on tight schedule and will only report one issue I currently have (at 2e04c2a):

Are there environment variables required for this target? If so, we need to explicitly call them out. Could you try adding replacing the current PASSENV setting in tox.ini with the below?

passenv = *

This should (if StackOverflow is to be believed) pass all configuration options through. If this is the case, we need to go find what options need to be passed and add them to that list.

[stephenfin]

I guess in your vision we will have to change outlook and consider we can pass it directly to sphinx-build as a "target/builder" ?

You read my mind 😄 I have added an additional builder called latexpdf (LaTeXPDFBuilder) which invokes the Makefile target on successful build (though we might skip the Makefile in the future). I'm currently trying to figure out what the -ja-suffixed builder is doing and will push my changes once I figure that out.

Yes, this is nice and the current docs (possibly as contributed by your commits, I did not check) do explain that clearly and even the ::test_foo_bar extra.

Good catch! I will add this shortly.

Currently I find it adds some time overhead if I use it for testing. These steps:

GLOB sdist-make: /path/to/sphinx/setup.py
py36 create: /path/to/sphinx/.tox/py36
py36 installdeps: .[test,websupport]
py36 inst: /path/to/sphinx/.tox/dist/Sphinx-1.7.dev20171222.zip

take each time noticeable time. But this is good as it forces me to think more and test less often after trivial changes ;-)

Yes, we should work on ways to improve this. Usually this should only rebuild if dependencies have changed (which doesn't happen often). If that's not the case, I need to fix something.

I hope however we do keep some possibility for less polished testing avoiding overhead.

Yes, as noted previously, if there are still people using this then I will not remove it. I just wanted to use tox for the 90% use case.

I do subscribe via Gmane interface and Thunderbid to the sphinx-dev list and I am quite willing to participate to discussions of this type over there rather than here at Github which provides no such thing and limits discussions to PR's and issues.

Good to know. I will likely post something over the Xmas period outlining my ideas in this area (I have lots, thanks to my experience with Sphinx from writing extensions and using it with pbr).

Also, good to know who the person to talk to about LaTeX issues is. I had never built LaTeX/PDF documents using Sphinx before last week so this is all new to me 😄.

[jfbu]

Could you try adding replacing the current PASSENV setting in tox.ini with the below?

passenv = *

This should (if StackOverflow is to be believed) pass all configuration options through. If this is the case, we need to go find what options need to be passed and add them to that list.

Confirmed!

============================= test session starts ==============================
platform darwin -- Python 3.6.4, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
rootdir: /Users/jfb/_gits/sphinx, inifile:
plugins: cov-2.5.1
collected 1 item                                                               

tests/test_build_texinfo.py .                                            [100%]

========================== slowest 25 test durations ===========================
2.10s call     tests/test_build_texinfo.py::test_texinfo
0.22s setup    tests/test_build_texinfo.py::test_texinfo
0.00s teardown tests/test_build_texinfo.py::test_texinfo
=========================== 1 passed in 2.35 seconds ===========================
___________________________________ summary ____________________________________
  py36: commands succeeded
  congratulations :)

[stephenfin]

Could you try adding replacing the current PASSENV setting in tox.ini with the below?

passenv = *

This should (if StackOverflow is to be believed) pass all configuration options through. If this is the case, we need to go find what options need to be passed and add them to that list.

Confirmed!

Excellent. If you could figure out what options are needed and add them to that, it would resolve the problem. I know you're busy though, so if you could provide a sanitized version of env (i.e. run env in your terminal) then I can try to debug this myself over the next few days.

Also, we should probably open a bug for this?

[jfbu]

I'm currently trying to figure out what the -ja-suffixed builder is doing and will push my changes once I figure that out.

The latexpdfja is not needed, when language is ja, the data put into latex build repertory (Makefile, latexmk config) is set-up for Japanese builds, and make all-pdf there will work.

[stephenfin]

I'm currently trying to figure out what the -ja-suffixed builder is doing and will push my changes once I figure that out.

The latexpdfja is not needed, when language is ja, the data put into latex build repertory (Makefile, latexmk config) is set-up for Japanese builds, and make all-pdf there will work.

Good to know. I should go and clean up some documentation so, as there are references to this in sphinx/make_mode.py, for example. Thanks for the info 👍

[jfbu]

I know you're busy though, so if you could provide a sanitized version of env (i.e. run env in your terminal) then I can try to debug this myself over the next few days.

Also, we should probably open a bug for this?

I will do it, I have to do my share here! but soup is boiling, so not now ;-)

[jfbu]

I removed recently mention of latexpdfja in the doc/man/sphinx-build.rst

[jfbu]

...hmm... wait about latexpdfja I will have to check the Windows situation, I can't test on Windows, and we ship the old latex Makefile with it, not using latexmk. I think setting language to 'ja' also turns make latexpdf into working for Japanese, but I am offhand not sure.

[stephenfin]

...hmm... wait about latexpdfja I will have to check the Windows situation, I can't test on Windows, and we ship the old latex Makefile with it, not using latexmk. I think setting language to 'ja' also turns make latexpdf into working for Japanese, but I am offhand not sure.

Yeah, the references I'm seeing as in sphinx/texinputs_win/Makefile_t. Guess I do need to investigate this further. Might be worth getting the latexmk-based Makefile working on Windows (and tested with Appveyor) at some point. I can also take a look at that.

[jfbu]

@stephenfin the needed environment variable to avoid the failed texinfo test is PERL5LIB.

Its setting in my environment is /sw/lib/perl5:/sw/lib/perl5/darwin, corresponding to a Fink install. Indeed I have

$ which -a makeinfo
/sw/bin/makeinfo
/usr/bin/makeinfo

$ /sw/bin/makeinfo --version
texi2any (GNU texinfo) 6.3
Copyright (C) 2016 Free Software Foundation, Inc.

vz the Apple shipped one

$ /usr/bin/makeinfo --version
makeinfo (GNU texinfo) 4.8

Copyright (C) 2004 Free Software Foundation, Inc.

With passenv = PERL5LIB in tox.ini, the texinfo test passes.

[tk0miya]

About latexpdfja, it is kept only for compatibility. Now it is almost same as latexpdf under language=ja setting. All Japanese users have used it for a long time. So I'd like to remove it in 2.0 release.

BTW, I also feel make-mode is ugly. So +1 for removing it. But I guess many projects uses it by default because it is enabled by default of Makefile since 1.6.
And I also +1 for support INI configuration.

[stephenfin]

@tk0miya Is there any other clarifications needed here?

[stephenfin]

@tk0miya Is this something we could consider for 1.7? I'm willing to do any rework necessary before the deadline.

[tk0miya]

I'm sorry. I don't have enough time to discuss this. So this would not merged before 1.7.
At this moment, I can't say okay to this idea. But it does not mean rejected.

TBH, I can understand your idea. But I feel it is not good to me. The basic concept of current sphinx-build is simple and making rich it through Makefile (or other build scripts). So I feel your proposal is strange.
But we introduced make-mode since 1.4 (before I joined to maintainers). So I've gotten confused a little.
So I need time to reconsider my opinion for new sphinx-build.

In addition, this is very big change for Sphinx. so all of maintainers should discuss for this. Now @shimizukawa ; the another maintainer of this project is very busy since last summer. So I'd like to wait his comeback.

[stephenfin]

I'm sorry. I don't have enough time to discuss this. So this would not merged before 1.7.
At this moment, I can't say okay to this idea. But it does not mean rejected.

That's no problem. I understand. I will pick this up again after 1.7 is released.

TBH, I can understand your idea. But I feel it is not good to me. The basic concept of current sphinx-build is simple and making rich it through Makefile (or other build scripts). So I feel your proposal is strange.
But we introduced make-mode since 1.4 (before I joined to maintainers). So I've gotten confused a little.
So I need time to reconsider my opinion for new sphinx-build.

Yes, the key thing I'm trying to remove here is the need to use the Makefile if you don't want to. The Makefile should be a nice-to-have and not a necessity. I do not want to break sphinx-build behaviour and if you call it the way you have always called it then nothing will change.

In addition, this is very big change for Sphinx.

I disagree 😉 I think this is a small change. Remember that nothing changes if you keep doing what you were doing - we don't break sphinx-build. Instead, we're simply making life easier for users by adding more sensible defaults to sphinx-build. We already do this for a lot of options using the default attribute so this seems like a good idea to me. In addition, we already try to guess sourcedir when using python setup.py build_sphinx so there is prior art here.

However, you have enough on your plate for now. Let's discuss this more once 1.8 window opens 👍 In the interim, feel free to ask me any questions you have about make-mode, including why I (and @jfbu, I think 😉) dislike it so much.

[shimizukawa]

(sorry for late replying..) IMO,

• -1 for guessing the document directory. Actually, I do not like that guessing the doc path on python setup.py build_sphinx.
• -1 to set the source directory path to conf.py. Convenience for users is important, but at this time I feel that the "guessing" code is long and maintenance is difficult. Above all, users will find it difficult to understand this 'guessing behavior'.
• make-mode succeeded in simplifying complex Makefile/make.bat. However, I think that make-mode is not an independent command, and incorporating it into the option of sphinx-build command makes the situation difficult. I think that this was an rough change to solve problems faster without largely changing existing mechanisms, and there were both advantages and disadvantages.

For me, it is good to redesign the configuration file location and startup process by making make-mode an independent command (like sphinx command?). But for now I do not have any specific ideas.

[stephenfin]

I'm picking this up again, but it's quite out of date and could do with a new approach, so I'm going to abandon this and start a new pull request.