Replace Makefile/make.bat with a Python script?

[brechtm]

A Python script would make it easier to support the various platforms (#3145, #3194). It would also mean that it can be used in the same way on all platforms, which is handy in a tox.ini, for example.

[brechtm]

Here is a (work-in-progress) script I wrote to replace the makefile:

#!/usr/bin/env python

# Build script for Sphinx documentation

import os
import shlex
import shutil
import subprocess
import sys

from collections import OrderedDict


# You can set these variables from the command line.
SPHINXOPTS = os.getenv('SPHINXOPTS', '')
SPHINXBUILD = os.getenv('SPHINXBUILD', 'sphinx-build')
PAPER = os.getenv('PAPER', None)
BUILDDIR = os.getenv('BUILDDIR', '_build')


TARGETS = OrderedDict()


def target(function):
    TARGETS[function.__name__] = function
    return function


# User-friendly check for sphinx-build
def check_sphinx_build():
    with open(os.devnull, 'w') as devnull:
        try:
            if subprocess.call([SPHINXBUILD, '--version'],
                               stdout=devnull, stderr=devnull) == 0:
                return
        except FileNotFoundError:
            pass
    print("The '{0}' command was not found. Make sure you have Sphinx "
          "installed, then set the SPHINXBUILD environment variable "
          "to point to the full path of the '{0}' executable. "
          "Alternatively you can add the directory with the "
          "executable to your PATH. If you don't have Sphinx "
          "installed, grab it from http://sphinx-doc.org/)"
          .format(SPHINXBUILD))
    sys.exit(1)


@target
def all():
    """the default target"""
    return html()


@target
def clean():
    """remove the build directory"""
    shutil.rmtree(BUILDDIR, ignore_errors=True)


def build(builder, success_msg=None, extra_opts=None, outdir=None,
          doctrees=True):
    builddir = os.path.join(BUILDDIR, outdir or builder)
    command = [SPHINXBUILD, '-b', builder]
    if doctrees:
        command.extend(['-d', os.path.join(BUILDDIR, 'doctrees')])
    if extra_opts:
        command.extend(extra_opts)
    command.extend(shlex.split(SPHINXOPTS))
    command.extend(['.', builddir])
    print(' '.join(command))
    if subprocess.call(command) == 0:
        print('Build finished. ' + success_msg.format(builddir))


@target
def html():
    """make standalone HTML files"""
    return build('html', 'The HTML pages are in {}.')


@target
def dirhtml():
    """make HTML files named index.html in directories"""
    return build('dirhtml', 'The HTML pages are in {}')


@target
def singlehtml():
    """make a single large HTML file"""
    return build('singlehtml', 'The HTML page is in {}.')


@target
def pickle():
    """make pickle files"""
    return build('pickle', 'Now you can process the pickle files.')


@target
def json():
    """make JSON files"""
    return build('json', 'Now you can process the JSON files.')


@target
def htmlhelp():
    """make HTML files and a HTML help project"""
    return build('htmlhelp', 'Now you can run HTML Help Workshop with the '
                             '.hhp project file in {}.')


@target
def qthelp():
    """make HTML files and a qthelp project"""
    return build('qthelp', 'Now you can run "qcollectiongenerator" with the '
                           '.qhcp project file in {0}, like this: \n'
                           '# qcollectiongenerator {0}/RinohType.qhcp\n'
                           'To view the help file:\n'
                           '# assistant -collectionFile {0}/RinohType.qhc')


@target
def devhelp():
    """make HTML files and a Devhelp project"""
    return build('devhelp', 'To view the help file:\n'
                            '# mkdir -p $HOME/.local/share/devhelp/RinohType\n'
                            '# ln -s {} $HOME/.local/share/devhelp/RinohType\n'
                            '# devhelp')


@target
def epub():
    """make an epub"""
    return build('epub', 'The epub file is in {}.')


@target
def rinoh():
    """make a PDF using rinohtype"""
    return build('rinoh', 'The PDF file is in {}.')


@target
def latex():
    """make LaTeX files, you can set PAPER=a4 or PAPER=letter"""
    extra_opts = ['-D', 'latex_paper_size={}'.format(PAPER)] if PAPER else None
    return build('latex', 'The LaTeX files are in {}.\n'
                          "Run 'make' in that directory to run these through "
                          "(pdf)latex (use the 'latexpdf' target to do that "
                          "automatically).", extra_opts)


@target
def latexpdf():
    """make LaTeX files and run them through pdflatex"""
    rc = latex()
    print('Running LaTeX files through pdflatex...')
    builddir = os.path.join(BUILDDIR, 'latex')
    subprocess.call(['make', '-C', builddir, 'all-pdf'])
    print('pdflatex finished; the PDF files are in {}.'.format(builddir))


@target
def latexpdfja():
    """make LaTeX files and run them through platex/dvipdfmx"""
    rc = latex()
    print('Running LaTeX files through platex and dvipdfmx...')
    builddir = os.path.join(BUILDDIR, 'latex')
    subprocess.call(['make', '-C', builddir, 'all-pdf-ja'])
    print('pdflatex finished; the PDF files are in {}.'.format(builddir))


@target
def text():
    """make text files"""
    return build('text', 'The text files are in {}.')


@target
def man():
    """make manual pages"""
    return build('man', 'The manual pages are in {}.')


@target
def texinfo():
    """make Texinfo files"""
    return build('texinfo', 'The Texinfo files are in {}.\n'
                            "Run 'make' in that directory to run these "
                            "through makeinfo (use the 'info' target to do "
                            "that automatically).")


@target
def info():
    """make Texinfo files and run them through makeinfo"""
    rc = texinfo()
    print('Running Texinfo files through makeinfo...')
    builddir = os.path.join(BUILDDIR, 'texinfo')
    subprocess.call(['make', '-C', builddir, 'info'])
    print('makeinfo finished; the Info files are in {}.'.format(builddir))


@target
def gettext():
    """make PO message catalogs"""
    return build('gettext', 'The message catalogs are in {}.', outdir='locale',
                 doctrees=False)


@target
def changes():
    """make an overview of all changed/added/deprecated items"""
    return build('changes', 'The overview file is in {}.')

@target
def xml():
    """make Docutils-native XML files"""
    return build('xml', 'The XML files are in {}.')


@target
def pseudoxml():
    """make pseudoxml-XML files for display purposes"""
    return build('pseudoxml', 'The pseudo-XML files are in {}.')


@target
def linkcheck():
    """check all external links for integrity"""
    return build('linkcheck', 'Look for any errors in the above output or in '
                              '{}/output.txt.')


@target
def doctest():
    """run all doctests embedded in the documentation (if enabled)"""
    return build('doctest', 'Look at the results in {}/output.txt.')


@target
def help():
    """List all targets"""
    print("Please use '{} <target>' where <target> is one of"
          .format(sys.argv[0]))
    width = max(len(name) for name in TARGETS)
    for name, target in TARGETS.items():
        print('  {name:{width}} {descr}'.format(name=name, width=width,
                                                descr=target.__doc__))


if __name__ == '__main__':
    check_sphinx_build()
    args = sys.argv[1:] or ['all']
    for arg in args:
        TARGETS[arg]()

Like the makefile, this still includes the project name (RinohType) here and there, which should be moved to a variable.

Some targets (latexpdf, info) also call make on a generated makefile. I think these should also be replaced with Python scripts.

[tk0miya]

The Makefile is well known and commonly used command. So to provide the Makefile helps the users who don't have python knowledges.
So I would like to keep providing Makefile for users.

BTW, better command line tools are welcome. Since 1.4, we provide make-mode (-M option) to sphinx-build. It tries to build document from python side. But I feel sphinx-build is still hard to use barely. So +1 for improving the command.

[shimizukawa]

+1 to keep providing Makefile/make.bat.
+1 for improving the sphinx-build command as tk0miya mentioned.

Just idea:

$ sphinx build html
$ sphinx quickstart
$ sphinx apidoc

[brechtm]

I didn't know about "make mode". This is even better than a Makefile-esque Python script!

I agree that there is room for improvement though:

• sphinx-build -h does not list the -M option
• the documentation only mentions make mode on the sphinx-quickstart page
• sphinx-build -M prints Error: at least 3 arguments (builder, source dir, build dir) are required.
• sphinx-build -M help prints the same, requiring the source and build directories to be specified as well, which makes no sense

And how can one add a "make target" for another builder? It would be nice if sphinx-build -M would accept and list (on -M help) all available (installed) builders). Each builder could supply the string to be displayed on -M help. Builders can be discovered using entry points as suggested in #2803. What do you think?

This wouldn't cover all make targets (such as latexpdf), which add another step after building with Sphinx. Would it make sense to add separate builders for this which extend the builder they depend on?

[brechtm]

Another thought: why not specify the source and build directories in conf.py? I think that would make sense. This would make them optional arguments to sphinx-build (still allowing to override).

[tk0miya]

Sorry, I don't know much about make-mode. But it looks not extensible with APIs.

Builders can be discovered using entry points as suggested in #2803. What do you think?

Personally, I agree that.
But, I heard the proposal using entry points was rejected once (before I joined to maintainers).
@shimizukawa Do you know about that?

Another thought: why not specify the source and build directories in conf.py? I think that would make sense. This would make them optional arguments to sphinx-build (still allowing to override).

If the environment is different, the paths are also different.
So I think it is not a configuration variable.

BTW, +1 for optional argument. It would be useful.

[brechtm]

If the environment is different, the paths are also different.
So I think it is not a configuration variable.

I'm not sure what you mean with "environment" here. In conf.py, I would specify the build and source directories passed to sphinx-build -M. The build directory is the "top-level" build directory (such as _build/), not the build directory specific to a particular builder.

[tk0miya]

The build directory is the "top-level" build directory (such as _build/), not the build directory specific to a particular builder.

The "top-level" build directory you said is not used in Sphinx application.
Sphinx application only uses an absolute build path to specific builder like /Users/tkomiya/work/tmp/doc/_build/html.
The path is different if users, host, OS and so on are different. This is "environment" I said.

Anyway, the Sphinx object requires both paths to instantiate. And conf.py are read after Sphinx app invoked.
So it is hard to specify the paths from conf.py during Sphinx 1.x series.

[brechtm]

@tk0miya Thanks for the explanation. Now I understand the problem.

But wouldn't it be possible for sphinx-build to also read conf.py, only extracting the values for the source and build directories? The Sphinx application can ignore them.

[tk0miya]

Yes, it's possible. But I can't still understand the advantage of conf.py.
Could you tell me the worth of your way?

[brechtm]

Currently they are specified in the Makefile, but I think the source and build directory paths belong in conf.py, just like other paths such as templates_path and html_static_path. In fact, the build path is currently duplicated in conf.py in the exclude_patterns variable.

If they are specified in conf.py, sphinx-build (in make mode) can retrieve them from there (but still allow overriding them). This would simplify sphinx-build -M invocation significantly:

sphinx-build -M html

Or, similar to @shimizukawa's suggestion above:

sphinx build html
# or perhaps
sphinx make html

[bskinn]

@tk0miya -- if the -M option is supposed to be part of the public command line API, it is not documented as such in the current stable documentation.

[tk0miya]

@bskinn Oh, I'd not noticed that. Could you file it as an issue?
Of course, PR is always welcome :-)

[ben-spiller]

+1 for the idea of having a Python script to orchestrate the documentation build process, i.e. calling the equivalent of sphinx-build, sphinx-apidoc, etc - I think there are big advantages to this approach!

Not everyone has make installed (especially on Windows; it'd be nice if pip-installing sphinx itself was sufficient), and many developers in our community will know Python much better than make (nb: even for non-Python projects the conf.py is in Python so requires some basic Python knowledge). Doc builds should run the same way on all supported platforms, and OS-specific shell scripts work against that goal - making people manually copy sphinx-build command line args and apidoc invocations between a windows make.bat script and unix makefile script is fragile.

We wouldn't have to remove the make.py/Makefile stuff for those who genuinely prefer to do it that way, but having the sphinx-quickstart generate a Python script in addition would be really nice and imho provide a cleaner and more cross-platform alternative. It would also give us a good place to give users a helping hand integrating typical usage of sphinx-apidoc for those who need it.

Generating a make.bat-style Python orchestration script (call it make.py perhaps?) would be more flexible than just improving the sphinx-build script arguments, since it'd give you a perfect place (and language!) to write custom logic if needed, e.g. if you need to copy or filter files before building the doc, or add some complex apidoc rules for specific packages, or somehow produce different doc sets for different purposes etc (internal vs external-facing doc) - all stuff that's way easier in lovely Python.

[tk0miya]

@ben-spiller What is different between the script and sphinx-build command? I still don't understand what is expected. Makefile and make.bat is very thin wrapper script. So you can use it directly in both Linux and Windows. What feature does the script have? I think we've already had -M option. Please let me know what we should do.

[ben-spiller]

Hi thanks for the reply. The make.py script I'm suggesting would be an alternative to make.bat/Makefile (not an alternative to sphinx-build - people can alread invoke the raw tool using the script provided with the distribution).

The way sphinx is architected (at least as far as I can tell) most projects would need somewhere to put some additional arguments and/or commands that are specific to their build, for example:

• sphinx-build command line args such as the sourcedir, builddir and builder name(s) used for the documentation current project - e.g. perhaps I need to run it twice, once with "html" and once with another builder type like pdf
• sphinx-apidoc/apigen command(s) to be executed before sphinx-build to generate the necessary rst files

Right now, where would be the recommended place to put those? After reading the doc I wasn't sure what the recommended way to orchestrate launching the various processes would be. Of course I could do it manually by writing those args out every time or put together a readme for our developers saying "run apidoc with these args from this directory, then run sphinx-build with these args etc", or copy the required commands into both make.bat and Makefile. But a pure python launcher could be a neater solution to that (I'm thinking just a few lines, very similar to make.bat, perhaps with commented out tetx to show how you'd add sphinx-apidoc).

I guess the other way of addressing the underlying requirement would be to add more automation options to the main build so that it's not necessary to run commands like apidoc/apigen separatly before invoking sphinx-build. That would be ideal but potentially a bigger ask.

[tk0miya]

I think that is a responsibility of task runner. AFAIK there are many kind of pure python task runners. So, IMO, no reason to implement it again. How about fabric, invoke and so on?

[ben-spiller]

But if using invoke/fabric to run sphinx-build/apidoc were the best/recommended approach then why does sphinx provide make.bat and Makefile at all? Whatever is provided in-the-box is going to encourage new users down a particular path, for good or ill.

This isn't a request for some kind of 'generic' task runner integration with sphinx, more about
a) discouraging people to start using and customizing OS-specific shell scripts like make.bat for orchestrating sphinx build processes such as apidoc/autogen and
b) giving people a helping hand to get started with sphinx-specific build steps such as running apidoc/apigen in a way that's cross-platform and likely to scale well when projects get more complicated

I still think make.py would be more cross-platform and pythonic than make.bat/Makefile, but actually the more I think about it the best solution to my underlying need is to have more powerful ways to auto-generate rst's without needing a separate invocation of sphinx-apigen/autogen, since it's the complexity of those extra processes which is really driving my desire for a better cross-platform invocation mechanism to invoken them. If all that make.bat/Makefile ever do is invoke sphinx-build, it's easy enough to just ignore them and do that manually and there's not such a compelling reason to need a pure-python implementation of the same.

(e.g. I'm thinking of a core capability to generate rst's from .py files a bit like what autosummary_generate is trying to do, but not requiring manual generation of the rst's containing the initial autosummary, or something along the lines of https://autoapi.readthedocs.io/ or https://sphinx-automodapi.readthedocs.io/en/latest/ which I only just found after quite a while trying to get something working with the core sphinx packages). I might create a separate enhancement once I have a more specific/actionable idea, but it's clearly a bit of a separate discussion to this issue. :) Thanks

[astrojuanlu]

Hi! While investigating Sphinx "make mode" -M (it's not clear to me, by reading the documentation, what does it offer on top of -b) I found this issue, and I think most of @brechtm comment is still relevant. Repeating here:

• sphinx-build -h does not list the -M option
• the documentation only mentions make mode on the sphinx-quickstart page (now it's mentioned, at least, in sphinx-build reference as well)
  sphinx-build -M prints Error: at least 3 arguments (builder, source dir, build dir) are required.
sphinx-build -M help prints the same, requiring the source and build directories to be specified as well, which makes no sense

A few questions to @tk0miya and other Sphinx maintainers:

• In your opinion, should we recommend sphinx-build -M html . _build as the "best practice" instead of relying on Make, as this issue suggests?
• If so, would you accept pull requests addressing @brechtm points about it?
• And finally, do you think we could stop requiring the third argument, and use {source dir}/_build as default?

[astrojuanlu]

Also, it's unclear to me if make mode -M does not accept extra options (like -W) on purpose, and why.

[tk0miya]

Now #6938 was posted as a successor of the sphinx-build command. But it has been perfectly stalled.

Note: I also don't know the -M option well. It was added before I joined this project. So I can't answer you its purpose and why.

[astrojuanlu]

Thanks for the quick response @tk0miya! Will have a look at those discussions.

[AA-Turner]

Closing in favour of #5618.

A