Provide sphinx command and integrate sphinx commands

[shimizukawa]

Proposal

I propose to integrate multiple commands into ONE sphinx command and provide existing commands as subcommands.

In Japanese

Subject: sphinx コマンドを提供し、sphinxの各コマンドを統合する

私は、複数あるコマンドを1つのsphinxコマンドに統合し、既存のコマンドをサブコマンドとして提供することを提案します。

Background

We have several sphinx related commands as:

• make (by Makefile, make.bat)
• sphinx-quickstart
• sphinx-build
• sphinx-apidoc
• sphinx-autogen

Makefile and make.bat are provided to use the make command as a thin wrapper for sphinx-build. The make command is very flexible and easy to understand for people familiar with make. However, it was unfortunate for some unfamiliar persons to hinder proper reporting when something goes wrong (#1693, #2521), or not to work in some shell environments (#3145). We are also receiving proposals to make Makefile/make.bat highly functional which wants to make such an unkind part more kind, more convenient (e3ae24f, #3741). At the same time, there is also a proposal to discard the make command and replace it with a make command that can be written in Python (#3196).

It is the purpose of this proposal to finish these situations and make it a simple world.
For this reason, I propose the following changes to the following.

1. take over existing command features by introducing sphinx command.
2. Optionalization of Makefile / make.bat generating on quickstart.
3. Stepwise deprecate and remove the existing sphinx- * commands
4. Enhancement of sphinx command (Provide subcommand plugin feature etc)

In Japanese

`Makefile` と `make.bat` は `sphinx-build` の薄いラッパーとしてmakeコマンドを使えるように提供されています。makeコマンドはmakeを使い慣れている人にとってはとても柔軟で分かりやすいものです。しかし、不慣れな人にとってはなにか問題が発生した場合に適切にレポートする妨げとなったり (#1693, #2521)、一部のシェル環境では動作しないといった不親切な面もありました(#3145)。こういった不親切な部分をより親切にしたい、より便利にしたいというMakefile/make.batを高機能化する提案もいただいています(e3ae24f, #3741)。同時に、makeコマンドを捨ててPythonで書けるmake系コマンドに置き換える提案もあります (#3196)。

こういった状況を終わらせ、シンプルな世界にするのがこの提案の目的です。
このため、大きく以下の変更を提案します。

1. sphinx コマンドの導入による既存コマンドの引き継ぎ
2. Makefile/make.bat 提供をオプション化
3. 段階的に既存の sphinx-* コマンドの廃止と削除
4. sphinx コマンドの高機能化（サブコマンドプラグイン機能を提供等）

Step1: taking over existing command features by introducing sphinx command.

• The sphinx command has global options such as --version, --verbose
• sphinx quickstart invoke sphinx-quickstart feature.
• sphinx build html invoke sphinx-build -M html feature
• such sphinx autodoc invoke sphinx-apidoc and/or sphinx-autogen feature

Step2: Optionalization of Makefile / make.bat generating on quickstart.

• Makefile / make.bat will not be created by default.
• Users can generate new Makefile/make.bat by option that invoke sphinx command instead of sphinx-build
• Users can generate traditional Makefile/make.bat by option that will be removed on Step3 because it invoke sphinx-build

Step3: Stepwise deprecate and remove the existing sphinx- * commands

• deprecating all sphinx- commands
• removing all sphinx- commands

Step4: enhancement of sphinx command

• command parameters from environment variables
• pluggable subcommands; ex. sphinx-intl will be able to provide sphinx intl sub command.
• multiple command target like: make clean html

Timeline

Maybe..

• Sphinx-2.0.0: Step1 providing sphinx command
• Sphinx-3.0.0: Step2, Step3, Step4
• Sphinx-4.0.0: Step3, Step4

Other related issues/PRs

• Add unified 'sphinx' command (WIP!) #6938
• [RFC] Simplified 'sphinx-build' #6939

[tk0miya]

+1

[jfbu]

+1

[timhoffm]

+1

[mgeier]

👍 Very good idea!

I've always hated the fact that two different platform-specific scripts (Makefile and make.bat) are the default way to build something that's entirely written in platform-independent Python!

I think having a single command sphinx with multiple sub-commands is a good idea.

However, please also consider the use-case when the sphinx module is called directly.
I'm mostly using Sphinx like this:

python3 -m sphinx <source-dir> <build-dir>

This works great if you have multiple Python versions and/or environments.
I'm using that for all Python-based commands, e.g. also for

python3 -m pip install sphinx

... and for

python3 -m jupyterlab

Instead of sphinx-quickstart, I'm using:

python3 -m sphinx.cmd.quickstart

I'm using many Python-based tools, and this way I never have to remember the names of the "executables" (because sometimes they are different than the module name, which is annoying) and I don't have to worry if I've set up the right $PATH. As long as python3 is available, everything just works.

If you change the main sphinx command to be used e.g. like this:

sphinx build ...

... please make it also possible to do exactly the same by running this:

python3 -m sphinx build ...

On top of all this, it would be great to have a similar interface to use Sphinx from a Python script.

Some time ago, I've tried to use the sphinx.application.Sphinx class to run the Sphinx build process, but it got very complicated. In the end I gave up and wrote a quick shell script, which was much simpler (but of course not platform-independent).

Ideally, the sphinx command would simply call __main__.py (or at least do exactly the same thing), which itself would process the command line arguments and then call a public Python API to do the actual work.

As users, we should have the free choice of using either of the 3 methods, and ideally all 3 methods should look similar and work in a similar way.

[shimizukawa]

relative issees/PRs

• How about a PowerShell based 'make' file? #3145, Implemented a first set of Sphinx targets in a PowerShell based "makefile". #3153 PowerShell based 'make' file
• Enabling multiple make targets and sphinx-build options in Windows make.bat (non-PowerShell) #3741 Enabling multiple make targets and sphinx-build options in Windows make.bat
• Replace Makefile/make.bat with a Python script? #3196 Replace Makefile/make.bat with a Python script?

[astromatt]

Idea

• Makefile requires lots of additional dependencies. This is particular a problem on Alpine Linux (docker container).
• Standalone sphinx-build works very well without any external dependency.
• Stop generating Makefile for new projects and move all sphinx-build arguments to conf.py.
• Start promoting use of sphinx build -c conf.py.
• If flag -c <FILE> is not specified, sphinx build should look in current directory for conf file.
• All sphinxopts, sourcedir and builddir should be in conf.py.
• sphinxopts should be configuration variables not just simply "list of opts to pass to sphinx build"
• This will also give opportunity to have multiple configuration in one project.

[tk0miya]

The goal of this issue is "Makefile-less sphinx project". So I agreed for @astromatt 's idea basically.

But we need to discuss about some points.

Stop generating Makefile for new projects and move all sphinx-build arguments to conf.py.

I suppose large amount of sphinx users use Makefile as an command line interface of Sphinx. So they will be surprised if Makefile is not generated suddenly. IMO, it would better to set migration period for a while after adding sphinx build command.

All sphinxopts, sourcedir and builddir should be in conf.py.

I don't think so. I think sourcedir and builddir are runtime options (some kinds of other options are also). For example, -v (verbosity) is not good to add configuration file. I can agree to make them optional and give better default settings.

[stephenfin]

The goal of this issue is "Makefile-less sphinx project". So I agreed for @astromatt 's idea basically.

But we need to discuss about some points.

Stop generating Makefile for new projects and move all sphinx-build arguments to conf.py.

I suppose large amount of sphinx users use Makefile as an command line interface of Sphinx. So they will be surprised if Makefile is not generated suddenly. IMO, it would better to set migration period for a while after adding sphinx build command.

Agreed. I think there should be a deprecation period for this. Perhaps for the new, unified sphinx init command, we could default generation of a Makefile to False but leave it as True for the old sphinx-quickstart command?

All sphinxopts, sourcedir and builddir should be in conf.py.

I don't think so. I think sourcedir and builddir are runtime options (some kinds of other options are also). For example, -v (verbosity) is not good to add configuration file. I can agree to make them optional and give better default settings.

We're discussing this at #6939, but I think builddir isn't really a runtime option and is usually static for a given project so it does make sense to be in conf.py. I'm okay with sourcedir being configured from the command line though (so long as it defaults to .)

[tk0miya]

Perhaps for the new, unified sphinx init command, we could default generation of a Makefile to False but leave it as True for the old sphinx-quickstart command?

+1!

[asmeurer]

All sphinxopts, sourcedir and builddir should be in conf.py.

I don't think so. I think sourcedir and builddir are runtime options (some kinds of other options are also). For example, -v (verbosity) is not good to add configuration file. I can agree to make them optional and give better default settings.

We're discussing this at #6939, but I think builddir isn't really a runtime option and is usually static for a given project so it does make sense to be in conf.py. I'm okay with sourcedir being configured from the command line though (so long as it defaults to .)

I would second this and say that all options should be configurable in conf.py (of course, passing options directly to the command line would override the configured defaults).

In general, the sphinx command needs to be able to build the documentation with just one command, e.g.,

sphinx build

not

sphinx build html

or

sphinx build html --output-dir=_build/ --source-dir=.

If the user is required to type all these things, projects will just continue to use a Makefile to wrap it. Right now, I can build the docs in just about any project that uses Sphinx with

make html

which is less than 10 characters.

[Secrus]

What is the status of this? There is sphinx._cli now, is there any roadmap on what is to be done to make this a complete feature? Anything people can contribute?