python --help output is too long

[merwok]

BPO	46142
Nosy	@warsaw, @vstinner, @merwok, @serhiy-storchaka, @eryksun
PRs	gh-90300: split --help output into separate options #30331

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Linked PRs:

• gh-90300: Reformat the Python CLI help output #93415

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-12-20.23:43:15.950>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'python --help output is too long'
updated_at = <Date 2022-01-12.01:45:18.299>
user = 'https://github.com/merwok'

bugs.python.org fields:

activity = <Date 2022-01-12.01:45:18.299>
actor = 'eric.araujo'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2021-12-20.23:43:15.950>
creator = 'eric.araujo'
dependencies = []
files = []
hgrepos = []
issue_num = 46142
keywords = ['patch']
message_count = 15.0
messages = ['408981', '408982', '409449', '409450', '409470', '409483', '409738', '409890', '409928', '410052', '410203', '410207', '410238', '410239', '410378']
nosy_count = 5.0
nosy_names = ['barry', 'vstinner', 'eric.araujo', 'serhiy.storchaka', 'eryksun']
pr_nums = ['30331']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46142'
versions = ['Python 3.11']

Linked PRs

• gh-90300: Reformat the Python CLI help output #93415
• [3.12] gh-90300: Reformat the Python CLI help output (GH-93415) #115847
• gh-90300: Improve the Python CLI help output #115853
• [3.11] gh-90300: Reformat the Python CLI help output (GH-93415) (GH-115847) #116418
• gh-90300: Fix cmdline.rst #116721
• [3.12] gh-90300: Fix cmdline.rst (GH-116721) #116724
• [3.11] gh-90300: Fix cmdline.rst (GH-116721) #116725
• gh-90300: Sort the -X options and some envvars in the Python CLI help #116739
• gh-90300: Document equivalent -X options for envvars in the Python CLI help #116756
• gh-90300: Fix undocumented envvars in the Python CLI help #116765
• [3.12] gh-90300: Sort the -X options and some envvars in the Python CLI help (GH-116739) #116766
• [3.11] gh-90300: Sort the -X options and some envvars in the Python CLI help (GH-116739) (GH-116766) #116779
• [3.12] gh-90300: Document equivalent -X options for envvars in the Python CLI help (GH-116756) #116786
• [3.11] gh-90300: Document equivalent -X options for envvars in the Python CLI help (GH-116756) (GH-116786) #116796
• [3.12] gh-90300: Fix undocumented envvars in the Python CLI help (GH-116765) #116797
• [3.11] gh-90300: Fix undocumented envvars in the Python CLI help (GH-116765) (GH-116797) #116803
• [3.12] gh-90300: Improve the Python CLI help output (GH-115853) #117022
• [3.11] gh-90300: Improve the Python CLI help output (GH-115853) (GH-117022) #117034
• [3.12] gh-90300: Remove reference to PYTHON_FROZEN_MODULES in Python CLI help #117035

[merwok]

From Serhiy in https://mail.python.org/archives/list/python-dev@python.org/thread/QUUBM7DGSXYWBOLZNWOSCQUDALWJIYZF/ :

The output of "python -h" is 104 lines long now. It was only 51 lines in
3.6. 35% of it is about the -X option, and 30% about environment
variables. Also some lines in the -X option description are too long
(102 columns). Both topics are "advanced" and mostly interested for
debugging. I suggest to move them out of the main help output.  […]

Guido:

-X opt : implementation-specific option; use -X help to list options.

We could also see if we can put the help text for each of the supported -X
flags in the table defining these flags […]

GvR again:

--help-env […] prints info about env vars (new flag)

I would enjoy trying to make a patch around next week!

[merwok]

Forgot to quote -X help to print help about X options.

[merwok]

Do people think the man page should be shortened too?

[serhiy-storchaka]

I think that the man page should contain all details related to the CLI (and may be even some examples).

[merwok]

Question about the implementation:
I’ve found the parsing of command-line params in Python/initconfig.c and Python/preconfig.c.
Help is handled in initconfig, X options in preconfig.

A) could add a value to the right struct for "-X help", set it in preconfig, handle it in initconfig (by printing X options help and exiting, maybe reusing the existing print help function or writing a very similar one next to it)
B) or detect the option in preconfig, print and exit right there

Which option is best?

[merwok]

It seems that preconfig is only for some options that change fundamental behaviour (isolated mode, default encoding), so I should be able to detect and handle '-X help' in initconfig.

[vstinner]

initconfig.c parses all -X options. preconfig.c also checks for a few specific -X options which are also checked in initconfig.c.

My notes on parsing command line options:
https://pythondev.readthedocs.io/pyconfig.html#add-a-new-command-line-option

[merwok]

Victor said on github that he would prefer --help-xoptions to -X help. It would align with --help and --help-env, but not with other -X abc options. I think that --help-X may be better, as it uses the same form (X) rather than the non-word xoptions.

What do people think?

[vstinner]

"xoptions" name comes from sys._xoptions (Python) and PyConfig.xoptions (C). Oh, sys._xoptions is a private API... but it's mentioned in the -X documentation:
https://docs.python.org/3/using/cmdline.html#cmdoption-X

[merwok]

I hard forgotten this bit in the email thread:

Serhiy: What do you think about -hh (and maybe —help-full) printing full help?

Guidp: Is there enough of a use case for this to bother?

Barry: Maybe not. I’d say if it was easy to implement, why not, but if it’s a pain, don't bother.

The option parsing code is easy to edit, and the function would only need to call the other usage functions with a blank line inbetween. Should I add it?

[vstinner]

Serhiy: What do you think about -hh (and maybe —help-full) printing full help?

Do you know other projects which dump the full help into stdout when asking for the "full help"?

For me, the best CLI is "git help", "git help init", etc. "git help init" opens "man git-init". I don't know its behaviour on platforms without manpage support, like Windows.

[eryksun]

Do you know other projects which dump the full help into stdout
when asking for the "full help"?

ps --help lists sections "<simple|list|output|threads|misc|all>". The "--help all" output is all sections, but truly all help is only available via man ps.

For me, the best CLI is "git help", "git help init", etc. "git help
init" opens "man git-init". I don't know its behaviour on platforms
without manpage support, like Windows.

With Git for Windows, git help <command> opens an HTML formatted manual page in a web browser. It should use a new tab in an existing browser window.

[merwok]

For me, the best CLI is "git help", "git help init", etc.

I don’t think that prior art applies here, as python does not have subcommands. --help and --help-env are straightforward, -X help is a bit unusual but at least it’s part of the optional -X options, --help-full or -all would be way to preserve the behaviour of printing full usage with one option.

[merwok]

BTW on the PR I am asking for help with string formatting, to print an invalid X option (a wchar_t string) using the PyStatus API (needs char). Help from a proper C programmer would be appreciated :)

[merwok]

The PR now has --help-env, --help-xoptions and --help-all !

[merwok]

I have extracted some bugfixes to ticket #93217