argparse sometimes tracebacks when all options in a mutually exclusive group are suppressed #96310

dmach · 2022-08-26T13:31:05Z

Bug report

When a subparser contains a mutually exclusive group and the all options in the group are suppressed,
it sometimes errors out with a traceback. The crash depends on other options (their ordering and length) and terminal size.

Reproducer:

#!/usr/bin/python3

import argparse

parser = argparse.ArgumentParser()
commands = parser.add_subparsers(title="commands", dest="command")
cmd_foo = commands.add_parser("foo")
group = cmd_foo.add_mutually_exclusive_group()
group.add_argument('--verbose', action='store_true', help=argparse.SUPPRESS)
group.add_argument('--quiet', action='store_true', help=argparse.SUPPRESS)
cmd_foo.add_argument("--longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglong")

parser.parse_args()

$ python3.11 reproducer.py foo --help
Traceback (most recent call last):
  File ".../reproducer.py", line 13, in <module>
    parser.parse_args()
  File "/usr/lib64/python3.11/argparse.py", line 1862, in parse_args
    args, argv = self.parse_known_args(args, namespace)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 1895, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 2085, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 2062, in consume_positionals
    take_action(action, args)
  File "/usr/lib64/python3.11/argparse.py", line 1971, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib64/python3.11/argparse.py", line 1234, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 1895, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 2103, in _parse_known_args
    start_index = consume_optional(start_index)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 2043, in consume_optional
    take_action(action, args, option_string)
  File "/usr/lib64/python3.11/argparse.py", line 1971, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib64/python3.11/argparse.py", line 1112, in __call__
    parser.print_help()
  File "/usr/lib64/python3.11/argparse.py", line 2590, in print_help
    self._print_message(self.format_help(), file)
                        ^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 2574, in format_help
    return formatter.format_help()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 286, in format_help
    help = self._root_section.format_help()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 217, in format_help
    item_help = join([func(*args) for func, args in self.items])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 217, in <listcomp>
    item_help = join([func(*args) for func, args in self.items])
                      ^^^^^^^^^^^
  File "/usr/lib64/python3.11/argparse.py", line 341, in _format_usage
    assert ' '.join(opt_parts) == opt_usage
AssertionError

Please note that setting COLUMNS to a high number doesn't trigger the traceback:

COLUMNS=1000 python3.11 reproducer.py foo --help
usage: reproducer.py foo [-h]  [--longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglong LONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONG]

options:
  -h, --help            show this help message and exit
  --longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglong LONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONGLONG

Your environment

CPython versions tested on: 3.8.13, 3.10.6, 3.11.0b5
openSUSE Tumbleweed 20220823

Linked PRs

The text was updated successfully, but these errors were encountered:

…ive group are suppressed Reproducer depends on terminal size - the traceback occurs when there's an option long enough so the usage line doesn't fit the terminal width. Option order is also important for reproducibility. Excluding empty groups (with all options suppressed) from inserts fixes the problem.

…ually exclusive group are suppressed Reproducer depends on terminal size - the traceback occurs when there's an option long enough so the usage line doesn't fit the terminal width. Option order is also important for reproducibility. Excluding empty groups (with all options suppressed) from inserts fixes the problem.

Rationale ========= argparse performs a complex formatting of the usage for argument grouping and for line wrapping to fit the terminal width. This formatting has been a constant source of bugs for at least 10 years (see linked issues below) where defensive assertion errors are triggered or brackets and paranthesis are not properly handeled. Problem ======= The current implementation of argparse usage formatting relies on regular expressions to group arguments usage only to separate them again later with another set of regular expressions. This is a complex and error prone approach that caused all the issues linked below. Special casing certain argument formats has not solved the problem. The following are some of the most common issues: - empty `metavar` - mutually exclusive groups with `SUPPRESS`ed arguments - metavars with whitespace - metavars with brackets or paranthesis Solution ======== The following two comments summarize the solution: - python#82091 (comment) - python#77048 (comment) Mainly, the solution is to rewrite the usage formatting to avoid the group-then-separate approach. Instead, the usage parts are kept separate and only joined together at the end. This allows for a much simpler implementation that is easier to understand and maintain. It avoids the regular expressions approach and fixes the corresponding issues. This closes the following issues: - Closes python#62090 - Closes python#62549 - Closes python#77048 - Closes python#82091 - Closes python#89743 - Closes python#96310 - Closes python#98666 These PRs become obsolete: - Closes python#15372 - Closes python#96311

…exclusive group are suppressed (GH-96311) Reproducer depends on terminal size - the traceback occurs when there's an option long enough so the usage line doesn't fit the terminal width. Option order is also important for reproducibility. Excluding empty groups (with all options suppressed) from inserts fixes the problem.

…ually exclusive group are suppressed (pythonGH-96311) Reproducer depends on terminal size - the traceback occurs when there's an option long enough so the usage line doesn't fit the terminal width. Option order is also important for reproducibility. Excluding empty groups (with all options suppressed) from inserts fixes the problem. (cherry picked from commit 5f7df88) Co-authored-by: Daniel Mach <daniel.mach@suse.com>

…tually exclusive group are suppressed (GH-96311) (GH-115768) Reproducer depends on terminal size - the traceback occurs when there's an option long enough so the usage line doesn't fit the terminal width. Option order is also important for reproducibility. Excluding empty groups (with all options suppressed) from inserts fixes the problem. (cherry picked from commit 5f7df88) Co-authored-by: Daniel Mach <daniel.mach@suse.com>

…tually exclusive group are suppressed (GH-96311) (GH-115767) Reproducer depends on terminal size - the traceback occurs when there's an option long enough so the usage line doesn't fit the terminal width. Option order is also important for reproducibility. Excluding empty groups (with all options suppressed) from inserts fixes the problem. (cherry picked from commit 5f7df88) Co-authored-by: Daniel Mach <daniel.mach@suse.com>

…ually exclusive group are suppressed (pythonGH-96311) Reproducer depends on terminal size - the traceback occurs when there's an option long enough so the usage line doesn't fit the terminal width. Option order is also important for reproducibility. Excluding empty groups (with all options suppressed) from inserts fixes the problem.

Rationale ========= argparse performs a complex formatting of the usage for argument grouping and for line wrapping to fit the terminal width. This formatting has been a constant source of bugs for at least 10 years (see linked issues below) where defensive assertion errors are triggered or brackets and paranthesis are not properly handeled. Problem ======= The current implementation of argparse usage formatting relies on regular expressions to group arguments usage only to separate them again later with another set of regular expressions. This is a complex and error prone approach that caused all the issues linked below. Special casing certain argument formats has not solved the problem. The following are some of the most common issues: - empty `metavar` - mutually exclusive groups with `SUPPRESS`ed arguments - metavars with whitespace - metavars with brackets or paranthesis Solution ======== The following two comments summarize the solution: - #82091 (comment) - #77048 (comment) Mainly, the solution is to rewrite the usage formatting to avoid the group-then-separate approach. Instead, the usage parts are kept separate and only joined together at the end. This allows for a much simpler implementation that is easier to understand and maintain. It avoids the regular expressions approach and fixes the corresponding issues. This closes the following GitHub issues: - #62090 - #62549 - #77048 - #82091 - #89743 - #96310 - #98666 These PRs become obsolete: - #15372 - #96311

Rationale ========= argparse performs a complex formatting of the usage for argument grouping and for line wrapping to fit the terminal width. This formatting has been a constant source of bugs for at least 10 years (see linked issues below) where defensive assertion errors are triggered or brackets and paranthesis are not properly handeled. Problem ======= The current implementation of argparse usage formatting relies on regular expressions to group arguments usage only to separate them again later with another set of regular expressions. This is a complex and error prone approach that caused all the issues linked below. Special casing certain argument formats has not solved the problem. The following are some of the most common issues: - empty `metavar` - mutually exclusive groups with `SUPPRESS`ed arguments - metavars with whitespace - metavars with brackets or paranthesis Solution ======== The following two comments summarize the solution: - python#82091 (comment) - python#77048 (comment) Mainly, the solution is to rewrite the usage formatting to avoid the group-then-separate approach. Instead, the usage parts are kept separate and only joined together at the end. This allows for a much simpler implementation that is easier to understand and maintain. It avoids the regular expressions approach and fixes the corresponding issues. This closes the following GitHub issues: - python#62090 - python#62549 - python#77048 - python#82091 - python#89743 - python#96310 - python#98666 These PRs become obsolete: - python#15372 - python#96311

tarunsagarpandey · 2024-06-04T12:19:44Z

Traceback (most recent call last):
File "/usr/bin/weevely", line 104, in
arguments = parser.parse_args()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/argparse.py", line 1886, in parse_args
args, argv = self.parse_known_args(args, namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/argparse.py", line 1919, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/argparse.py", line 2143, in _parse_known_args
stop_index = consume_positionals(start_index)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/argparse.py", line 2099, in consume_positionals
take_action(action, args)
File "/usr/lib/python3.11/argparse.py", line 1995, in take_action
action(self, namespace, argument_values, option_string)
File "/usr/lib/python3.11/argparse.py", line 1258, in call
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/argparse.py", line 1919, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/argparse.py", line 2168, in parse_known_args
self.error(('the following arguments are required: %s') %
File "/usr/share/weevely/core/argparsers.py", line 56, in error
raise ArgparseError(message)
core.weexceptions.ArgparseError: the following arguments are required: url, password

How to fix this problem ??

…ually exclusive group are suppressed (pythonGH-96311) Reproducer depends on terminal size - the traceback occurs when there's an option long enough so the usage line doesn't fit the terminal width. Option order is also important for reproducibility. Excluding empty groups (with all options suppressed) from inserts fixes the problem.

dmach added the type-bug An unexpected behavior, bug, or error label Aug 26, 2022

bedevere-bot mentioned this issue Aug 26, 2022

gh-96310: Fix a traceback in argparse when all options in a mutually exclusive group are suppressed #96311

Merged

AlexWaygood added this to Argparse issues Aug 26, 2022

AlexWaygood added the stdlib Python modules in the Lib dir label Aug 26, 2022

hamdanal mentioned this issue May 28, 2023

gh-62090: Simplify argparse usage formatting #105039

Merged

terryjreedy moved this to Bugs in Argparse issues Aug 16, 2023

serhiy-storchaka closed this as completed Feb 21, 2024

github-project-automation bot moved this from Bugs to Doc issues in Argparse issues Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

argparse sometimes tracebacks when all options in a mutually exclusive group are suppressed #96310

argparse sometimes tracebacks when all options in a mutually exclusive group are suppressed #96310

dmach commented Aug 26, 2022 •

edited by bedevere-app bot

Loading

tarunsagarpandey commented Jun 4, 2024

argparse sometimes tracebacks when all options in a mutually exclusive group are suppressed #96310

argparse sometimes tracebacks when all options in a mutually exclusive group are suppressed #96310

Comments

dmach commented Aug 26, 2022 • edited by bedevere-app bot Loading

Bug report

Your environment

Linked PRs

tarunsagarpandey commented Jun 4, 2024

dmach commented Aug 26, 2022 •

edited by bedevere-app bot

Loading