argparse assertion failure with multiline metavars

[MaT1g3R]

BPO	32867
Nosy	@terryjreedy, @vadmium, @MaT1g3R, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-02-18.05:06:42.980>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'argparse assertion failure with multiline metavars'
updated_at = <Date 2021-12-15.19:58:48.287>
user = 'https://github.com/MaT1g3R'

bugs.python.org fields:

activity = <Date 2021-12-15.19:58:48.287>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-02-18.05:06:42.980>
creator = 'MaT1g3R'
dependencies = []
files = []
hgrepos = []
issue_num = 32867
keywords = []
message_count = 7.0
messages = ['312299', '312674', '312680', '312683', '312693', '314063', '408641']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'bethard', 'martin.panter', 'paul.j3', 'MaT1g3R', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'test needed'
status = 'pending'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32867'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[MaT1g3R]

If I run this script with -h
-----8<------------------

from argparse import ArgumentParser
mapping = ['123456', '12345', '12345', '123']
p = ArgumentParser('11111111111111')
p.add_argument('-v', '--verbose', help='verbose mode', action='store_true')
p.add_argument('targets', help='installation targets',  nargs='+', metavar='\n'.join(mapping))
p.parse_args()
---------8<

---

I get an error:
---------8<--------------------

Traceback (most recent call last):
  File "tmp.py", line 7, in <module>
    p.parse_args()
  File "/usr/lib/python3.6/argparse.py", line 1730, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python3.6/argparse.py", line 1762, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python3.6/argparse.py", line 1968, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/lib/python3.6/argparse.py", line 1908, in consume_optional
    take_action(action, args, option_string)
  File "/usr/lib/python3.6/argparse.py", line 1836, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python3.6/argparse.py", line 1020, in __call__
    parser.print_help()
  File "/usr/lib/python3.6/argparse.py", line 2362, in print_help
    self._print_message(self.format_help(), file)
  File "/usr/lib/python3.6/argparse.py", line 2346, in format_help
    return formatter.format_help()
  File "/usr/lib/python3.6/argparse.py", line 282, in format_help
    help = self._root_section.format_help()
  File "/usr/lib/python3.6/argparse.py", line 213, in format_help
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.6/argparse.py", line 213, in <listcomp>
    item_help = join([func(*args) for func, args in self.items])
  File "/usr/lib/python3.6/argparse.py", line 334, in _format_usage
    assert ' '.join(pos_parts) == pos_usage
AssertionError
-----8<
```------------------

[terryjreedy]

I don't understand the -h (display help part) but pasting code into repository 3.8 IDLE editor and running, I get same traceback.

Never having used argparse, I don't know if a multiline metavar is expected to work. Assuming so, can you suggest a fix?

[vadmium]

This looks like the same assertion failure as described in bpo-16360. Paul pointed to a patch in bpo-11874, so that may also be relevant.

However I agree that embedding newlines in a metavar doesn’t make much sense. What’s the use case?

[MaT1g3R]

I tried to include line breaks for listing options for a positional argument.

The default metavar for that is something like {opt1, opt2, op3},
however I wanted it to display the options on separate lines.

[terryjreedy]

If newlines are not permitted in metavars, the user should see
ValueError: newline not permitted in metavar

[paulj3]

I haven't seen anyone try to use \n in a metavar before, but other special characters like [] and () produce this assertion error. At this point the code is trying split the usage into 2 or more lines, because it's too long for one. It creates a usage for optionals and positionals separately.

In a clumsy way, it formats the whole usage string, and tries to split it into pieces so it can decide to split long lines. The 'assert' is used to make sure it has split the line into meaningful blocks. It splits with

r'\(.*?\)+|\[.*?\]+|\S+'

basically white space (including nl) and [] and () which are used to mark optional arguments and groups. So including any of these characters in text via metavar will screwup this split.

We could try to refine this splitting expression, but that feels like a never ending task as users become more inventive.

I suggested a major rewrite of this section, one that keeps the pieces a list, and joins them after deciding how many can fit on a line.

No one has, to my knowledge, come up with a comprehensive list of characters that will cause problems here.

argparse does provide a backup - a user provided usage string. That's not as nice as a automatically generated one, but if you have to have something special, that's the way to go. In the long run there's only so much that general purpose parser can do to accommodate special needs.

[iritkatriel]

It works for me on 3.11:

% ./python.exe b.py -h
usage: 11111111111111 [-h] [-v] 123456
12345
12345
123 [123456
12345
12345
123 ...]

positional arguments:
123456
12345
12345
123
installation targets

options:
-h, --help show this help message and exit
-v, --verbose verbose mode

[encukou]

This was fixed some time between Python 3.7.17 and 3.8.19.