argparse: Allow the use of -- to break out of nargs and into subparser

[elsdoerfer]

BPO	9571
Nosy	@merwok, @bitdancer
Files	issue9571_1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-08-11.19:21:25.755>
labels = ['type-feature', 'library']
title = 'argparse: Allow the use of -- to break out of nargs and into subparser'
updated_at = <Date 2014-07-07.17:12:55.826>
user = 'https://bugs.python.org/elsdoerfer'

bugs.python.org fields:

activity = <Date 2014-07-07.17:12:55.826>
actor = 'paul.j3'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-08-11.19:21:25.755>
creator = 'elsdoerfer'
dependencies = []
files = ['35886']
hgrepos = []
issue_num = 9571
keywords = ['patch']
message_count = 5.0
messages = ['113616', '113647', '113668', '222427', '222478']
nosy_count = 5.0
nosy_names = ['bethard', 'eric.araujo', 'r.david.murray', 'elsdoerfer', 'paul.j3']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9571'
versions = ['Python 3.5']

Linked PRs

• gh-53780: [argparse] ignore the first "--" (double dash) between an op… #124275
• [3.12] gh-53780: Ignore the first "--" (double dash) between an option and command in argparse (GH-124275) #124420
• [3.13] gh-53780: Ignore the first "--" (double dash) between an option and command in argparse (GH-124275) #125073

[elsdoerfer]

argparse already seems to support -- to indicate that what follows are positional arguments. However, I would like to parse something like:

./script.py --ignore one two -- COMMAND

I.e., --ignore is an nargs='+' argument, and I need a way to break out of --ignore and have argparse consider what follows on it's own merits. If COMMAND in the above example refers to a subparser, this won't work:

error: invalid choice: '--' (choose from 'command1', 'command2', 'command3')

I'm not sure what's the best solution here. Allowing -- here would change the semantics of forcing everything that follows to be positional arguments, since the subparser might have flags. I'm not sure if that is what is required by Unix conventions, but if not, then I think it makes sense to allow -- to be followed by a subparser.

[bitdancer]

It looks like, if accepted, this would be a feature request,so I'm marking it as such and setting versions to 3.2. You'd have to provide a patch pretty soon to get it in to 3.2, though.

However, I'm guessing that this is something better off implemented via subclassing in your code, since it sounds like a fairly unusual command pattern. (I've only ever seen options after a -- in a unix comand when those options were being passed unparsed to some *other* command that the first command was a wrapper for.)

I've added Steven as nosy, we'll see what he thinks.

[bethard]

This is closely related to bpo-9338. The parser should know that your command line requires at least the COMMAND argument, so it should stop parsing in time for that. However, in the case of subcommands, even if we solved bpo-9338, you would still get the behavior that

./script.py --ignore one two COMMAND arg1 arg2

would get parsed as "arg2" being the command. So I guess there still ought to be a way to tell argparse to stop parsing nargs='+' optionals.

Seems like there's also a bug in the current behavior - you should get an error saying that no command was given, not an error saying you issued the command "--".

[paulj3]

In elsdoerfer's example, the '--' effectively ends the argument list for '--ignore'. '--' are not allowed in the arguments of an optional (see the end of '_get_nargs_pattern()').

Rather the problem is at the start of _get_values()

        if action.nargs not in [PARSER, REMAINDER]:
            arg_strings = [s for s in arg_strings if s != '--']

'--' is not stripped out of the 'parser' input, hence the error message:

error: invalid choice: '--'

You can see this by replacing the subparsers with an argument with 'nargs=PARSER'. The argument will get ['--','COMMAND',...].

http://bugs.python.org/issue13922 tries to rework how '--' are handled. Ideally only the current '--' should be removed, leaving the rest to be handled by the subparser (or whoever else gets the strings).

Looks like the 13922 fix needs another fix, one that removes '--' if it is the 1st string for a REMAINDER or PARSER argument.

With that fix:

./script.py --ignore one two -- COMMAND arg1 arg2

should work, assigning ['one','two'] to 'ignore', and ['arg1','arg2'] to 'COMMAND's positional.

(I've tested this in a development version with many other changes. I'll try to write a simpler patch.)

----------------

If a 'end of a list' flag is still needed (as between 2 * positionals), a 'counter' or 'store_true' optional could be used. Or a new action class that doesn't write anything to the namespace could be written.

[paulj3]

This patch modifies that '--' handling from 13922, removing '--' from PARSER and REMAINDER if it is the first string:

        if action.nargs not in [PARSER, REMAINDER]:
            try:
                arg_strings.remove('--')
            except ValueError:
                pass
        else:
            # remove '--' if it is the first string, issue9571
            if arg_strings and arg_strings[0] == '--':
                arg_strings = arg_strings[1:]

Doing this for PARSER addresses this issue, and should not have any backward compatibility issues, since starting a 'A...' list with '--' does not make sense.

The argument for doing this with REMAINDER is a bit weaker. I can't think of why a user would expect or want an initial '--', but someone might have written code to compensate for this flaw.

test_argparse has 2 new tests, added at the end of the subparsers class. The one for subparser has to contend with a bug that makes subparsers optional (http://bugs.python.org/issue9253). The REMAINDER one is clearly related, though it doesn't fit the test class theme.

[serhiy-storchaka]

Currently positional arguments consume the following "--", but options do not. The first "--" is ignored in positional arguments with nargs different from PARSER and REMAINDER, so this usually do not cause issues. But we have the following inconsistency:
... --option optarg -- pasarg command ... -- okay
... --option optarg posarg -- command ... -- okay
... posarg --option optarg command ... -- okay
... posarg --option optarg -- command ... -- fail

So in case of nargs=PARSER we should ignore the first "--" before the command. But not after the command, where it is a part of subparser arguments.

There were several bugs in handling "--", and this is just one of them.

[serhiy-storchaka]

Creating a backport for 3.13 requires merging #124266.

[savannahostrowski]

It looks like the above PR and backports were merged, so this seems closable.