Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fallbackValue is not applied for vararg multi-value options (was: Support fallbackValue with arity="0..*") #1904

Closed
sewe opened this issue Dec 23, 2022 · 10 comments
Labels
theme: parser An issue or change related to the parser type: bug 🐛
Milestone

Comments

@sewe
Copy link

sewe commented Dec 23, 2022

Please support fallbackValue with "zero-based" arities other than 0..1, in particular with 0..*.

To see why this might be useful, consider the following @Option definition:

enum DebugFacility { FOO, BAR, BAZ, ALL }

@Option(
    names = { "--debug" },
    paramLabel = "FACILITY", //
    split = ",",
    arity = "0..*", //
    fallbackValue = "ALL", //
    description = "The facilities for which to log debug information (one or more of ${COMPLETION-CANDIDATES}). "
        + "If none are given, debug information will be logged for ${FALLBACK-VALUE} categories.")
public void setDebugFacilities(Collection<DebugFacility> facilities) { ... }

This should support the following scenarios

--debug FOO               -> FOO
--debug FOO --debug BAR   -> FOO, BAR
--debug FOO,BAR           -> FOO, BAR
--debug all               -> ALL (i.e., FOO, BAR, BAZ)
--debug                   -> ALL (i.e., FOO, BAR, BAZ)

The last one unfortunately doesn't work, as the fallbackValue does not get passed to setDebugFacilities; it merely gets an empty facilities Collection.

And we can't insert ALL programmatically in setDebugFacilities, either. To see why, consider the following scenario:

--debug FOO --debug       -> ALL (i.e., FOO, BAR, BAZ)

Here, the parameterless --debug simply results in another call of setDebugFacilities with FOO as single parameter. In other words, one cannot detect the parameterless case

Maybe there is a workaround that I am unable to see, but IMHO fallbackValue support for any "zero-based" arity might be the most idiomatic solution. WDYT?

@remkop
Copy link
Owner

remkop commented Dec 23, 2022

I believe all the examples mentioned would work when the option is defined with arity = "0..1".
That would allow you to make your use case work with the current version of picocli (unless I’m missing something).

I still need to think more about the general use case you mentioned…

@remkop
Copy link
Owner

remkop commented Dec 23, 2022

What comes to mind for the general case is that in the setter method, the logic could compare the parameter collection with the current value (the field where the collection is stored) and if both collections are equal, the end user didn’t specify a parameter. The logic can then programmatically add the fallback value.

I haven’t tested this though, does it work? 😅

@remkop
Copy link
Owner

remkop commented Dec 23, 2022

One more thing: I generally recommend against using arity = "0..*" because it introduces parser ambiguity when the application also has positional parameters.

The question to ask is “does my application really need to support cases like the below?”

--debug FOO BAR BAZ

For the cases you mentioned above, arity 0..1 is sufficient. It still has some potential ambiguity. If this ambiguity becomes a problem, please check the section on Custom Parameter Processing.

@sewe
Copy link
Author

sewe commented Dec 23, 2022

The question to ask is “does my application really need to support cases like the below?”

--debug FOO BAR BAZ

No, that is not a case I need to support, but maybe I misunderstood arity, then.

Interestingly, I can use an arity of 0..1 on a Collection valued setter.

@Option(
    names = { "--debug" },
    paramLabel = "FACILITY", //
    split = ",",
    arity = "0..1", //
    fallbackValue = "ALL", //
    description = "The facilities for which to log debug information (one or more of ${COMPLETION-CANDIDATES}). "
        + "If none are given, debug information will be logged for ${FALLBACK-VALUE} categories.")
public void setDebugFacilities(Collection<DebugFacility> facilities) { ... }

And then picocli still accepts the following as before (unit tests for the win!).

--debug FOO --debug BAR 
--debug FOO,BAR

Apparently, my mental model of arity is wrong. 😞

But even when changing arity to 0..1, picocli doesn't seem apply the fallbackValue, at least in some cases.

--debug /some/positional/parameter/and/most/certainly/not/an/enum/value

calls setDebugFacilities with an empty list but

--debug -- /some/positional/parameter/and/most/certainly/not/an/enum/value

works and injects the fallbackValue of ALL. It does call setDebugFacilities twice, though. 🤯

I just don't get why picocli wouldn't, in the first case, either throw an Invalid value for option '--debug: /some/positional/parameter/and/most/certainly/not/an/enum/value or, preferrably, pass the fallthroughValue. An empty facilities Collection seems to be wrong:

If an option is defined with arity = "0..1", it may or may not have a parameter value. If such an option is specified without a value on the command line, it is assigned the fallback value.

@sewe
Copy link
Author

sewe commented Dec 23, 2022

What comes to mind for the general case is that in the setter method, the logic could compare the parameter collection with the current value (the field where the collection is stored) and if both collections are equal, the end user didn’t specify a parameter. The logic can then programmatically add the fallback value.

I haven’t tested this though, does it work? 😅

Not without further ugliness.

ATM I treat setDebugFacilities as a setter, i.e., it completely replaces the existing this.facilities as, AFAICT, picocli keeps its own list of FACILITY values so future calls pass not only the values of the current option but of all previously parsed options, too.

Unfortunately, this means I cannot simply do this:

public void setDebugFacilities(Set<DebugFacility> facilities) {
    if (facilities.equals(this.debugCategories)) {
        this.facilities = DebugFacility.values();
    } else {
        this.facilities = facilities;
    }
}

Even in the simplest case

--debug /some/positional/parameter/and/most/certainly/not/an/enum/value

setDebugFacilities is called twice (why?) and in both cases without parameters. Now, as intended the first call updates this.facilities to contain all values. Alas, as picocli keeps its own list of FACILITY values, the second call is still parameterless and then I end up in the else case.

Maybe a workaround is still possible by refining this further and being even more stateful in the @Option handler, but somehow that feels wrong. After all, the nice thing about picocli is its declarativeness.

@remkop
Copy link
Owner

remkop commented Dec 23, 2022

Apparently, my mental model of arity is wrong. 😞

Picocli's concept of arity is documented in the user manual: https://picocli.info/#_arity
It is the number of parameters for each option occurrence.
For example, arity=2 would mean that the end user must specify 2 parameters after the option name: --option 1 2.

The section on Default Arity may also be of interest.

But even when changing arity to 0..1, picocli doesn't seem apply the fallbackValue, at least in some cases.

--debug /some/positional/parameter/and/most/certainly/not/an/enum/value

calls setDebugFacilities with an empty list but

--debug -- /some/positional/parameter/and/most/certainly/not/an/enum/value

works and injects the fallbackValue of ALL. It does call setDebugFacilities twice, though. 🤯

Just a note that what we are encountering here is the parser ambiguity I mentioned earlier. A parameter preprocessor may be needed to resolve this.

I just don't get why picocli wouldn't, in the first case, either throw an Invalid value for option '--debug: /some/positional/parameter/and/most/certainly/not/an/enum/value or, preferrably, pass the fallthroughValue. An empty facilities Collection seems to be wrong:

If an option is defined with arity = "0..1", it may or may not have a parameter value. If such an option is specified without a value on the command line, it is assigned the fallback value.

Not sure about the 2 calls or about why the setter is called with an empty collection when the value cannot be converted to a DebugFacility instance. Is DebugFacility an enum or do you have a custom type converter? (And if the latter, what does this converter do for invalid values? Does it throw an exception?)

You can see a bit more what is going on internally by switching on tracing: https://picocli.info/#_tracing
and looking at the output for this case:

--debug /some/positional/parameter/and/most/certainly/not/an/enum/value

Can you post the output here?

sewe added a commit to sewe/picocli that referenced this issue Jan 3, 2023
sewe added a commit to sewe/picocli that referenced this issue Jan 3, 2023
@sewe
Copy link
Author

sewe commented Jan 3, 2023

Sorry for the late replay; I was away for the holidays.

You can see a bit more what is going on internally by switching on tracing: https://picocli.info/#_tracing
and looking at the output for this case:

--debug /some/positional/parameter/and/most/certainly/not/an/enum/value

Can you post the output here?

I created a test that reproduces this issue: sewe@7525c9c (feel free to copy it)

Running the only test case which fails (singleOptionWithFallback) with DEBUG tracing yields the following:

[picocli DEBUG] Creating CommandSpec for picocli.Issue1904$TestCommand@490caf5f with factory picocli.CommandLine$DefaultFactory
[picocli INFO] Picocli version: 4.7.1-SNAPSHOT, JVM: 17.0.4 (Debian OpenJDK 64-Bit Server VM 17.0.4+8-Debian-1deb11u1), OS: Linux 5.10.0-19-amd64 amd64
[picocli INFO] Parsing 2 command line args [--debug, example.txt]
[picocli DEBUG] Parser configuration: optionsCaseInsensitive=false, subcommandsCaseInsensitive=false, abbreviatedOptionsAllowed=false, abbreviatedSubcommandsAllowed=false, allowOptionsAsOptionParameters=false, allowSubcommandsAsOptionParameters=false, aritySatisfiedByAttachedOptionParam=false, atFileCommentChar=#, caseInsensitiveEnumValuesAllowed=false, collectErrors=false, endOfOptionsDelimiter=--, expandAtFiles=true, limitSplit=false, overwrittenOptionsAllowed=false, posixClusteredShortOptionsAllowed=true, separator=null, splitQuotedStrings=false, stopAtPositional=false, stopAtUnmatched=false, toggleBooleanFlags=false, trimQuotes=false, unmatchedArgumentsAllowed=false, unmatchedOptionsAllowedAsOptionParameters=true, unmatchedOptionsArePositionalParams=false, useSimplifiedAtFiles=false
[picocli DEBUG] (ANSI is disabled by default: systemproperty[picocli.ansi]=null, isatty=false, TERM=xterm-256color, OSTYPE=null, isWindows=false, JansiConsoleInstalled=false, ANSICON=null, ConEmuANSI=null, NO_COLOR=null, CLICOLOR=null, CLICOLOR_FORCE=null)
[picocli DEBUG] Initializing command 'null' (user object: picocli.Issue1904$TestCommand@490caf5f): 1 options, 1 positional parameters, 1 required, 0 groups, 0 subcommands.
[picocli DEBUG] Set initial value for field java.util.Set<picocli.Issue1904$DebugFacility> picocli.Issue1904$TestCommand.debugFacilities of type interface java.util.Set to [].
[picocli DEBUG] Set initial value for field java.util.List<java.io.File> picocli.Issue1904$TestCommand.paths of type interface java.util.List to null.
[picocli DEBUG] [0] Processing argument '--debug'. Remainder=[example.txt]
[picocli DEBUG] '--debug' cannot be separated into <option>=<option-parameter>
[picocli DEBUG] Found option named '--debug': field java.util.Set<picocli.Issue1904$DebugFacility> picocli.Issue1904$TestCommand.debugFacilities, arity=0..1
[picocli DEBUG] Split with regex ',' resulted in 1 parts: [example.txt]
[picocli DEBUG] example.txt cannot be assigned to option --debug: type conversion fails: Invalid value for option '--debug': expected one of [FOO, BAR, BAZ, ALL] (case-sensitive) but was 'example.txt'.
[picocli DEBUG] Initializing binding for option '--debug' (FACILITY) on TestCommand@490caf5f with empty Set
[picocli DEBUG] [1] Processing argument 'example.txt'. Remainder=[]
[picocli DEBUG] 'example.txt' cannot be separated into <option>=<option-parameter>
[picocli DEBUG] Could not find option 'example.txt', deciding whether to treat as unmatched option or positional parameter...
[picocli DEBUG] No option named 'example.txt' found. Processing as positional parameter
[picocli DEBUG] 'example.txt' doesn't resemble an option: 0 matching prefix chars out of 1 option names
[picocli DEBUG] [1] Processing next arg as a positional parameter. Command-local position=0. Remainder=[example.txt]
[picocli DEBUG] Position 0 (command-local) is in index range 0..*. Trying to assign args to field java.util.List<java.io.File> picocli.Issue1904$TestCommand.paths, arity=1..*
[picocli DEBUG] 'example.txt' doesn't resemble an option: 0 matching prefix chars out of 1 option names
[picocli INFO] Adding [example.txt] to field java.util.List<java.io.File> picocli.Issue1904$TestCommand.paths for args[0..*] at position 0 on TestCommand@490caf5f
[picocli DEBUG] Initializing binding for positional parameter at index 0..* (<paths>) on TestCommand@490caf5f with empty List
[picocli DEBUG] Consumed 1 arguments and 0 interactive values, moving command-local position to index 1.
[picocli DEBUG] Applying default values for command '<main class>'

Hope this helps.

At any rate, the test also nicely illustrates all cases that already do work, which is most of them 😃.

@remkop
Copy link
Owner

remkop commented Jan 4, 2023

Thanks for posting this.

After looking at the debug trace and the code, I see now that this is an existing bug.
This bug does not manifest for single-value options, but does manifest for collections or maps (multi-value) options.

Analysis

The parser tries to determine for "vararg" options whether the next arg is an option parameter or not.
If the option has a fallbackValue, and the parser can determine in advance that the next arg is not an option parameter, the fallbackValue is inserted before the next arg. So far, so good, but...

The problem is that picocli has a quick check (varargCanConsumeNextValue) and a thorough check (canConsumeOneArgument and canConsumeOneMapArgument) to determine whether an arg is an option parameter.
The thorough check includes parameter value splitting and type conversion.

The fallbackValue is only applied if the quick check fails. <-- This is the bug...

In this use case, the quick check passes, but the thorough check (the type conversion to enum value) fails.
Because of the bug, the fallbackValue is not applied in this case.

Solution

The fix is also perform the thorough check when determining whether to apply the fallbackValue.

Index: src/main/java/picocli/CommandLine.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/main/java/picocli/CommandLine.java b/src/main/java/picocli/CommandLine.java
--- a/src/main/java/picocli/CommandLine.java	(date 1672818924228)
+++ b/src/main/java/picocli/CommandLine.java	(date 1672818924228)
@@ -14259,7 +14259,8 @@
             String fallback = consumed == 0 && argSpec.isOption() && !OptionSpec.DEFAULT_FALLBACK_VALUE.equals(((OptionSpec) argSpec).fallbackValue())
                     ? ((OptionSpec) argSpec).fallbackValue()
                     : null;
-            if (fallback != null && (args.isEmpty() || !varargCanConsumeNextValue(argSpec, args.peek()))) {
+            if (fallback != null && (args.isEmpty() || !varargCanConsumeNextValue(argSpec, args.peek())
+                    || !canConsumeOneMapArgument(argSpec, lookBehind, alreadyUnquoted, arity, consumed, args.peek(), classes, keyConverter, valueConverter, argDescription))) {
                 args.push(fallback);
             }
             for (int i = consumed; consumed < arity.max && !args.isEmpty(); i++) {
@@ -14480,7 +14481,8 @@
             String fallback = consumed == 0 && argSpec.isOption() && !OptionSpec.DEFAULT_FALLBACK_VALUE.equals(((OptionSpec) argSpec).fallbackValue())
                     ? ((OptionSpec) argSpec).fallbackValue()
                     : null;
-            if (fallback != null && (args.isEmpty() || !varargCanConsumeNextValue(argSpec, args.peek()))) {
+            if (fallback != null && (args.isEmpty() || !varargCanConsumeNextValue(argSpec, args.peek())
+                    || (!canConsumeOneArgument(argSpec, lookBehind, alreadyUnquoted, arity, consumed, args.peek(), argDescription)))) {
                 args.push(fallback);
             }
             for (int i = consumed; consumed < arity.max && !args.isEmpty(); i++) {

TODO:

add tests for collection and map options, for arities 0..1, 0..*, 1..* and 2..*, with a mixture of test args including positional params following the option name and without any params following the option name (after the minimum number of mandatory option parameters).

@remkop remkop added this to the 4.7.1 milestone Jan 4, 2023
@remkop remkop added type: bug 🐛 theme: parser An issue or change related to the parser labels Jan 4, 2023
@remkop remkop changed the title Support fallbackValue with arity="0..*" fallbackValue is not applied for vararg multi-value options (was: Support fallbackValue with arity="0..*") Jan 17, 2023
@remkop
Copy link
Owner

remkop commented Jan 26, 2023

Tests:

    static class Issue1904 {
        enum DebugFacility { FOO, BAR, BAZ, ALL, DEFAULT, FALLBACK }

        @Option(
            names = { "--debug" },
            paramLabel = "FACILITY",
            split = ",",
            arity = "0..*",
            defaultValue = "DEFAULT",
            fallbackValue = "FALLBACK")
        Collection<DebugFacility> facilities;

        @Option(names = { "--map" }, arity = "0..*",
            defaultValue = "DEFAULT=0",
            fallbackValue = "FALLBACK=1")
        Map<DebugFacility, String> map;

        @Option(names = "-x")
        String x;

        @Parameters
        List<String> remainder;
    }

    @Test
    public void testIssue1904CollectionFallback_NoOptions() {
        Issue1904 obj = CommandLine.populateCommand(new Issue1904()); // options are not specified
        assertEquals(Collections.singletonList(Issue1904.DebugFacility.DEFAULT), obj.facilities);
        assertEquals(Collections.singletonMap(Issue1904.DebugFacility.DEFAULT, "0"), obj.map);
    }

    @Test
    public void testIssue1904CollectionFallback_NoArgs() {
        Issue1904 obj = CommandLine.populateCommand(new Issue1904(), "--debug"); // no args specified
        assertEquals(Collections.singletonList(Issue1904.DebugFacility.FALLBACK), obj.facilities);
        assertEquals(Collections.singletonMap(Issue1904.DebugFacility.DEFAULT, "0"), obj.map);
    }

    @Test
    public void testIssue1904CollectionFallback_NoArgs_map() {
        Issue1904 obj = CommandLine.populateCommand(new Issue1904(), "--map"); // no args specified
        assertEquals(Collections.singletonList(Issue1904.DebugFacility.DEFAULT), obj.facilities);
        assertEquals(Collections.singletonMap(Issue1904.DebugFacility.FALLBACK, "1"), obj.map);
    }

    @Test
    public void testIssue1904CollectionFallback_otherOption() {
        Issue1904 obj = CommandLine.populateCommand(new Issue1904(), "--debug", "-x", "xarg"); // other option
        assertEquals(Collections.singletonList(Issue1904.DebugFacility.FALLBACK), obj.facilities);
        assertEquals(Collections.singletonMap(Issue1904.DebugFacility.DEFAULT, "0"), obj.map);
    }

    @Test
    public void testIssue1904CollectionFallback_otherOption_map() {
        Issue1904 obj = CommandLine.populateCommand(new Issue1904(), "--map", "-x", "xarg"); // other option
        assertEquals(Collections.singletonList(Issue1904.DebugFacility.DEFAULT), obj.facilities);
        assertEquals(Collections.singletonMap(Issue1904.DebugFacility.FALLBACK, "1"), obj.map);
    }

    @Test
    public void testIssue1904CollectionFallback_positional() {
        Issue1904 obj = CommandLine.populateCommand(new Issue1904(), "--debug", "123"); // positional
        assertEquals(Collections.singletonList(Issue1904.DebugFacility.FALLBACK), obj.facilities);
        assertEquals(Collections.singletonMap(Issue1904.DebugFacility.DEFAULT, "0"), obj.map);
    }

    @Test
    public void testIssue1904CollectionFallback_positional_map() {
        Issue1904 obj = CommandLine.populateCommand(new Issue1904(), "--map", "123"); // positional
        assertEquals(Collections.singletonList(Issue1904.DebugFacility.DEFAULT), obj.facilities);
        assertEquals(Collections.singletonMap(Issue1904.DebugFacility.FALLBACK, "1"), obj.map);
    }

remkop added a commit that referenced this issue Jan 26, 2023
@remkop
Copy link
Owner

remkop commented Jan 26, 2023

The fix has been pushed to the main branch and will be included in the upcoming 4.7.1 release.
Thank you again for raising this!

@remkop remkop closed this as completed Jan 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
theme: parser An issue or change related to the parser type: bug 🐛
Projects
None yet
Development

No branches or pull requests

2 participants