Tab completion behavior for click.File and click.Path types

[gtristan]

Lets say this is related to #241 ...

Recently support was added to have argument autocompletion, custom completion functions/lists and a special case for click.Choice options too.

This is great ! I would like to see better support for filenames and directories, have been looking at the code and wanted to share my thoughts here... as far as I can see the problems with filename/directory are multiple but hopefully addressed with a couple of simple patches.

Undesirable Autocompletion of paths

In the case that the user is completing an option which takes one or more arguments, and those arguments cannot be completed (i.e. they are just a string or integer type), then the builtin completion mechanism correctly reports no results, however the bash completion glue (here) specifies "-o default"

This parameter to the bash complete function tells bash to default to completing paths when the resulting COMPREPLY array is empty, and the side effect is undesirable filenames being completed.

Filenames reported for directory arguments

Because click does not do anything for filenames or directories itself, it instead falls back on the undesirable complete -o default behavior mentioned above for filenames and all arguments which did not supply any custom autocompletion list or function.

This means that filenames are suggested when the argument is in fact a directory, and also directory names are suggested for filenames.

Filename and directory completion has different behavior

The default shell behavior is to take the reported list in COMPREPLY, and if there is only one element, it will append a space and the next TAB is ready to complete the next token, but this is not the desired behavior when completing paths, where you want to complete only one directory at a time.

This can be addressed however by using the "complete -o nospace ..." option, which will just cause the shell to not append any space for a single available completion, allowing the program to continue completing the last completion on the next iteration.

In this case, click can:

• Append a space to the reported completion in the case that only one completion was available
• Append a slash (if none is already there), in the case that the completion is a directory name

Proposed Approach

Instead of handling click.Choice explicitly in _bashcompletion.py, we should instead have the base ParamType cooperate in the process of completion, so that paths and filenames can have an opportunity to decide if the parameter is entirely complete or could be completed further (i.e. whether a space should be appended to the reported completion, or whether it should remain, as in the case for a filename which is a directory that could be completed further).

click.Choice would implement completions by overriding an abstract method of click.ParamType, click.BOOL could also implement this, because why not.

The new autocompletion attribute would be checked before ever consulting the click.ParamType abstract method, so the user can always override the default completions for any parameter.

Also, we should change the completion script to not use "-o default" and use "-o nospace" instead, ensuring that filenames dont show up by default.

Finally, the click.Path and click.File need to do the work we were previously offloading to the shell, this is the most tricky (or "meaty") part of the patch I guess, hopefully a python implementation using os.listdir() and file concatenation would not slow down the completion process too much.

[gtristan]

So I'm personally quite satisfied with #782 as a solution for this, sorry at first I missed the test cases and came back to adjust the existing ones and add some additional test cases for Path types.

One thing I'm not sure of is if I'm entirely satisfied with the API surface, although it does work.

In order to deal with completion of tree like data, we have two types of completion result; those with trailing spaces and those without. For the bash facing results in COMPREPLY, this must remain the case for reasons explained in the report above. However for the developer facing APIs of ParamType implementations and the autocompletion attribute now available in the decorators, this might not be the most comfortable.

The options I could see for this are:

• Let the implementors append a space for completely completed suggestions, and returning strings that lack a trailing space is an indicator that this completion could be completed further (that is the current approach)
• Have the completion functions (ParamType->completions() and autocompletion callbacks) return a tuple of two lists instead of one

With the second approach, we would append spaces to completely completed suggestions automatically inside _bashcompletion.py and have the implementors report results without the appended spaces, which may be a more comfortable API.

Also, I agree with comments in #428 that autocompletion could be named better, either suggests or completions would be nicer words for this, but that is a separate issue.

Would love to hear feedback from the click maintainership on this patch series :)

[gtristan]

Adding @untitaker on cc...

[untitaker]

I'm rather against removing -o default. It appears to be convention to allow filenames when the custom completion doesn't work, at least with git it does.

This can be addressed however by using the "complete -o nospace ..." option, which will just cause the shell to not append any space for a single available completion, allowing the program to continue completing the last completion on the next iteration.

This appears to be inacceptable to me, since it affects completion of e.g. subcommands as well. I'd rather accept that files and directories are mixed up in the completion.

If one were to implement custom completion for directories, that would mean one would have to recursively traverse and output all directories to avoid appending the space in the case of only one top-level directory name matching. That is slow.

[gtristan]

I'm rather against removing -o default. It appears to be convention to allow
filenames when the custom completion doesn't work, at least with git it does.

For what it's worth, git is not a great example. However the effort various applications have made vary quite widely, a lot of coreutils and standard gnu tools never complete option names for instance. Here are some comparisons:

# -f is a file argument, completes directories and filenames
$ make -f <TAB>
dira/ dirb/ file1 file2

# -C/--directory is directory option, completes only directories
$ make -C <TAB>
dira/ dirb/

# tar's -C option is not as smart
$ tar -C <TAB>
dira/ dirb/ file1 file2

# tar's -f option is however smart, it will only complete filenames
# of the expected filename extensions, it will even take the expected
# compression algorithm into account if it was specified
$ tar -f <TAB>
tarball.tar tarball.tgz tarball.tar.bz2

$ tar -zxf <TAB>
tarball.tgz

# tar at least does not complete filenames when a string is expected
$ tar --quote-chars <TAB>
(no completeions here)

# git's -C is like make's directory, but it fails to complete a directory
# and goes on to complete subcommands
$ git -C <TAB>
add am annotate apply archive ...

This appears to be inacceptable to me, since it affects completion of e.g. subcommands
as well. I'd rather accept that files and directories are mixed up in the completion.

Behavior of subcommand and option name completion still works exactly as expected, in fact the user experience for this is unchanged. Note the test case is changed to expect the trailing spaces which are needed to report to the shell, but that is just the internal get_choices() api.

If one were to implement custom completion for directories, that would mean one would have to
recursively traverse and output all directories to avoid appending the space in the case of only one
top-level directory name matching. That is slow.

In fact, without -o nospace your statement is true; with the current API it is impossible to efficiently implement custom completion of any large tree/path like data sets.

Note that this is implemented in #782, what you do is not append a space to complete one directory at a time, in the next iteration the incomplete text contains to so far completed text.

Currently, if you want to do completions in iteration on any tree/path like dataset, either:

• You list one component, and bash appends a space, so all bets are off for the next iteration, you are already completing the next argument
• You list every branch/leaf node recursively, which will
  • Be slow, as you mentioned
  • Have an unfriendly experience, bash will ask you if you really want to display all 11894 possibilities in the shell, for example

[epruesse]

+1 for having a way to customize argument expansion in particular. In the popular git multi command example, completing e.g. branches to checkout is very helpful, but you wouldn't want to add pseudo-subcommands for each and every one of them. Just having API to extend completion would be great!

[bisho]

I also experienced the same issue. main() is never called, so the context is not properly populated, and you can't reuse the shared state for the autocomplete.

In my case, main() has db connection settings. Defines a DB object in the context, and commands can use the db directly. But the autocomplete now doesn't have access to the DB without the proper context.

[louisguitton]

This is not exactly solving this issue, but can take you a long way and hasn't been mentioned on this thread: check out https://github.com/click-contrib/click-completion

[max-sixty]

Could I confirm that file completions aren't expected to work at all on master?

I'm not able to get any file or directory completions working. When I monkey patch something like #1403, it does work.

[kx-chen]

@max-sixty Work is being done on #1622 that should change that soon. The docs talk a bit more about completion support https://click.palletsprojects.com/en/7.x/bashcomplete/#what-it-completes (and what it supports currently)

[max-sixty]

Thanks @kx-chen ! That looks very encouraging!

[epruesse]

Will this merge https://github.com/click-contrib/click-completion then?

[davidism]

The new design is discussed in #1484. click-completion will not be merged, and may not be compatible with the new system at first. The new system will be much more extensible though, so click-completion or other extensions can add to it more easily.

[kortina]

Hi, I was looking into an issue that seems related to this one, so I thought I'd ask about it here vs create a new issue (lmk if I should create a separate one, tho).

I am using zsh on mac:

fpath=($HOME/.zsh/completion $HOME/.zsh/zsh-completions $fpath)
autoload -Uz compinit
compinit -i
export PATH="$PATH:$HOME/src/sq"
eval "$(_SQ_COMPLETE=source_zsh sq)"

I have a click python file, sq and am trying to get autocompletion working for filepaths:

@dev.command(help="Touch file")
@click.argument("input_filename", type=click.Path(exists=True))
def touch(input_filename):
    _run_command(f"touch {input_filename}")

sq d[tab] → sq dev
sq dev t[tab] → sq dev touch
sq dev touch t[tab] → NOTHING (even if test.txt exists on filesystem)

I would expect this last case to complete to test.txt, especially since i used click.Path for the argument type, esp given this comment in the thread above:

In the case that the user is completing an option which takes one or more arguments, and those arguments cannot be completed (i.e. they are just a string or integer type), then the builtin completion mechanism correctly reports no results, however the bash completion glue (here) specifies "-o default"

Is the behavior I am seeing expected, or do I need to do something special to get file path completions? Is this related to #1622 or user error on my part w the current version of click?

[amy-lei]

@kortina file/path completion currently does not work, but #1622 addresses it

[ronykoz]

@davidism are you planning to release a new Click version that includes this feature?

[samschott]

I have looked through the changes in #1622 and think this is a huge improvement!

Something which I am a bit unclear about -- not sure if this is the right place to ask: The OP discussed not only issues with completing files / directories from the local file system but also completing custom tree-like data.

As far as I can see, this is still difficult with the new shell completion system. Directory or file completion can be requested by setting the CompletionItem type to "dir" or "file" but this seems to result in completion being entirely handled by the shell based on the current file system. Alternatively, if completion is handled manually, the issue with a trailing space appended to the competed string still exists.

Is it possible to implement ParamType.shell_complete in a way which allows completion of a custom tree? In my use case, I would like to implement completion for paths from a virtual or remote file system.

[davidism]

The problem you describe is not generally solvable (at least in a way that seems at all maintainable). If you need to customize at that level and the existing facilities don't provide that for you, you'll need to register a new shell completion script that does what you want for your case.

[samschott]

Ok, that's what I was afraid of :/ At least with the rewrite it is a lot easier to extend the existing completion system.