Skip to content

glob.glob('**/**', recursive=True) yields duplicate results #104269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
barneygale opened this issue May 7, 2023 · 2 comments
Closed

glob.glob('**/**', recursive=True) yields duplicate results #104269

barneygale opened this issue May 7, 2023 · 2 comments
Labels
3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@barneygale
Copy link
Contributor

barneygale commented May 7, 2023 •
edited by bedevere-app bot
Loading

Calling glob.glob(pattern, recursive=True), where pattern contains two or more **/ segments, can yield the same paths multiple times:

>>> import glob
>>> len(glob.glob('**/**', recursive=True))
314206
>>> len(set(glob.glob('**/**', recursive=True)))
44849

Linked PRs
@barneygale barneygale added type-bug An unexpected behavior, bug, or error 3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes labels May 7, 2023
@barneygale
Copy link
Contributor Author

Reasonably minor bug I think as these patterns are pretty rare. A note in the docs might be the best solution.

@bedevere-bot bedevere-bot mentioned this issue Jun 6, 2023
Merged
@iritkatriel iritkatriel added the stdlib Python modules in the Lib dir label Nov 24, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Mar 4, 2024
@barneygale
pythonGH-106747: Prepare pathlib globbing for dir_fd support.
6932522 
The present implementation of `pathlib.Path.glob()` creates a series of
'selectors' that each handle a part of the pattern. The selectors are
connected together in `glob()`, without the use of recursion. One very
subtle property of this scheme is that each selector is exhaused *before*
its successor selector - for example when globbing `*/*.py`, the selector
for `*` is exhausted prior to the selector for `*.py`. This doesn't make
any difference when globbing strings, but it does prevent us from adding
`dir_fd` support, because there's no good moment to call `os.close(fd)`
after opening a directory for scanning.

This patch refactors globbing to work much as it did in 3.12, where each
selector is responsible for creating and feeding its own successor. This
inverts the order of selector exhaustion, and so will make it much easier
to add `dir_fd` support.

There's one behaviour change here: I've removes deduplication of results,
and so in some very specific circumstances (multiple non-consecutive `**`
segments in pattern, and either `follow_symlinks=None` or `..` segments
separating them), `glob()` can yield the same path more than once. Note
that `glob.glob()` can also yield duplicate results - see pythonGH-104269.
This was referenced Mar 4, 2024
Closed
Open
@barneygale barneygale mentioned this issue Mar 15, 2024
Merged
barneygale pushed a commit that referenced this issue Apr 11, 2024
@tomasr8
gh-104269: Document glob.glob duplicates when using multiple ** p…
c06be6b 
…atterns (#105406)
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Apr 11, 2024
@tomasr8 @miss-islington
pythongh-104269: Document glob.glob duplicates when using multiple …
364cd66 
…`**` patterns (pythonGH-105406)

(cherry picked from commit c06be6b)

Co-authored-by: Tomas R <tomas.roun8@gmail.com>
@bedevere-app bedevere-app bot mentioned this issue Apr 11, 2024
Merged
barneygale pushed a commit that referenced this issue Apr 11, 2024
@miss-islington @tomasr8
[3.12] gh-104269: Document glob.glob duplicates when using multiple…
84fb531 
… `**` patterns (GH-105406) (#117757)

gh-104269: Document `glob.glob` duplicates when using multiple `**` patterns (GH-105406)
(cherry picked from commit c06be6b)

Co-authored-by: Tomas R <tomas.roun8@gmail.com>
@barneygale
Copy link
Contributor Author

Thanks for sorting this, @tomasr8!

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@tomasr8 @diegorusso
pythongh-104269: Document glob.glob duplicates when using multiple …
4c350d2 
…`**` patterns (python#105406)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
Fnmatch and glob issues
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@barneygale @iritkatriel