Globs ignore exclude_path

[Wumpf]

I wanted lychee to check rust files, so I run lychee **/*.rs, confused why it takes so long to scan for links, I ran lychee **/*.rs --dump-inputs which showed that paths excluded via exclude_path options (e.g. target) did not influence the glob.

This is quite inconvenient since what I'm actually after is adding extensions to be checked. At the very least this should be documented better.

[mre]

I think it should actually work, though.
At least that's the idea of this part:

lychee/lychee-lib/src/types/input.rs

Lines 337 to 338 in eb12064

	if self.is_excluded_path(&path) {
	continue;

At this point, I don't know why it doesn't work.
One note, though: you're using your operating system's glob handling. Glob expansion happens in your shell.
To use the Rust-based one, you need to quote the glob.

lychee '**/*.rs' --dump-inputs

so

lychee --exclude-path './target/' '**/*.rs' --dump-inputs

However, this gives me the same incorrect results as you described.
If you find the correct incantation, we should certainly document that.
Otherwise, I consider this a bug and would love to have someone investigate that.
Not sure if you have the time, but I'd certainly appreciate your help.

As a side note, glob filtering is not ideal right now. Would love to move to globset, which would allow us to provide multiple globs and so on. That shouldn't be a blocker for better exclude handling for globs with the current implementation, though.

[Aariq]

Also having issues with this (but an R project where I'd like to exclude the renv/ directory and use a glob to check .md files). I can't seem to create a simple reproducible example though. I have discovered that --dump-inputs doesn't produce accurate output though (I think). E.g.

$ ls -R
nope yep

./nope:
a.txt b.md

./yep:
a.md  b.txt

--dump-inputs says nope/ isn't excluded

$ lychee --exclude-path nope './**/*.txt' --dump-inputs
nope/a.txt
yep/b.txt

but actually it is excluded

$ lychee --exclude-path nope './**/*.txt'
  2/2 ━━━━━━━━━━━━━━━━━━━━ Finished extracting links                                                                                                Issues found in 1 input. Find details below.

[yep/b.txt]:
✗ [ERR] https://qwkjlkrwjelrjwerlwekj.jp/ | Failed: Network error: error sending request for url (https://qwkjlkrwjelrjwerlwekj.jp/)

🔍 2 Total (in 0s) ✅ 1 OK 🚫 1 Error

[mre]

I think dump-inputs currently is pretty simple. As you discovered, we don't seem to respect excludes yet. We should add that.

[mre]

@Aariq, I tested dumping the links in combination with globs again locally, and they seem to work now with the latest lychee version. Can you confirm?

Never mind, I just realized we were talking about --dump-inputs here, which is still broken.

[mre]

Fixed in #1556.