Too many directories are scanned with multi-patterns

[fabianwohlfart]

If I have two patterns like this

const files = await glob(['../app/**/*.vue', '../utilities/**/*.vue'])

and a folder structure like this

- app
- should-not-matter
- utilities

not only the app and utilities folder are searched, but also the should-not-matter folder.
This leads to huge performance breakdowns, if not only one, but many folders are »in between«.

I came to this behavior through a weird warning in Nuxt, where suddenly my build process increased from 500ms to 25seconds.
nuxt/nuxt#30137

I tracked it down to a scanDirectory function in unimport, that took longer than expected
unjs/unimport#401

I came to the conclusion that they recently switched from glob to tinyglobby
Yet glob only takes a few milliseconds, but tinyglobby takes seconds to scan everything.

Here is a reproduction.
https://github.com/fabianwohlfart/test__nuxt-layers-slowdown

You can simply go to the app folder and run the tinyglobby.mjs, you will see that it searches the folder should-not-matter, which it shouldn't.

[fabianwohlfart]

What I cannot get my head around is:

With this pattern:
/Users/Development/Foundation/media/utils/*.{mts,cts,ts,tsx,mjs,cjs,js,jsx}

And this cwd:
/Users/Development/Project/website__bszb

The result of normalizePattern is:
../../Foundation/media/utils/*.{mts,cts,ts,tsx,mjs,cjs,js,jsx}/**

And it will search our complete potentialRoot
/Users/Development
(Which may holds soooo much content)

But why are we not directly searching this as a root?
/Users/Development/Foundation/media/utils

[SuperchupuDev]

already fixed in main by #69 (and is being improved in #76), you can test the latest commit by adding

  "overrides": {
    "tinyglobby": "https://pkg.pr.new/tinyglobby@d51e2b9"
  }

to your package.json. let me know if it works and i'll reopen this if it doesn't :)

[SuperchupuDev]

What I cannot get my head around is:

With this pattern: /Users/Development/Foundation/media/utils/*.{mts,cts,ts,tsx,mjs,cjs,js,jsx}

And this cwd: /Users/Development/Project/website__bszb

The result of normalizePattern is: ../../Foundation/media/utils/*.{mts,cts,ts,tsx,mjs,cjs,js,jsx}/**

And it will search our complete potentialRoot /Users/Development (Which may holds soooo much content)

But why are we not directly searching this as a root? /Users/Development/Foundation/media/utils

the old root optimizer didn't run if the pattern started with .., broke some tests. shouldn't be a problem ever since #69 got merged though (although there's yet to be a release that includes the commit)

[fabianwohlfart]

Runs smoothly! Thanks for your work!

[fabianwohlfart]

EDIT:
Ignore all the following. I made a mistake with the naming of my folders. All works. _ANOTHER does not exist; I renamed it recently to _Another.

(Leaving the comment for completeness)

Somehow this options

{ absolute: true,                                                                                                                                                                                         22:47:01
  cwd: '/Users/xyz/__Development/Client/website__client-v2',
  onlyFiles: true,
  followSymbolicLinks: true,
  patterns:
   [ '/Users/xyz/__Development/Client/website__client-v2/server/utils/*.{mts,cts,ts,tsx,mjs,cjs,js,jsx}',
     '/Users/xyz/__Development/_ANOTHER/another__foundation/content/server/utils/*.{mts,cts,ts,tsx,mjs,cjs,js,jsx}',
     '/Users/xyz/__Development/_ANOTHER/another__foundation/media/server/utils/*.{mts,cts,ts,tsx,mjs,cjs,js,jsx}',
     '/Users/xyz/__Development/_ANOTHER/another__foundation/website/server/utils/*.{mts,cts,ts,tsx,mjs,cjs,js,jsx}',
     '/Users/xyz/__Development/_ANOTHER/another__foundation/utilities/server/utils/*.{mts,cts,ts,tsx,mjs,cjs,js,jsx}' ] }

Will skip the _ANOTHER folder:

[tinyglobby 22:47:01] crawling /Users/xyz/__Development/Client/
[tinyglobby 22:47:01] skipped /Users/xyz/__Development/ProjectA/
[tinyglobby 22:47:01] skipped /Users/xyz/__Development/ProjectB/
[tinyglobby 22:47:01] skipped /Users/xyz/__Development/ProjectC/                                                                                                                             
[tinyglobby 22:47:01] skipped /Users/xyz/__Development/_Another/   

Due to exclude() most likely

[SuperchupuDev]

@fabianwohlfart FYI the newest release 0.2.11 should fix these problems. Can you test with that version?

[fabianwohlfart]

yes works nuxt/nuxt#30137 (comment)