[Bug]: New caching mechanism is too slow/fragile to be practical

[catboxanon]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

I wrote a comment about this on the related commit, which I'm copy-pasting below:

FYI, this is still extremely slow if someone has a bunch of extra networks, I believe mostly due to the fact for each item it needs to load the full cache from disk and then write the cache to disk, and repeat that each time, instead of keeping everything in memory. I put tqdm on the main iteration loop for file candidates and it estimates caching everything will take over an hour.

Edit: Though it finished after 4 hours, the problem is now that any time a hash is calculated, it takes several seconds to load and then save the cache file (which is now around ~150MB). Interrupting this operation also causes the cache file to become incomplete/corrupt. May want to consider using an SQLite DB to store all this. https://github.com/lllyasviel/SingleFileDB

The gist is that the new caching mechanism (currently on the dev branch, and part of the upcoming 1.5.0) is impractical when a user has thousands of extra networks. There is also no indication of this activity happening in the console. If we're going to start caching all this information, we shouldn't be storing JSON files that are >100MB. I don't see the reason to store this information in a human-readable format anymore due to this and also the fact the web UI makes all this information available.

IMO this should also be solved before #11808 and #11821 get merged.

What should have happened?

The caching mechanism should be fast and not interrupt startup time and disk activity.

Version or Commit where the problem happens

7d26c47

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Windows

[AUTOMATIC1111]

The problem most likely is not that it's a json, but that it's dumping the cache to disk after every addition. I'll see if I can remedy this.

[catboxanon]

Alright. If it helps, this LoRA in particular seemed to be problematic because it trained on 40k+ images, and all their associated tags are recorded in the metadata, so it bloats the cache file. https://civitai.com/models/108649

[evanjs]

Absolutely no clue if it's a valid approach or whether it covers all resources, but I just made it so dumping is only done after every 128 entries, and the application does load properly at least.

Ah... yeah, that's not going to work; dumping like that isn't grabbing all the LORAs 😬

[Panchovix]

I can confirm, I have ton of LoRAs/TIs and it took ~2 hours to load the webui.

[AUTOMATIC1111]

See if commit 9251ae3 in the dev branch solves this.

[catboxanon]

Looks like that does mostly resolve it -- it still took an extra few minutes for me and I had to add tqdm again to double check it was actually doing something. Because of that I think there should be a progress indicator but I'm leaving that up to you.

[evanjs]

9251ae3 appears to resolve the issue on my end as well
Thank you

[catboxanon]

Closing since the root issue is solved.