Avoid full vault scan on incremental indexing

[zeroliu]

Previously, we added an incremental indexing feature for copilot plus users. On any note update, It generates the excluded file list and checks whether the current file should be updated. This process can take 5 seconds for large vaults and causes the plugin to be unusable in those extreme scenarios. This PR fixes the issue by stopping the generation of included/excluded file lists and matching the current file path against the matching patterns.

• Added a new shouldIndexFile method that checks whether a TFile should be indexed. It consolidates all the custom validation logic that checks inclusions and exclusions.
• Updated the pattern-matching logic to use native obsidian API when possible.

[logancyang]

Thanks for fixing this! How did you test it?

[zeroliu]

In terms of functionality, I tested inclusion and exclusion with tags, file extension, and single file match with [[]]. I observe whether shouldIndexFile returns true for the matching file path. I also tested different patterns with a full vault index and verified that the number of files indexed was correct.

In terms of performance, this part gets confusing. I created a vault with 40000 files. I was hoping to see the 5-second delay as described in the issue, but I couldn't reproduce it. As you can see in my screenshot, even after I added ,,,,,,,,,,, as exclusion patterns that create a number of excluded strings. It still only takes ~40ms to run the file change handler. There must be a scenario that the user is in, but I can't reproduce it. My change shows that the change handler is now consistently faster, so this is still a good PR to merge. I hope it will resolve the reported performance issue, too.

Before:

After:

[logancyang]

This is now returning both inline and frontmatter tags, while before it only returns frontmatter tags.

The reason I only get frontmatter tags is to

1. Limit the number of notes from tags for custom prompt templating {#tag1, #tags2...} and avoiding overcrowding the context window
2. Have a differentiation between frontmatter tags and inline tags. Consider frontmatter tags a "note-level" property and inline tags as "line/block-level" property.

So this change will affect not only index filtering but also custom prompt processing. Currently, inline tags are used via salient terms during vault search.

Should we put a parameter here to control what tags to return? Such as frontmatterOnly = true

[zeroliu]

Updated

[zeroliu]

need to mock these because of the usage of app.vault.metadataCache

[zeroliu]

Updated