Scope setting

[quelsemme]

At the moment, a 'based on my notes' inquiry feeds about 30 notes to the chat bot for its context.

Would it be possible to implement a scope setting? This could be set for either 'how many notes' to send or a 'how many tokens/words to send'. I'm not too worried about API costs, though others might be.

There are some questions where I'd like a larger range of notes to be used (even if that meant splitting the prompt for a greater scope) and others where I think just the 'TOP 10' response will be more precise.

Potentially, a better option is a word limit scope in the Settings. So if the most relevant note is 20,000 words, it focuses on that note, whereas if there are 300 twenty-word notes, it can expand its search to allow for that different form.

Alternatively, and I imagine this is far more complicated, could a 'based on my notes' prompt bring up a dialogue box of the top 50 related notes? This could allow a choice to exclude certain notes and not send them to the API.

My approach, as always, has to do with academic research and using the Chat as an intelligent research assistant across my own notes. Some of my research might be split across 100 notes, while others may have less.

For example, one database I keep in Obsidian has over 300 core notes and 5-10 notes linked within each. While, at present, I can narrow the field to a folder containing these notes, but the scope of the database, given it uses a lot of small notes, means that the API can't access everything and is less likely to be able to give accurate responses.

[quelsemme]

At the moment, it feels like SmartChat takes about 7000-8000 tokens for a standard prompt. (approx 30000 characters or 6000 words)

Let's say the minimum could be set at 8K tokens and the maximum at 127K, with a warning that every prompt at 127K will cost about $1.30, I feel like that's a power scale that would skyrocket the efficacy.

As always, I'm approaching this as a PhD researcher. If I could access all my own notes, I would pay $1.30 for certain questions (the most general ones)... I'm also just spitballing, but I wouldn't mind a similar scope for prompt response, even though I know that GPT has a tough time understanding word counts for its responses...

I do intend this as a helpful perspective from a user, @brianpetro and not as a criticism of your work which I clearly adore you for.

[brianpetro]

@quelsemme sorry I missed your first post.

In short, I do plan on addressing this in the upcoming Smart Chat improvements.

The Smart Chat will have an added settings view, and I'm anticipating solving this directly using a max tokens input (likely two "range slider" inputs, one of input and one for output tokens). This setting will be available as an overlay on the chat, so quickly changing it between messages should be easy.

Indirectly, I also plan on adding retrieval options, which implicitly may not use as many tokens. Regardless, the above slider inputs will act as hard limits.

bring up a dialogue box of the top 50 related notes? This could allow a choice to exclude certain notes and not send them to the API.

I think that's a cool idea. I haven't quite thought it through, but it's definitely something I'll be considering.

One last possibility is adjusting the cut-off point. For example, the current retrieved notes are stopped when the subsequent note exceeds the standard deviation between the prior notes. This is increased to something like two standard deviations intentionally in the 128k context window to increase the amount of context used. However, this might be another good input to include alongside the aforementioned sliders. Though, it might be a bit difficult to explain exactly how this works in a brief description to a general audience.

Let me know what you think & thanks for following up on this!

🌴

[quelsemme]

This all sounds great.

Here's my 'thinking out loud' response:
I hadn't considered your cut-off in regards to standard deviations... I can see why that would make the prompt more economical certainly, filtering out information that isn't specific to the prompt.

I would say that's problematic because it relies upon the prompt being precise, but the way embeddings work, even imprecise prompts should work out with vectors which are more precise (that is, if someone says 'cat', the virtue of embeddings is that it will find 'animals' if it can't find 'cats' just by probability.

I guess it comes down to how much you trust the GPT itself to parse the information. If you trust it fully, giving it everything in the vault can't fail. If you think it'll misinterpret the vault, then limiting via standard deviation helps to only feed it what is necessary.

Potentially, the input token limit does the same thing, although not so elegantly. It could be a matter of use. A description that says 'higher input maximums should be used for finding information spread across many notes'...

What brings me to the thought for a Scope function is this...
At the moment, an effective prompt in Smart Connections could be (to use my own research):

Based on my notes, what are the different components of Jacques Derrida's archive fever or the 'mal d'archive' and how do other writers interpret that concept beyond Derrida's 1995 essay?

The vector should be pretty precise and return 30 notes very specific to that topic, using smart blocks and so forth. That said, this is a major concept of mine, so even at this level of precision it's not going to cover all of the other perspectives beyond Derrida's. My notes on this concept are pretty thorough and could span 50 notes at least.

On the other hand, a question like
Based on my notes, what are the most common ideas in the field of 'archive studies'
is bound to miss things. It will give a good answer based on about 30 notes, but won't have any sense of 'common ideas' because it can't take everything in.

Another reason for coming to this...
I have a part of my vault which acts kind of like a database. Each note is a list of 10-15 names and their job titles. These are small notes, but there are 300 of them. Smart Chat could do some great analysis of that information, if it had access to all the notes. Right now, it does a decent job if I give it a specific name, since that name doesn't pop up 300 times, but is less accurate if I give it a job title which appears across all notes.

Maybe an elegant solution is to combine the two (maximum tokens and maximum SDs) into one slider?
That is, a slider that goes from 8K to 128K while simultaneously going from 1 standard deviation to 10 (just throwing numbers out there) - so the increase in scope still factors in accuracy via embeddings (and doesn't just send the entire vault every time). It could still display a maximum cost.
In some ways, kind of like the 'depth' function on the Local Graph. In fact, maybe 'Depth' is better than 'Scope' for both Input and Output sliders. You could still measure both in words.

Query depth: 6000 to 90,000 words ($0.01 - $1.30)

As for output, I'm not sure you'd have much luck convincing GPT-4 to give you 1000 words just by asking for it in the prompt. At least, not in my experience. It might be less predictable, but you could still do a similar thing on that slider.
Response depth: 100 to 3000 words ($0.01 to $0.12)

Potentially an OFF switch for depth would set the query to 2 standard deviations and remove any length specifications so the prompt was whatever the GPT decides.

Words vs Tokens
Not sure whether it's better for the user if you use 'words' instead of 'tokens' on the sliders. I would prefer 'words' if it had a price guide next to it... I think it's intuitive to think 'I'm sending a maximum of 10000 words to GPT now'

[quelsemme]

As for the dialogue box for the 'here are the notes we plan to send', the 'Note Linker' plugin does it when you click 'Scan vault' or 'Scan note' if that gives you a helpful model. It can be heavy, so it might be the kind of thing that can be turned on or off:

Inspect source notes before submission [X]

Particularly because it would slow the process of chatting with GPT if it was on my default.

[localjo]

Adding my 2 cents to this discussion. I'm encountering an issue where SC regularly picks up notes to add to my context which are in reality not related to my query at all, and then the response gets mixed in with things that are incorrect or missing the point of my query. I would love the ability to inspect source notes before submission, or manually edit which notes the chat is using for the context.

[brianpetro]

@localjo the review-retrieved-context-before-response feature is planned for the next major iteration of the Smart Chat 🌴