Questions about the inner workings of the plugin

[tophee]

I am working my way into the whole text-embedding field. I created a canvas in my Obsidian vault in which I'm trying to map the different apps/approaches/technologies available (or perhaps I should say: under development).

For each app/approach, I'm trying to answer a couple of questions, such as

1. How are the embedding vectors generated (usually it's OpenAI's text-embedding-ada-002)?
2. How are these stored (e.g. qdrant, pinecone, weaviate, postgresDB, ...)?
3. Where is the embedding database is hosted (locally or in the cloud)?
4. What happens when I ask a question? (what chunks of text are retrieved from the vector database and how are they processed and passed on to the LLM?)
5. How the app can be accessed (e.g. python CLI, web-ui, etc) and how it integrates with other apps?
6. Overall approach/ focus/ strenghts/ aim compared to others (why was the app created in the first place, given the alternatives)

These questions are in constant flux as I learn about what dimensions are worth comparing or for understanding text-embedding more broadly. For example, when I learned that smart connections plugin stores the vectors locally, I added this as a salient question. Another emerging dimension seems to be to what extent the app is open source and, perhaps more importantly, to what extent it is locking users into OpenAI's models (with the ChatGPT retrieval plugin obviously sitting at one end of that spectrum).

But I have not been able to find much information on how the plugin works behind the scenes, i.e. questions 2, 4, 5, and 6. This may well be due to my limited understanding of how embeddings work (for example, it is beginning to dawn on me that question 4 is more or less the same for all) but I would nevertheless appreciate any clarifications you might be able to provide.

To be a bit more specific, I guess one thing I'm trying to understand is whether smart connections is building on any existing embedding solutions (e.g. langchain, Llamaindex, embedbase, ...) or building something new from the ground up.

P.S: I have seen the high-level answer to question 6 in this post, but even though I'm not a developer, it is a bit too high level for me. I would like to better understand how embeddings are leveraged to achieve these features. Of course, I fully respect if you're not willing to share these things at this point.

[brianpetro]

@tophee

2. Single file embeddings-2.json
3. Depends, but for queries across multiple notes, HyDE is used. See this diagram.
4. Obsidian, Smart Connections is a plugin for Obsidian.
5. Smart Connections is built from the ground up with a focus on markdown, the primary format of notes in Obsidian. Whether there are alternatives is a matter of perspective, and from mine, there aren't any.

Brian 🌴

[tophee]

Oh, javascript, good point! So you're saying that LangChain.js doesn't work with Obsidian?

[tophee]

Oh, I just realized that LangChain.js was only released very recently...

[brianpetro]

@tophee what benefits do you think this project would gain from using LangChain?

[tophee]

If I sounded like I was advocating for LangChain, that was not my intention. At this point, my knowledge is way too limited for me to have a preference for any specific approach or product. I'm asking these questions (as a non-developer) because I'm trying to understand the different approaches and the benefits connected to them.

So, with regards to LangChain, I guess my question is (or was): what are the benefits of not using LangChain, given that it seems to be able to do at least part of what Smart Connections needs (it even supports HyDE). But, again, I don't mean to say that you should use it.

If you say "I didn't use LangChain back in December 2022 because it was only available in Python and now there is no point in switching, because I already wrote my own code", I'm happy, because I learned something.

If you say "I didn't use LangChain back in December 2022 because it was only available in Python but even now that there is a javascript version, switching is not an option because LangChain can't do X", I'm happy, because I learned something.

If you say "I didn't use LangChain back in December 2022 because it was only available in Python but now that they have a java version, I might actually incorporate it, as it will speed up my 'journey to JARVIS'", I'm happy, because I learned something.

So, these are my three hypothetical documents. They are probably wrong, but their purpose is, obviously, to facilitate the retrieval of the correct answer ;-)

[brianpetro]

I was curious if you saw some possibility for improvement with the integration of LangChain. Given my being comfortable with implementing many of these concepts, it's just easier for me to avoid extra layers of abstraction without obvious benefits. In my experience, abstraction layers make initial development quicker with a trade-off of future flexibility.

For Smart Connections, I've so far avoided any external dependencies that aren't included in Obsidian. So, from my perspective, that makes the software very flexible.

But these design choices are just opinions necessary for progress. And I like to go by the "strong opinions loosely held" motto. So if you were to say that LangChain has some features that would undoubtedly make the future of this project brighter, I'd be happy to learn more about them and revisit the relevant design choices if it made sense.

And thank you for taking interest in this project. It's a culmination of my passions like building/designing PKM systems, ontological organization, and of course AI. So it's always nice to see that people are interested and benefiting from it.

Thanks,
Brian 🌴

[tophee]

I was curious if you saw some possibility for improvement with the integration of LangChain.

If I ever come up with an idea (once I have a better understanding of both), I'll let you know. But what you say makes sense. Especially:

abstraction layers make initial development quicker with a trade-off of future flexibility.

I didn't think of that but when you say it, I see the point. As a non-developer, I guess I was too focused on opportunities for "out-sourcing". But it just strikes me that I recently basically used the same argument myself in a discussion about using or not using templates in a home automation system. Interesting!

And thank you for taking interest in this project. It's a culmination of my passions like building/designing PKM systems, ontological organization, and of course AI. So it's always nice to see that people are interested and benefiting from it.

Thank you for sharing this plugin! I hope to digg deeper into it soon. At the moment, I am just generating new notes on various AI related concepts at unorecedented rates rather than fiddling with plugins...

[sergioatp]

I'm sorry but aren't JSON files limited to 1 GB?
If so, one unique file is not enough to track the vector base parameters for a large vault.
Also, it would be more beneficial to use a database instead, there are so many obsidian database integrations.
Plus after the JSON file because considerably large we lose speed and efficiency.

I actually use this to manage several websites and write content with factual data.
For example, I grab software documentation and convert it to md format.
Next, I use note refactor to slip the H2 and H3 into smaller files to facilitate the embedding and reduce API calls token usage.
From there, I write the article referring to my notes and get super accurate documentation.
For example, I asked for the Cloudflare implementation and possible error fixing of an issue related to the software. The chat responded with an original response with the correct code and implementation.

The possibilities are endless.
No AI hallucinations.
Correct information.
But having sites in different niches can cause organization issues which is why we can separate the vaults or have separate JSON files or better yet a database to organize everything.

Basically, we are recreating Mem,io with Obsidian, without the ridiculous Mem ai generations prices.