feat: add Voyage AI vectorizer integration #256
Conversation
JamesGuthrie
force-pushed
the
jg/voyageai-vectorizer
branch
4 times, most recently
from
ce01c57 to
baab4d2
Compare
There was a problem hiding this comment.
Note: this file is 11MB because the call to voyageai.Client().tokenizer(<model name here>) uses huggingface under the hood, which dynamically downloads the tokenizer.
There was a problem hiding this comment.
There was also an issue with huggingface's caching making the sequence of requests it makes non-deterministic, so then tests would fail.
I fixed that and the "this file is huge" problem by ignoring requests to huggingface.co in VCR. The API is public, so that actually makes sense.
There was a problem hiding this comment.
That means tests are going to always make calls to that API?
There was a problem hiding this comment.
Yes. It's the huggingface tokenizer API. It returns an 11MB tokenizer file.
There was a problem hiding this comment.
There was also an issue with huggingface's caching making the sequence of requests it makes non-deterministic, so then tests would fail.
Can't we just do expectations without caring on the order?
JamesGuthrie
force-pushed
the
jg/voyageai-vectorizer
branch
from
baab4d2 to
81a6ac4
Compare
There was a problem hiding this comment.
I basically copy-pasted this from docs/openai.md and search-replaced "openai" with "voyageai", which points to the documentation here being quite duplicated.
JamesGuthrie
force-pushed
the
jg/voyageai-vectorizer
branch
3 times, most recently
from
64fc4cd to
2c9aca5
Compare
To configure a vectorizer with Voyage AI:
```sql
SELECT ai.create_vectorizer(
'my_table'::regclass,
embedding => ai.embedding_voyageai(
'voyage-3-lite',
512,
),
-- other parameters...
);
```
The vectorizer worker connects to the Voyage AI API with the API
specified in the `VOYAGE_API_KEY` environment variable.
To get a vector embedding from SQL, use the `ai.voyageai_embed`
function:
```sql
SELECT ai.voyageai_embed('voyage-3-lite', 'text to embed');
```
JamesGuthrie
force-pushed
the
jg/voyageai-vectorizer
branch
from
2c9aca5 to
a4df434
Compare
docs/vectorizer-api-reference.md
Outdated
| embedding => ai.embedding_voyageai( | ||
| 'voyage-3-lite', | ||
| 512, | ||
| truncate => false, |
There was a problem hiding this comment.
I believe we should not set the truncate to false in all of our examples unless we explicitly want to show the behaviour when is set to false. Otherwise, we might confuse users, who 99.9% of the time will want this to be true as default.
| truncate => false, |
JamesGuthrie
force-pushed
the
jg/voyageai-vectorizer
branch
from
b1a2f9f to
30d4566
Compare
|
|
||
| ## Configure pgai for Voyage AI | ||
|
|
||
| Most pgai functions require a [Voyage AI API key](https://docs.voyageai.com/docs/api-key-and-installation#authentication-with-api-keys). |
There was a problem hiding this comment.
nit: We should reword this sentence (and the corresponding ones in the other docs). When it was first authored, we only supported OpenAI, so "most pgai functions" DID require an OpenAI key. This sentence was copy/pasted around. Now, most OpenAI functions require an openai API key, but MOST pgai functions do not. Same goes for VoyageAI and all the other providers.
There was a problem hiding this comment.
I will take this up as a follow-on task.
To configure a vectorizer with Voyage AI:
The vectorizer worker connects to the Voyage AI API with the API
specified in the
VOYAGE_API_KEYenvironment variable.To get a vector embedding from SQL, use the
ai.voyageai_embedfunction: