Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

embeddings: clean up code using new inferaces/type #379

Open
4 of 5 tasks
eliben opened this issue Nov 30, 2023 · 6 comments
Open
4 of 5 tasks

embeddings: clean up code using new inferaces/type #379

eliben opened this issue Nov 30, 2023 · 6 comments

Comments

@eliben
Copy link
Collaborator

eliben commented Nov 30, 2023 •
edited by tmc
Loading

#374 introduced NewEmbedder which can be used to clean up much of the code in the embeddings package, following #372

This is a tracking issue for that work.

  • tei
  • ollama
  • openai
  • vertex
  • huggingface (details here)
@eliben eliben mentioned this issue Nov 30, 2023
Closed
eliben added a commit to eliben/langchaingo that referenced this issue Dec 1, 2023
@eliben
embeddings: use NewEmbedder for OpenAI embeddings
a4e38bc 
This is the first use of the refactoring made in tmc#374 -- removing the
embeddings/openai directory entirely, since it can be now implemented
with NewEmbedder without duplicating code. The test was moved one
dir down and works with the new setup.

This is a model that can be used to similarly refactor the other
embeddings/* subdirs.

This is a large PR because it also updates example go.mods

For tmc#379
@eliben eliben mentioned this issue Dec 1, 2023
Merged
7 tasks
tmc pushed a commit that referenced this issue Dec 1, 2023
@eliben
embeddings: use NewEmbedder for OpenAI embeddings (#385)
00bd62f 
This is the first use of the refactoring made in #374 -- removing the
embeddings/openai directory entirely, since it can be now implemented
with NewEmbedder without duplicating code. The test was moved one
dir down and works with the new setup.

This is a model that can be used to similarly refactor the other
embeddings/* subdirs.

This is a large PR because it also updates example go.mods

For #379
@eliben eliben mentioned this issue Dec 1, 2023
Merged
3 tasks
@tmc tmc mentioned this issue Dec 1, 2023
Closed
@eliben eliben mentioned this issue Dec 1, 2023
Merged
4 tasks
eliben added a commit that referenced this issue Dec 1, 2023
@eliben
embeddings: use NewEmbedder for Vertex embeddings
295b76a 
For #379
@eliben eliben mentioned this issue Dec 1, 2023
Merged
eliben added a commit that referenced this issue Dec 1, 2023
@eliben
embeddings: use NewEmbedder for Vertex embeddings (#391)
4182ba4 
For #379
eliben added a commit that referenced this issue Dec 1, 2023
@eliben
embeddings: use NewEmbedder for Ernie embeddings
c397957 
For #379

Updates examples go.mod because it's used in an example
This was referenced Dec 1, 2023
Merged
Merged
eliben added a commit that referenced this issue Dec 1, 2023
@eliben
embeddings: use NewEmbedder for Ernie embeddings (#392)
bef6d7e 
For #379

Updates examples go.mod because it's used in an example
eliben added a commit that referenced this issue Dec 1, 2023
@eliben
embeddings: add example of proper usage, and tweak doc.go accordingly
ce6962e 
For #379
@eliben eliben mentioned this issue Dec 1, 2023
Merged
eliben added a commit that referenced this issue Dec 1, 2023
@eliben
embeddings: add example of proper usage, and tweak doc.go accordingly (…
9508a34 
…#393)

* embeddings: add example of proper usage, and tweak doc.go accordingly

For #379

* Appease lint and add comment
tmc pushed a commit that referenced this issue Dec 5, 2023
@eliben @tmc
embeddings: use NewEmbedder for OpenAI embeddings (#385)
fc033c3 
This is the first use of the refactoring made in #374 -- removing the
embeddings/openai directory entirely, since it can be now implemented
with NewEmbedder without duplicating code. The test was moved one
dir down and works with the new setup.

This is a model that can be used to similarly refactor the other
embeddings/* subdirs.

This is a large PR because it also updates example go.mods

For #379
tmc pushed a commit that referenced this issue Dec 5, 2023
@eliben @tmc
embeddings: use NewEmbedder for Vertex embeddings (#391)
f2439a3 
For #379
tmc pushed a commit that referenced this issue Dec 5, 2023
@eliben @tmc
embeddings: use NewEmbedder for Ernie embeddings (#392)
4bb2840 
For #379

Updates examples go.mod because it's used in an example
tmc pushed a commit that referenced this issue Dec 5, 2023
@eliben @tmc
embeddings: add example of proper usage, and tweak doc.go accordingly (…
357a3b5 
…#393)

* embeddings: add example of proper usage, and tweak doc.go accordingly

For #379

* Appease lint and add comment
@eliben
Copy link
Collaborator Author

eliben commented Dec 6, 2023

#396 is retrofitting TEI into this pattern

@tmc
Copy link
Owner

tmc commented Dec 13, 2023

@eliben was TEI the last instance? are we good to close?

@eliben
Copy link
Collaborator Author

eliben commented Dec 13, 2023

Nope there's still huggingface, I think you and @zivkovicn worked on it (based on git log). It'd be nice to retrofit that into the new pattern as well.

@eliben
Copy link
Collaborator Author

eliben commented Dec 13, 2023

There are a few more details here: #388 (comment)

@tmc
Copy link
Owner

tmc commented Dec 13, 2023

There are a few more details here: #388 (comment)

Ah thanks, I had missed that comment. Closing that issue as we have a tagged release but will leave this issue open.

@eliben eliben mentioned this issue Dec 16, 2023
Open
@eliben
Copy link
Collaborator Author

eliben commented Dec 29, 2023

@tmc and @zivkovicn any chance one of you wants to take a look at the huggingface embeddings implementation to make it work with this refactoring so we can close the issue?

I have almost no experience working with Hugging Face

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tmc @eliben