Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong cache hits for LlamaIndexEmbeddingsWrapper #1915

Open
triedel42 opened this issue Feb 11, 2025 · 0 comments
Open

Wrong cache hits for LlamaIndexEmbeddingsWrapper #1915

triedel42 opened this issue Feb 11, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@triedel42
Copy link

triedel42 commented Feb 11, 2025

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
When generating embeddings with the LlamaIndexEmbeddingsWrapper a wrong cache hit leads to all embeddings being the same.
This is due to the _generate_cache_key function removing the first argument for methods. However it seems self is not passed here as an argument, hence valid arguments are sliced away here and every cache key becomes the same.

Ragas version: 0.2.13
Python version: 3.11.2

Code to Reproduce
see this gist

Error trace
N/A (see assertion in example code)

Expected behavior
Distinct embed_text arguments need to result in distinct vectors.
Distinct embed_text arguments need to be cached under distinct cache keys.

Additional context
We noticed this bug when running metrics and receiving implausible results (e.g. SemanticSimilarity always being 1).

This test case did not catch it either, since it's only tested with functions, not methods.

@triedel42 triedel42 added the bug Something isn't working label Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant