Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pinecone support for dataprep and retrieval microservice #157

Merged
merged 24 commits into from
Jun 26, 2024
Merged

Pinecone support for dataprep and retrieval microservice #157

merged 24 commits into from
Jun 26, 2024

Conversation

pallavijaini0525
Copy link
Contributor

Description

Added the pinecone vector database-related config and docker files in data prep and retrieval microservice.

Issues

n/a

Type of change

New Feature

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds new functionality)
  • Breaking change (fix or feature that would break existing design and interface)

Dependencies

pinecone-client
langchain-pinecone

Tests

Describe the tests that you ran to verify your changes.

pallavijaini0525 and others added 19 commits June 14, 2024 19:15
@pallavijaini0525
Added all teh files for the pinecone
ca37461 
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@pre-commit-ci @pallavijaini0525
[pre-commit.ci] auto fixes from pre-commit.com hooks
03310ec 
for more information, see https://pre-commit.ci

Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@pallavijaini0525
Corrected the path of retriever docker files
90886fc 
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@pallavijaini0525
Adding more files
7785612 
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@pallavijaini0525
Corrected the data ingestion path
17464b7 
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@lvliang-intel @pallavijaini0525
Fix dataprep microservice path issue (opea-project#163)
54a7d81 
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@chensuyue @pallavijaini0525
update CI to support dataprep_redis path level change (opea-project#155)
cb687d3 
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@zehao-intel @pre-commit-ci @pallavijaini0525
Add Gateway for Translation (opea-project#169)
f69cf84 
* add translation gateway

Signed-off-by: zehao-intel <zehao.huang@intel.com>

* fix import

Signed-off-by: zehao-intel <zehao.huang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: zehao-intel <zehao.huang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@lvliang-intel @pre-commit-ci @pallavijaini0525
Update LLM readme (opea-project#172)
b1b820d 
* Update LLM readme

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* update readme

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* update tgi readme

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* rollback requirements.txt

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@jinjunzh @chensuyue
@pre-commit-ci @pallavijaini0525
add milvus microservice (opea-project#158)
10d0cb7 
* Use common security content for OPEA projects (opea-project#151)

* add python coverage

Signed-off-by: chensuyue <suyue.chen@intel.com>

* docs update

Signed-off-by: chensuyue <suyue.chen@intel.com>

* Revert "add python coverage"

This reverts commit 69615b1.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: chensuyue <suyue.chen@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: jinjunzh <jasper.zhu@intel.com>

* add milvus microservice

Signed-off-by: jinjunzh <jasper.zhu@intel.com>

* fix the typo

Signed-off-by: jinjunzh <jasper.zhu@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: jinjunzh <jasper.zhu@intel.com>

---------

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: jinjunzh <jasper.zhu@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@chensuyue @pallavijaini0525
enable python coverage (opea-project#149)
42aa78b 
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@xuechendi @pre-commit-ci @pallavijaini0525
Add Ray version for multi file process (opea-project#119)
0e701ac 
* add ray version document to redis

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

* update test

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

* Add test

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add TIMEOUT in container environment and return status

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

* rebase on new folder layout

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@XuehaoSun @pre-commit-ci @pallavijaini0525
Add codecov (opea-project#178)
de0a209 
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@changwangss @pallavijaini0525
Rename lm-eval folder to utils/lm-eval (opea-project#179)
77e13ae 
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@XinyuYe-Intel @pre-commit-ci @pallavijaini0525
Rebased the changes
493a35e 
Support rerank and retrieval of RAG OPT (opea-project#164)

* supported bce model for rerank.

Signed-off-by: Xinyu Ye <xinyu.ye@intel.com>

* change folder

Signed-off-by: Xinyu Ye <xinyu.ye@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change path in test file.

Signed-off-by: Xinyu Ye <xinyu.ye@intel.com>

---------

Signed-off-by: Xinyu Ye <xinyu.ye@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@lvliang-intel @pre-commit-ci @pallavijaini0525
Support vLLM XFT LLM microservice (opea-project#174)
c3a5032 
* Support vLLM XFT serving

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix access vllm issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add permission for run.sh

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* add readme

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix proxy issue

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@letonghan @pre-commit-ci @pallavijaini0525
Update Dataprep Microservice README (opea-project#173)
c707a31 
* update dataprep readme

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@pallavijaini0525
Rebased the files
15ee89f 
Added all teh files for the pinecone

Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@pre-commit-ci @pallavijaini0525
[pre-commit.ci] auto fixes from pre-commit.com hooks
7d6151a 
for more information, see https://pre-commit.ci

Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
pallavijaini0525 and others added 2 commits June 14, 2024 19:19
@pallavijaini0525
fixed the conflicts
eba8970 
Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
@lvliang-intel
Merge branch 'main' into pinecone_dataprep
067dd4b
@chensuyue
Copy link
Collaborator

chensuyue commented Jun 24, 2024 •
edited
Loading

Require test scripts test_vectorstores_langchain_pinecone.sh, test_retrievers_langchain_pinecone.sh, test_dataprep_pinecone.sh, like this one https://github.com/opea-project/GenAIComps/blob/main/tests/test_retrievers_langchain_pgvector.sh

@chensuyue chensuyue added this to the v0.7 milestone Jun 24, 2024
@chensuyue
Copy link
Collaborator

Require test scripts test_vectorstores_langchain_pinecone.sh, test_retrievers_langchain_pinecone.sh, test_dataprep_pinecone.sh, like this one https://github.com/opea-project/GenAIComps/blob/main/tests/test_retrievers_langchain_pgvector.sh

@pallavijaini0525 please add the e2e test scripts.

@pallavijaini0525
Copy link
Contributor Author

@chensuyue -> Since Pinecone functions as a managed service and operates through APIs, I'm utilizing the Pinecone token to access its hosted vector database. Given that these tests are authentic functional tests, I'll need to pass my Pinecone token by configuring an environment variable and then integrating it into the following tests:

test_vectorstores_langchain_pinecone.sh
test_retrievers_langchain_pinecone.sh
test_dataprep_pinecone.sh

any suggestions for this usecase?

@chensuyue
Copy link
Collaborator

chensuyue commented Jun 26, 2024 •
edited
Loading

I can pass a secrets for the test, like this one, it won't be shown in the test scripts. You can send me by teams.
image

And we can merge this PR first and add the test in another PR.

@chensuyue chensuyue merged commit 8b6486b into opea-project:main Jun 26, 2024
9 checks passed
@pallavijaini0525 pallavijaini0525 deleted the pinecone_dataprep branch June 26, 2024 15:32
sharanshirodkar7 pushed a commit to sharanshirodkar7/GenAIComps that referenced this pull request Jul 9, 2024
@pallavijaini0525 @sharanshirodkar7
Pinecone support for dataprep and retrieval microservice (opea-projec…
286b231 
…t#157)

Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>
yogeshmpandey pushed a commit to yogeshmpandey/GenAIComps that referenced this pull request Jul 10, 2024
@pallavijaini0525 @yogeshmpandey
Pinecone support for dataprep and retrieval microservice (opea-projec…
d73b7e1 
…t#157)

Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
Signed-off-by: Yogesh Pandey <yogesh.pandey@intel.com>
dwhitena pushed a commit to predictionguard/GenAIComps that referenced this pull request Jul 24, 2024
@pallavijaini0525 @dwhitena
Pinecone support for dataprep and retrieval microservice (opea-projec…
0e52862 
…t#157)

Signed-off-by: Pallavi Jaini <pallavi.jaini@intel.com>
Signed-off-by: Daniel Whitenack <whitenack.daniel@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lvliang-intel lvliang-intel lvliang-intel approved these changes

@chensuyue chensuyue chensuyue approved these changes

@XinyuYe-Intel XinyuYe-Intel Awaiting requested review from XinyuYe-Intel

@letonghan letonghan Awaiting requested review from letonghan

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.7
Development

Successfully merging this pull request may close these issues.
10 participants
@pallavijaini0525 @chensuyue @lvliang-intel @zehao-intel @jinjunzh @xuechendi @XuehaoSun @changwangss @XinyuYe-Intel @letonghan