Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG FIX: LVM security fix #572

Merged
merged 4 commits into from
Aug 28, 2024
Merged

BUG FIX: LVM security fix #572

merged 4 commits into from
Aug 28, 2024

Conversation

BaoHuiling
Copy link
Collaborator

Description

Fix the security issue in video-llama LVM.

Issues

List the issue or RFC link this PR is working on. If there is no such link, please mark it as n/a.
https://github.com/opea-project/GenAIComps/security/code-scanning/6
https://github.com/opea-project/GenAIComps/security/code-scanning/7
https://github.com/opea-project/GenAIComps/security/code-scanning/8

Type of change

List the type of change like below. Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds new functionality)
  • Breaking change (fix or feature that would break existing design and interface)
  • Others (enhancement, documentation, validation, etc.)

Dependencies

List the newly introduced 3rd party dependency if exists.

Tests

Describe the tests that you ran to verify your changes.

BaoHuiling and others added 2 commits August 27, 2024 17:14
@BaoHuiling
add url validator
05167dd 
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
6a948f0 
for more information, see https://pre-commit.ci
@lvliang-intel lvliang-intel merged commit 3e548f3 into opea-project:main Aug 28, 2024
10 checks passed
a32543254 pushed a commit to a32543254/GenAIComps that referenced this pull request Sep 3, 2024
@BaoHuiling @pre-commit-ci @a32543254
BUG FIX: LVM security fix (opea-project#572)
29a5234 
* add url validator

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation for video_url

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>
sharanshirodkar7 pushed a commit to predictionguard/pg-GenAIComps that referenced this pull request Sep 3, 2024
@BaoHuiling @pre-commit-ci @sharanshirodkar7
BUG FIX: LVM security fix (opea-project#572)
46a8e2c 
* add url validator

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation for video_url

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
lvliang-intel added a commit that referenced this pull request Sep 10, 2024
@a32543254 @pre-commit-ci
@Spycsh @BaoHuiling @ZePan110 @lkk12014402 @sharanshirodkar7 @lvliang-intel @letonghan @chensuyue @tileintel @berkecanrizai @Liangyx2 @kevinintel
optimize rerank with backend ref (#579)
d76751a 
* add rerank with neural speed

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the code

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the code

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix mismatched response format w/wo streaming guardrails (#568)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Fix guardrails out handle logics for space linebreak and quote (#571)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* debug

* debug

* debug

* fix pre-space and linebreak

* fix pre-space and linebreak

* fix single/double quote

* fix single/double quote

* remove debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* BUG FIX: LVM security fix (#572)

* add url validator

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation for video_url

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Modify output messages. (#569)

* Reduced output.

Signed-off-by: zepan <ze.pan@intel.com>

* Output the location where the modified Dockerfile file is referenced.

Signed-off-by: zepan <ze.pan@intel.com>

* for test

Signed-off-by: zepan <ze.pan@intel.com>

* Restore test file.

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* refine logging code. (#559)

* add ut and refine logging code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update microservice port.

---------

Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* adding lancedb to langchain vectorstores (#291)

* adding lancedb to langchain vectorstores

Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Refine Dataprep Milvus MS (#570)

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* final version

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update the readme

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the sign

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix error for pre ci

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the ut

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update docker file

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update CI test log achieve (#577)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Multimodal dataprep (#575)

* multimodal embedding for MM RAG for videos

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* develop data prep first commit

Signed-off-by: Tiep Le <tiep.le@intel.com>

* develop dataprep microservice for multimodal data

Signed-off-by: Tiep Le <tiep.le@intel.com>

* multimodal langchain for dataprep

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update README

Signed-off-by: Tiep Le <tiep.le@intel.com>

* cosmetic

Signed-off-by: Tiep Le <tiep.le@intel.com>

* test for multimodal dataprep

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cosmetic update

Signed-off-by: Tiep Le <tiep.le@intel.com>

* remove langsmith

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update API to remove /dataprep from API names and remove langsmith

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update test

Signed-off-by: Tiep Le <tiep.le@intel.com>

* update the error message per PR reviewer

Signed-off-by: Tiep Le <tiep.le@intel.com>

---------

Signed-off-by: Tiep Le <tiep.le@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add: Pathway vector store and retriever as LangChain component (#342)

* nb

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* init changes

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* example data

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update, add commands

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: formatting, data sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused parts

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: image name, compose env vars

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused part

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: logging name

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: env var

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rename pw docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update input sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* nb

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* init changes

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: formatting, data sources

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rm unused part

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* fix: rename pw docker

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: mv vector store, naming, clarify instructions, improve ingestion components

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests: add pw retriever test
fix: update docker to include libmagic

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* implement suggestions from review, entrypoint, reqs, comments, https_proxy.

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: update docker tags in test and readme

Signed-off-by: Berke <berkecanrizai1@gmail.com>

* tests: add separate pathway vectorstore test

Signed-off-by: Berke <berkecanrizai1@gmail.com>

---------

Signed-off-by: Berke <berkecanrizai1@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Add local Rerank microservice for VideoRAGQnA (#496)

* initial commit

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* save

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* add readme, test script, fix bug

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* update video URL

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* use default

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update core dependency

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use p 5000

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* use 5037

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* update ctnr name

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* remove langsmith

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add rerank algo desc in readme

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Add Scan Container. (#560)

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix SearchedMultimodalDoc in docarray (#583)

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update image build yaml (#529)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add microservice for intent detection (#131)

* add microservice for intent detection

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update license copyright

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* add ut

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* refine

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update folder

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix test

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>

---------

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Make the scanning method optional. (#580)

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add code owners (#586)

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* remove revision for tei (#584)

Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* Bug fix (#591)

* Check if the document exists.

Signed-off-by: zepan <ze.pan@intel.com>

* Add flag output.

Signed-off-by: zepan <ze.pan@intel.com>

* Modify nginx readme.

Signed-off-by: zepan <ze.pan@intel.com>

* Modify document detection logic

Signed-off-by: zepan <ze.pan@intel.com>

---------

Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* fix ut issue

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* merge the main

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* align with new pipeline

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* align with newest pipeline

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* upload code

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* update the ut

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add docker path

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

* add the docker path

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>

---------

Signed-off-by: Dong, Bo1 <bo1.dong@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: zepan <ze.pan@intel.com>
Signed-off-by: sharanshirodkar7 <ssharanshirodkar7@gmail.com>
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Tiep Le <tiep.le@intel.com>
Signed-off-by: Berke <berkecanrizai1@gmail.com>
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Co-authored-by: Huiling Bao <huiling.bao@intel.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com>
Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: Sharan Shirodkar <91109427+sharanshirodkar7@users.noreply.github.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Tiep Le <97980157+tileintel@users.noreply.github.com>
Co-authored-by: berkecanrizai <63911408+berkecanrizai@users.noreply.github.com>
Co-authored-by: Liangyx2 <yuxiang.liang@intel.com>
Co-authored-by: kevinintel <hanwen.chang@intel.com>
lkk12014402 pushed a commit that referenced this pull request Sep 19, 2024
@WenjiaoYue @pre-commit-ci
Add VisualQnA UI (#572)
923cf69 
* update VisualQnA front_end

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update ViualQnA into the ui folder

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

* Add dockerFile

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>

---------

Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lvliang-intel lvliang-intel lvliang-intel approved these changes

@ZePan110 ZePan110 ZePan110 approved these changes

@yunowo yunowo yunowo approved these changes

@chensuyue chensuyue Awaiting requested review from chensuyue

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@BaoHuiling @yunowo @ZePan110 @lvliang-intel