Add unsupervised-qa pipelines #3605
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
westfish
force-pushed
the
unsupervised_qa_pipelines
branch
from
d2e2b24 to
b6c3589
Compare
wawltor
reviewed
Nov 1, 2022
| + 可控性好，合成语料和语义检索解耦合，可以人工筛查和删除合成的问答对，也可以添加人工标注的问答对 | ||
| + 端到端 | ||
| + 提供包括问答语料生成、模型服务部署、WebUI可视化一整套端到端智能问答系统能力 | ||
| + 多源数据支持: 支持对 Txt、Word、PDF、Image 多源数据进行解析、识别并写入 ANN 数据库 |
There was a problem hiding this comment.
等高升升级了相关pipeline之后接入
There was a problem hiding this comment.
最新的commit已经支持上传文档并自动解析
pipelines/examples/unsupervised-question-answering/unsupervised_question_answering_example.py
Outdated
Show resolved
Hide resolved
wawltor
reviewed
Nov 6, 2022
|
|
||
| # 环境变量设置 | ||
| export PYTHONPATH=/root/project/paddle/paddlenlp/unsupervised_qa_pipelines/PaddleNLP/pipelines:$PYTHONPATH | ||
| export CUDA_VISIBLE_DEVICES=1 |
|
|
||
| # 环境变量设置 | ||
| unset http_proxy && unset https_proxy | ||
| export CUDA_VISIBLE_DEVICES=1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Models
Description
新增无监督问答pipelines，主要包含以下内容：
-无监督问答pipelines说明文档
-pipeline运行示例 examples/unsupervised-question-answering/unsupervised_question_answering_example.py
-离线生成问答对脚本
-新增节点QAFilter、AnswerExtractor、QuestionGenerator、AnswerExtractorPreprocessor（用于支持文档操作）、QAFilterPostprocessor（用于支持文档操作）
-新增pipeline QAGenerationPipeline
-FastAPI后端代码，承接ElasticSearch ANN检索库、QAGenerationPipeline和SemanticSearchPipeline
-streamlit无监督问答UI前端方案
WEB可视化系统功能：
问答检索
在线问答对生成
在线更新索引库
文件上传并自动生成和载入问答对
问答对生成可选择过滤
问答检索可选择返回答案数量和最大检索数量
效果展示：
