Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SPLADE++ ED w/ ONNX on BEIR #2354

Merged
merged 7 commits into from
Feb 12, 2024
Merged

SPLADE++ ED w/ ONNX on BEIR #2354

merged 7 commits into from
Feb 12, 2024

Conversation

lintool
Copy link
Member

@lintool lintool commented Jan 25, 2024

Here's the snippet for running everything from scratch:

MODEL="splade-pp-ed-onnx"; CORPORA=(trec-covid bioasq nfcorpus nq hotpotqa fiqa signal1m trec-news robust04 arguana webis-touche2020 cqadupstack-android cqadupstack-english cqadupstack-gaming cqadupstack-gis cqadupstack-mathematica cqadupstack-physics cqadupstack-programmers cqadupstack-stats cqadupstack-tex cqadupstack-unix cqadupstack-webmasters cqadupstack-wordpress quora dbpedia-entity scidocs fever climate-fever scifact); for c in "${CORPORA[@]}"
do
    echo "Running $c..."
    python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-${c}-${MODEL} > logs/log.beir-v1.0.0-${c}-${MODEL} 2>&1
done

Everything works except for:

python src/main/python/run_regression.py --index --search --regression beir-v1.0.0-arguana-splade-pp-ed-onnx

@ArthurChen189 can you please confirm repo of error? and then try to fix?

@lintool lintool marked this pull request as draft January 25, 2024 22:38
Copy link

codecov bot commented Jan 25, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f86a65f) 65.66% compared to head (2fd6544) 65.66%.

Additional details and impacted files
@@            Coverage Diff            @@
##             master    #2354   +/-   ##
=========================================
  Coverage     65.66%   65.66%           
  Complexity     1398     1398           
=========================================
  Files           207      207           
  Lines         11631    11631           
  Branches       1471     1471           
=========================================
  Hits           7637     7637           
  Misses         3484     3484           
  Partials        510      510           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ArthurChen189 ArthurChen189 self-assigned this Jan 26, 2024
@lintool lintool reopened this Feb 9, 2024
@lintool lintool marked this pull request as ready for review February 12, 2024 14:04
@lintool
Copy link
Member Author

lintool commented Feb 12, 2024

Everything works except for arguana (still), which @ArthurChen189 is looking into. I'm going to merge in this PR since we need it regardless; @ArthurChen189 can work on fixing the bug in parallel.

@lintool lintool merged commit 9a5bb60 into master Feb 12, 2024
4 checks passed
@lintool lintool deleted the splade-onnx branch February 12, 2024 16:39
@carlos-lassance
Copy link

@ArthurChen189 arguana could be a max length on the query side? We used 256 on both query and doc from what I remember.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants