Skip to content
This repository has been archived by the owner on Jan 15, 2024. It is now read-only.

Add sorting of chunks to evaluation #1588

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

bartekkuncer
Copy link

Description

This change introduces sorting of chunks before executing evaluation to reduce padding to minimum and in this way improve performance.

How the change works

As every input feature has unique qas_id it can be used for sorting. With the sorting evaluation function goes like this:

  1. sort input features by qas_id
  2. chunk data
  3. sort chunks by their length (to reduce padding to minimum)
  4. perform inference
  5. sort chunks and results by qas_id
  6. evaluate data

Step number 1 is performed so that chunks and their inference results can be easily put in proper order in step number 5 for evaluation in step 6.

Performance

Results for max_seq_length=128, doc_stride=32:
no sort:
image

sorted:
image

Performance did not improve much due to most of the chunks being of same 128 length due to relatively small values of max_seq_length and doc_stride.

Results for max_seq_length=512, doc_stride=128 (default values in run_squad.py script):
no sort:
image

sorted:
image

As you can see the performance improved significantly (~20%) without any loss of accuracy.

cc @dmlc/gluon-nlp-team

@bartekkuncer bartekkuncer requested a review from a team as a code owner April 9, 2022 00:03
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants