Skip to content

Commit

Permalink
gracefully handle length feature used for group by
Browse files Browse the repository at this point in the history
  • Loading branch information
winglian committed Sep 13, 2023
1 parent e5bb22a commit bb115a6
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions src/axolotl/utils/dataloader.py
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,8 @@ def __iter__(self):
concatenated = {}
batched_data = [self.dataset[batch_idx] for batch_idx in batch]
for feature in features:
if feature == "length":
continue
if feature == "attention_mask":
arrays = [
(attn_mask_cum_idx + idx + 1) * np.array(item[feature])
Expand Down

0 comments on commit bb115a6

Please sign in to comment.