Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix whisper tokenizer saving #1334

Merged
merged 1 commit into from
Nov 30, 2023
Merged

Fix whisper tokenizer saving #1334

merged 1 commit into from
Nov 30, 2023

Conversation

mattdangerw
Copy link
Member

@mattdangerw mattdangerw commented Nov 29, 2023 •
edited
Loading

This is an initial pass for whisper focused on preserving all the existing from_preset(id) usages.

We should answer more questions long term:

  • Do we want to allow loading a AudioFeatureExtractor from_preset() with a custom json saved to the preset dir? How should we go about scaling our xx.json names for stuff like this?
  • Should whisper tokenizer save it's special tokens and language tokens as additional assets? Or can we fold all this into the main vocab?

@mattdangerw mattdangerw requested a review from nkovela1 November 29, 2023 23:35
@mattdangerw
Copy link
Member Author

/gcbrun

nkovela1
nkovela1 approved these changes Nov 30, 2023
View reviewed changes
Copy link
Collaborator

@nkovela1 nkovela1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should revisit this in the future (for the audio preprocessor), but LGTM for now!

@mattdangerw mattdangerw merged commit 3cdf449 into keras-team:kaggle Nov 30, 2023
mattdangerw added a commit that referenced this pull request Dec 7, 2023
@mattdangerw
Fix whisper tokenizer saving (#1334)
a3f5b60
mattdangerw added a commit that referenced this pull request Dec 7, 2023
@mattdangerw
Fix whisper tokenizer saving (#1334)
9b9374f
mattdangerw added a commit that referenced this pull request Dec 8, 2023
@mattdangerw
Fix whisper tokenizer saving (#1334)
086910e
sampathweb pushed a commit that referenced this pull request Dec 12, 2023
@mattdangerw @sampathweb
Fix whisper tokenizer saving (#1334)
f4058e1
mattdangerw added a commit that referenced this pull request Jan 4, 2024
@mattdangerw
Fix whisper tokenizer saving (#1334)
c4b0c3c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nkovela1 nkovela1 nkovela1 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mattdangerw @nkovela1