-
Notifications
You must be signed in to change notification settings - Fork 84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support loading tokenizer from local folder #76 #81
Support loading tokenizer from local folder #76 #81
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice, thanks a lot for working on this! 🙌
This is in very good shape, I only have a couple of suggestions and we can merge.
Maybe we should rename LanguageModelConfigurationFromHub
in a future PR.
Sources/Hub/HubApi.swift
Outdated
/// Assumes the file has already present at local url. | ||
/// `url` is complete local file path for given model |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// Assumes the file has already present at local url. | |
/// `url` is complete local file path for given model | |
/// Assumes the file is already present at local url. | |
/// `url` is a complete local file path for the given model |
Sources/Hub/HubApi.swift
Outdated
|
||
/// Assumes the file has already present at local url. | ||
/// `url` is complete local file path for given model | ||
func configuration(url: URL) throws -> Config { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe call the parameter fileURL
to make it clear?
08d4b56
to
800353c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! I'll try to trigger a CI run before we merge.
Something's up with the CI, but the tests pass locally so I'll merge. Thanks a lot @vinu-vanjari! |
Resolves #76
Added new method in
AutoTokenizer
to load from local model folder.Also added supporting unit tests at
Tests/TokenizersTests/FactoryTests.swift
Although we could have simply loaded
tokenizer_config.json
andtokenizer.json
files directly into configurations, I chose to go viaLanguageModelConfigurationFromHub
because it has it's own magic for choosing appropriate tokenizer class and it also selects fallback tokenizer if needed refer getter var tokenizerConfig: Config?