update from_pretrained method

[not-lain]

fixes #14
a bit of a hacky approach since i'm using wrappers & kwargs instead of the original parameter names in the from_pretrained but let me know if you approve of this or if we should pass the parameters to the methods as well.

still very useful when running

help(AutoTikTokenizer.from_pretrained) 

[bhavnicksm]

Hey @not-lain!

Thanks for opening a PR!

Could you explain what exactly is happening in the code here? I see **kwargs being passed around, but not sure what are some of the parameters these arguments could potentially have.

Or if you could point me to some link that could explain kwargs that are important to have, that would also be great.

Thanks!

[not-lain]

sure thing, let's consider the following minimalistic function

# expects keyword arguments
def placeholder(**kwargs) : 
	print(kwargs)


placeholder(a=5, b="hello" ) 
>>> {'a': 5, 'b': 'hello'}

basically kwargs expects ulimited keyword arguments and stores them in a dictionary

[bhavnicksm]

sure thing, let's consider the following minimalistic function

# expects keyword arguments
def placeholder(**kwargs) : 
	print(kwargs)


placeholder(a=5, b="hello" ) 
>>> {'a': 5, 'b': 'hello'}

basically kwargs expects ulimited keyword arguments and stores them in a dictionary

Hey @not-lain! 😆

Thanks for the explaination~ What I meant was, were there any specific kwargs we wanted to support? Sorry, that wasn't clear earlier...

Thanks!

[not-lain]

Hi @bhavnicksm
I think the top ones would be :

• token
• cache_dir & local_dir
• force_download (useful in case of internet failures or corrupted files or something)

let me know if you approve of these parameters or you have any other recommendations in mind and i'll update the PR accordingly