Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can not read yttm trained model #8

Open
fkurushin opened this issue Mar 7, 2024 · 0 comments
Open

can not read yttm trained model #8

fkurushin opened this issue Mar 7, 2024 · 0 comments

Comments

@fkurushin
Copy link

fkurushin commented Mar 7, 2024

HI!

I want to read previously trained bpe model using yttm python module, with your lib, this is my code snippet:

bpeModel, err := os.Open("data/yttm.model")
reader := bufio.NewReader(bpeModel)

m, err := bpe.ReadModel(reader)
if err != nil {
print(fmt.Errorf("failed to read model %v", err))
}

config := bpe.NewConfig(false, false, false)
fmt.Println(m.EncodeSentence("мама мыла раму", *config))

and I have Broken input: unexpected EOF. This error appears on this line of code

logrus.Error("Broken input: ", err)

As I understood there may be some incompatibles between yttm model dump file extension and you ReadModel method? Have I train the model with some specific parameters or load it in go with some extra parameters?

P.S. I also need to mention that I encountered an import error while loading the configuration, so I implemented the NewConfig method to resolve it.

Thanks a lot

UPD: as I understood yttm load model as a txt file of char2id and then merges, but your package loads as a binary, so they are incompatible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant