Downloading from Github Releases sometimes times out

[micky-git]

tokenizer = Tokenizer.load("en")

ValueError: error decoding response body: operation timed out

[bminixhofer]

That's an issue with file hosting via Github Releases. You're the second person to report this, I might check if it's possible to host the files elsewhere.
In the meantime, the workaround mentioned in #6 (comment) works. Download the files manually from the Releases and load them with:

tokenizer = Tokenizer("path/to/en_tokenizer.bin")

[micky-git]

gunzipping failed: Custom { kind: UnexpectedEof, error: "failed to fill whole buffer" }

[bminixhofer]

Can you paste your code? And make sure the version of the binary and the library match, maybe that's the issue.

[micky-git]

version info:
platform:windows64 & mac bigsur
rustc 1.49.0 (e1884a8e3 2020-12-29)
Python 3.7.6
conda 4.9.2
nlprule 0.3.0
tokenizer:https://github.com/bminixhofer/nlprule/releases/download/0.3.0/en_tokenizer.bin.gz
IDE Spyder

code here(python3.7):

# -- coding: utf-8 --
"""
Created on Tue Jan 19 18:41:01 2021
@author: A
"""
from nlprule import Tokenizer, Rules, SplitOn
tokenizer = Tokenizer.load("en_tokenizer.bin")
rules = Rules.load("en", tokenizer, SplitOn([".", "?", "!"]))
rules.correct("He wants that you send him an email.")
rules.correct("Thanks for your’s and Lucy’s help.")
rules.correct("I can due his homework.")
suggestions = rules.suggest("She was not been here since Monday.")
for s in suggestions:
print(s.start, s.end, s.text, s.source, s.message)

output here:

runfile('D:/ai/test/helloworld/test.py', wdir='D:/ai/test/helloworld')
Traceback (most recent call last):

File "D:\ai\test\helloworld\test.py", line 8, in
tokenizer = Tokenizer.load("en_tokenizer.bin")

PanicException: gunzipping failed: Custom { kind: UnexpectedEof, error: "failed to fill whole buffer" }`

[micky-git]

btw:unzip with bandizip(windows_64):http://www.bandisoft.com/

[micky-git]

I tried:
tokenizer = Tokenizer.load("en_tokenizer.bin")#copy en_tokenizer.bin to workpath
tokenizer = Tokenizer.load("en_tokenizer.bin")#copy en_tokenizer.bin to package_nlprule_path

tokenizer = Tokenizer.load("D:\ai\test\helloworld\en_tokenizer.bin")
tokenizer = Tokenizer.load("D:\ai\test\helloworld\en_tokenizer.bin")
tokenizer = Tokenizer.load("D:/ai/test/helloworld/en_tokenizer.bin")

tokenizer = Tokenizer.load(r"D:\ai\test\helloworld\en_tokenizer.bin")
tokenizer = Tokenizer.load(r"D:\ai\test\helloworld\en_tokenizer.bin")
tokenizer = Tokenizer.load(r"D:\ai\test\helloworld\en_tokenizer.bin")

tokenizer = Tokenizer.load(f"D:\ai\test\helloworld\en_tokenizer.bin")
tokenizer = Tokenizer.load(f"D:\ai\test\helloworld\en_tokenizer.bin")
tokenizer = Tokenizer.load(r"D:\ai\test\helloworld\en_tokenizer.bin")

tokenizer = Tokenizer.load(f"en_tokenizer.bin")
tokenizer = Tokenizer.load("../en_tokenizer.bin")

I'm not quite understand the argument that func_load passes

[bminixhofer]

Thanks for the code. So .load takes a language code e.g "en" or "de" as input and downloads the binary automatically. Since this does not seem to work in your case, you can load it manually with the constructor of the Tokenizer, not with .load.

This code should work:

tokenizer = Tokenizer("en_tokenizer.bin") # no .load, just Tokenizer(..)!

[bminixhofer]

The binaries are now significantly smaller so I hope this won't happen anymore. I'm closing this for now, please comment / reopen if there's anymore issues with timeouts.