GiNZA >= 5.1 cannot process long (over 49149 bytes) texts

[TatsuyaShirakawa]

This is essentially due to sudachi.rs's limitation but texts over 49149 bytes cannot be processed by GiNZA >= 5.1.

According to the sudachi.rs's code, the maximum text length (in bytes) is defined as u16::MAX / 4 * 3 (= 49149), so if a given text is longer than this size in bytes, sudachipy (sudachi.rs) raises an InputTooLong error.

Here is the related lines in the sudachi.rs's repo, which might help.

• https://github.com/WorksApplications/sudachi.rs/blob/c7d20b22c68bb3f6585351847ae91bc9c7a61ec5/sudachi/src/input_text/buffer/mod.rs#L32
• https://github.com/WorksApplications/sudachi.rs/blob/c7d20b22c68bb3f6585351847ae91bc9c7a61ec5/sudachi/src/input_text/buffer/mod.rs#L125
• https://github.com/WorksApplications/sudachi.rs/blob/c7d20b22c68bb3f6585351847ae91bc9c7a61ec5/sudachi/src/input_text/buffer/mod.rs#L244

(I personally asked sudachi's developpers about this limitation and they gave me a feedback that the max length (u16::MAX / 4 * 3) is chosen for performance.)