Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New output format: Seven-bit coding (-F7) #30

Open
lifthrasiir opened this issue Sep 25, 2021 · 0 comments
Open

New output format: Seven-bit coding (-F7) #30

lifthrasiir opened this issue Sep 25, 2021 · 0 comments
Assignees

Comments

@lifthrasiir
Copy link
Owner

This is an approach used by base-122 and optimal if you are using UTF-8. It recognizes the forbidden code units (in this case, \r, $, `, \ and possibly \0) and encodes it and the following code unit into two-byte UTF-8 sequences:

0xxxxxxx                for allowed code units xx
110xxxyy 10yyyyyz       for forbidden code units xx (indexed) and following code unit yy

It should be noted that two-byte UTF-8 sequences have up to two spare bits to spare.

  • The index to forbidden code units are encoded to 3 bits (the first index 0 should be reserved to make it valid UTF-8), so we definitely have a single unused bit (denoted as z above).
  • Not all indices to forbidden code units would be used. Since coding into two-byte sequence actually saves the space, we can assign otherwise allowed code units into these indices.

We may need a change to ANSEncoder to leverage this spare bit, so it is not as trivial.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant