You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is an approach used by base-122 and optimal if you are using UTF-8. It recognizes the forbidden code units (in this case, \r, $, `, \ and possibly \0) and encodes it and the following code unit into two-byte UTF-8 sequences:
0xxxxxxx for allowed code units xx
110xxxyy 10yyyyyz for forbidden code units xx (indexed) and following code unit yy
It should be noted that two-byte UTF-8 sequences have up to two spare bits to spare.
The index to forbidden code units are encoded to 3 bits (the first index 0 should be reserved to make it valid UTF-8), so we definitely have a single unused bit (denoted as z above).
Not all indices to forbidden code units would be used. Since coding into two-byte sequence actually saves the space, we can assign otherwise allowed code units into these indices.
We may need a change to ANSEncoder to leverage this spare bit, so it is not as trivial.
The text was updated successfully, but these errors were encountered:
This is an approach used by base-122 and optimal if you are using UTF-8. It recognizes the forbidden code units (in this case,
\r
,$
,`
,\
and possibly\0
) and encodes it and the following code unit into two-byte UTF-8 sequences:It should be noted that two-byte UTF-8 sequences have up to two spare bits to spare.
z
above).We may need a change to ANSEncoder to leverage this spare bit, so it is not as trivial.
The text was updated successfully, but these errors were encountered: