How do you represent bencoded strings that contain a `<hex>` tag to avoid ambiguity?

[josecelano]

Relates to: Chocobo1/bencode_online#3

How do you represent bencoded strings that contain a tag to avoid ambiguity with the tags introduced by not utf-8 bencoded strings?

Submitted on reddit by Icarium-Lifestealer

UPDATE

Example, two bencoded values producing the same output:

printf "13:<hex>ff</hex>" | cargo run -> "<hex>ff</hex>"  (13 bytes)
printf "1:\xff"           | cargo run -> "<hex>ff</hex>"  (1 byte)

List of special characters used in JSON that need to be escaped in string values:

• \b Backspace (ascii code 08)
• \f Form feed (ascii code 0C)
• \n New line
• \r Carriage return
• \t Tab
• \" Double quote
• \\ Backslash character

[josecelano]

Proposal 1

Maybe we can always include some tags. For byte sequences containing valid UTF-8:

"<utf8>spam</utf8>"

And not UTF-8 sequences:

"<bytes>fffe</bytes>"

The encoded value 6:<utf8> would be "<utf8><utf8></utf8>".

[josecelano]

Proposal 2

"<string>spam</string>"

And not UTF-8 sequences:

"<hex>fffe</hex>"

This would be half compatible with the other implementation.

[josecelano]

Proposal 3

Maybe we can event simplify the metadata using just a prefix instead of a html-style tag:

"string:spam"

And not UTF-8 sequences:

"hex:fffe"

[josecelano]

Hey, @da2ce7, can you add an example of what we discussed today?

I mean one example like this:

printf "13:<hex>ff</hex>" | cargo run -> "<hex>ff</hex>"  (13 bytes)
printf "1:\xff"           | cargo run -> "<hex>ff</hex>"  (1 byte)

For the problem with escaped chars you mentioned in the meeting. I can't come up with one example.

[josecelano]

Hey, @da2ce7, can you add an example of what we discussed today?

I mean one example like this:

printf "13:<hex>ff</hex>" | cargo run -> "<hex>ff</hex>"  (13 bytes)
printf "1:\xff"           | cargo run -> "<hex>ff</hex>"  (1 byte)

For the problem with escaped chars you mentioned in the meeting. I can't come up with one example.

Hi @da2ce7, if you have a concrete example for this ☝🏼 that would be awesome! Thanks.

[da2ce7]

The problem is that it is lossy transformation.

There are multiple bencode inputs that can produce identical json results.

The issue is if we escape XML style tags, then there is two inputs that produces the same output.

The <\/> input and </> inputs produce the same <\/> output.

So we don't know if the bencode input was "pre-escaped" or not.

[josecelano]

The problem is that it is lossy transformation.

There are multiple bencode inputs that can produce identical json results.

The issue is if we escape XML style tags, then there is two inputs that produces the same output.

The <\/> input and </> inputs produce the same <\/> output.

So we don't know if the bencode input was "pre-escaped" or not.

Hi @da2ce7 forward slash does not need to be escaped in JSON. The two examples you mention <\/> and </> produce a different JSON value:

printf "3:</>"  | cargo run -> "</>"
printf "4:<\/>" | cargo run -> "<\\/>"

For the backslash (hex 5C), it's escaped but I guess we can revert it without ambiguity. We only need to undo escaped chars in the JSON value before converting them into bytes, right?

printf "1:\x5c" | cargo run -> "\\"

[da2ce7]

I think that you are correct. If we always insert the escaping character, then it will have a reversible encoding
. 👍

[josecelano]

I think that you are correct. If we always insert the escaping character, then it will have a reversible encoding . 👍

Hey @da2ce7 and what proposal do you prefer? I prefer the proposal 2

"<string>spam</string>"

"<hex>ff</hex>"

Because:

• It's half-compatible with the other implementation.
• It's extensible, we can add more tags later like:

"<hex>ff</hex><metada>...</medatada>"

as long as we include <hex>...</hex> at the "root" level of this kind of HTML doc.

With proposal 3, we can only add new prefixes.

[josecelano]

@da2ce7 has just told me that proposal 2 is fine.

[josecelano]

After merging this change, we have to update the draft TEP.

[josecelano]

After merging this change, we have to update the draft TEP.

And also bump the version to 0.2.0 and publish it on crates.io.