Buffer.from(string, 'hex') needs a documentation update (was: Buffer.from(x, hex).toString(hex) does not return x for certain values of x)

[iamrenejr]

• Version: 10.16.3
• Platform: Darwin Renes-MacBook-Pro.local 18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64
• Subsystem: Buffer

Run these in the REPL:

 Buffer.from('0', 'hex').toString('hex')               // returns ''
 Buffer.from('08', 'hex').toString('hex')              // returns '08'
 Buffer.from('088', 'hex').toString('hex')             // returns '08'
 Buffer.from('0888', 'hex').toString('hex')            // returns '0888'
 Buffer.from('08888', 'hex').toString('hex')           // returns '0888'
 Buffer.from('088888', 'hex').toString('hex')          // returns '088888'

I expected .toString('hex') to be the inverse operation of Buffer.from(x, 'hex'), but it doesn't seem to be the case. Why not?

I was able to catch this error using fast-check and it found a few other strings that behave this way:

• 8501eb788
• eb788
• 788

[Trott]

All of your problematic examples have an odd number of characters. Buffer.from() can't know how your hex-encoded string should be treated if it has an odd number of characters. Hex encoding means each byte is encoded as two hexadecimal characters. If you only provide one character instead, I imagine behavior is undefined, although I don't actually know. Surprisingly, I can't find mention of this in the documentation, so there may be a documentation pull request to be done here. A case can be made that this ought to throw an error but I wouldn't want to make that case before updating the docs.

People file issues like this from time to time which suggests that a doc update is in order. Here's a previous example: #24491

[Trott]

Added hacktoberfest and good first issue labels in case someone wants to try their hand at updating the relevant documentation.

[AdityaSrivast]

@Trott I would like to work on this issue

[Trott]

@Trott I would like to work on this issue

Go for it.

[reneklootwijk]

All of your problematic examples have an odd number of characters. Buffer.from() can't know how your hex-encoded string should be treated if it has an odd number of characters. Hex encoding means each byte is encoded as two hexadecimal characters. If you only provide one character instead, I imagine behavior is undefined, although I don't actually know. Surprisingly, I can't find mention of this in the documentation, so there may be a documentation pull request to be done here. A case can be made that this ought to throw an error but I wouldn't want to make that case before updating the docs.

People file issues like this from time to time which suggests that a doc update is in order. Here's a previous example: #24491

Then it is very inconsistent:
let n = 1000
console.log(n.toString(16)) => results in 3e8
console.log(Buffer.from(n.toString(16), 'hex')) => results in <Buffer 3e>

[Trott]

Then it is very inconsistent:
let n = 1000
console.log(n.toString(16)) => results in 3e8
console.log(Buffer.from(n.toString(16), 'hex')) => results in <Buffer 3e>

I suppose that suggests another possibility: add a leading 0 if needed so that '3e8' is treated like '03e8'. In addition to being a breaking change, that gets challenging for edge cases and may end up affecting performance. You wouldn't want to pad '3ez' but you would want to pad '3e8'. Things probably get weird fast there. @nodejs/buffer