base64_encode_expected_len() returns value that is not large enough

[jackjansen]

base64_encode_expected_len() (from libb64/cencode.h) returns a number that is too small if it is used to allocate a buffer to subsequently pass to base64_encode_chars().

The reason is that the simple base64_encode_chars() includes a newline every 72 characters or so, and these newlines are not catered for in base64_encode_expected_len().

The result of this is that if you try to encode a binary blob larger than about 60 characters you will write past the end of your malloced space.

[d-a-v]

Can you verify if this change in cores/esp8266/libb64/cencode.h fixes the buffer overflow ?

#define base64_encode_expected_len_nonewlines(n) ((((4 * n) / 3) + 3) & ~3)
#define base64_encode_expected_len(n) (base64_encode_expected_len_nonewlines(n) + ((n / 18) + 1))

[jackjansen]

The returned number is now too high. For example, in my test case code the real encoded length will be 823 bytes, but base64_encode_expected_len() returns 846.

I've changed the number 18 to 54 (being CHARS_PER_LINE divided by 4, the multiplied by 3) and then 824 is returned, which is what is expected.

[devyte]

Relevant PR is merged, closing.