encoding/base64: decoding is slow

[josselin-c]

What version of Go are you using (go version)?

Go 1.8

What operating system and processor architecture are you using (go env)?

amd64

What did you do?

On my slow computer, using encoding/base64, I can decode data at ~100MB/s.
It should be much faster as shown by https://github.com/aklomp/base64

I'm planning to work on this in my spare time. This issue tracks this effort.

[gopherbot]

CL https://golang.org/cl/34950 mentions this issue.

[dgryski]

Also https://github.com/powturbo/TurboBase64 for a fast encoder/decoder that doesn't need assembly.

[gopherbot]

CL https://golang.org/cl/38632 mentions this issue.

[josselin-c]

@dgryski I saw such LUT based and I'm wondering if it's acceptable. What's go philosophy about embedding "large" LUT? Won't such tables trash the L1 when used and penalize the rest of the application?
I have an SSE implementation in the pipe that will be another improvement to CL 38632 but it's limited to amd64.

[powturbo]

@josselin-c: I saw such LUT based and I'm wondering if it's acceptable.

Strange rumours, but the short scalar "turbob64encs" encoding uses 64 bytes and the "turbob64decs" decoding function just uses 80 bytes (from 256 bytes LUT) delivering 2 GB/s in practical szenarios, This is less memory than other SIMD/AVX2 base64 functions. The fast "turbob64dec" function needs in fact less than 1k and is decoding faster than other scalar/SSE functions. See benchmark: Turbo-Base64