Better support for binary data

[na--]

We currently don't have a good way to deal with binary data on the JS side yet 😞. We rely on []byte from the Go code, which translates to int arrays in JS, which is far from optimal. These are the places where we currently have binary data that I can think of:

• open("somefile", "b")
• HTTP response bodies when the request had { responseType: "binary" } as params
• in crypto.randomBytes() (Add crypto.randomBytes #922)
• in the crypto hashing and digest functions (Add binary outputEncoding to the crypto functions #952)
• in the new PKI crypto functions (PKI extension to crypto module [bounty: $650] #900)
• base64 encoding and decoding

We need a native analogue of the node.js Buffer, or better yet - to properly support ES ArrayBuffer objects. We actually have ArrayBuffer, the code below compiles and runs as you'd expect:

export default function () {
    var buffer = new ArrayBuffer(16);

    // Create a couple of views
    var view1 = new DataView(buffer);
    var view2 = new DataView(buffer, 12, 4); //from byte 12 for the next 4 bytes
    view1.setInt8(12, 42); // put 42 in slot 12

    console.log(view2.getInt8(0));

    var z = new Int32Array(buffer, 0, 4);
    console.log(z);
};

but I'm not sure if this ArrayBuffer comes from the bundled core.js or from goja. It seems like the ArrayBuffer is from goja (though I wouldn't rely on it very much), but the DataView and Int32Array seem to be from core.js 😕

So we need to investigate exactly what the situation is and what we can use to handle binary data sensibly...

Related issues: #856, #874, #873

[na--]

A user found out that the current websocket API lacks support for binary messages: https://community.k6.io/t/converting-audio-file-into-bytes-and-send-over-web-socket/272/

But in order to fix the websocket code, it would be best if we first standardized how we support binary data in general, i.e. this issue... So, because of that and all of the other dependent issues, I'm upping the priority of this issue.

[imiric]

I took a brief look at this, and the current ArrayBuffer implementation comes from core.js, as do DataView and all typed arrays. The Goja implementation is actually disabled, and doesn't seem to have ever been used. (You can now confirm this easily with --compatibility-mode base. ;)

As suggested in #420 and dop251/goja#51, the way forward seems to be having native support in Goja, which would mean resurrecting the currently unused ArrayBuffer implementation, and ensuring it works transparently with core.js polyfills.

It doesn't seem like a gargantuan amount of work, but I'm a bit out of my depths here, as Goja internals are quite complex. I can give this a shot if you agree with the approach, let me know.

[caseylucas]

I found this issue after needing to post some binary data. I ended up making a change to k6 that would allow me to use Uint8Array in the js code but it's a little hacky. Just wanted to share here in case anyone else needs it or it sparks any other ideas. Basically, I just look at the body parameter and if it quacks like an Uint8Array, I call the goja Get func to get all the bytes and put them in a byte[].

diff --git a/js/modules/k6/http/request.go b/js/modules/k6/http/request.go
index d448c3a0..07fec21f 100644
--- a/js/modules/k6/http/request.go
+++ b/js/modules/k6/http/request.go
@@ -29,6 +29,7 @@ import (
        "net/textproto"
        "net/url"
        "reflect"
+       "strconv"
        "strings"
        "sync"
        "time"
@@ -97,7 +98,24 @@ func (h *HTTP) Request(ctx context.Context, method string, url goja.Value, args
        var params goja.Value

        if len(args) > 0 {
-               body = args[0].Export()
+               // check to see if body param looks like a Uint8Array and if so, copy bytes out of it into a []byte.
+               if obj, ok := args[0].(*goja.Object); ok {
+                       bytesPerElementVal := obj.Get("BYTES_PER_ELEMENT")
+                       byteLengthVal := obj.Get("byteLength")
+                       if bytesPerElementVal != nil &&
+                               bytesPerElementVal.ToInteger() == 1 &&
+                               byteLengthVal != nil {
+                               byteLength := byteLengthVal.ToInteger()
+                               byteBuf := make([]byte, byteLength)
+                               for i := int64(0); i < byteLength; i++ {
+                                       byteBuf[i] = (byte)(obj.Get(strconv.FormatInt(i, 10)).ToInteger() & 0xff)
+                               }
+                               body = byteBuf
+                       }
+               }
+               if body == nil {
+                       body = args[0].Export()
+               }
        }
        if len(args) > 1 {
                params = args[1]

With this change, you can use like:

      const binaryBuf = new Uint8Array(4);
      binaryBuf[0] = 0;
      binaryBuf[1] = 1;
      binaryBuf[2] = 2;
      binaryBuf[3] = 3;
      http.post(
        someURL,
        binaryBuf,
      );

[caseylucas]

@imiric Did you have any luck with the goja ArrayBuffer implementation? Is this still the preferred strategy vs something like I did above? Need any help?

[imiric]

@caseylucas No, I didn't make much progress with exposing the native ArrayBuffer. Our top priority right now is getting #1007 merged, so while this issue is high on the priority list, it's unlikely to be worked on for a few weeks at least.

Your approach looks interesting, though I'm curious about the performance, and it should probably be done outside of HTTP.Request, as k6 would benefit from a more generic and lower-level implementation. Pull requests are always welcome, so if you think this approach could work for the other issues mentioned here (open(), crypto, etc.), feel free to contribute. :)

[sirianni]

Is there a workaround that can be used currently, even if it is inefficient? I am using protobufjs which encodes protobuf messages as a Uint8Array. When I try to pass this as a request body I get this error. It seems that the golang http code has support for byte[], but I can't figure out how to pass that from the javascript side.

ERRO[0000] GoError: unknown request body type []interface {}
	at native

[na--]

Prompted by this forum topic, it might not be a bad idea if we also support "generator functions" when we have binary data. So, instead of http.post(url, someBigChunkOfData), users should be able to use http.post(url, someJSFunctionThatGeneratesDataOnDemand)...