buffer: add swap16() and swap32() methods #5724

jasnell · 2016-03-15T18:15:37Z

Pull Request check-list

Does make -j8 test (UNIX) or vcbuild test nosign (Windows) pass with
this change (including linting)?
Is the commit message formatted according to [CONTRIBUTING.md][0]?
If this change fixes a bug (or a performance problem), is a regression
test (or a benchmark) included?
Is a documentation update included (if this change modifies
existing APIs, or introduces new ones)?

Affected core subsystem(s)

buffer

Description of change

Adds ~~Buffer.prototype.swapShort()~~ Buffer.prototype.swap16() and
~~Buffer.prototype.swapLong()~~ Buffer.prototype.swap32() methods that
mutate the Buffer instance in-place by swapping the 16-bit and 32-bit
byte-order.

Example:

const buf = Buffer([0x1, 0x2, 0x3, 0x4]);
buf.swap16();
console.log(buf);
  // prints Buffer(0x2, 0x1, 0x4, 0x3);

buf.swap32();
console.log(buf);
  // prints Buffer(0x3, 0x4, 0x1, 0x2);

Was looking at a number of use cases recently around reading in UTF16BE data and converting that to UTF8 and realized that we really didn't expose an efficient mechanism for doing a byte-swap in Buffer. This seemed like a useful API addition.

/cc @trevnorris @srl295

mscdex · 2016-03-15T18:22:57Z

I think swap16() and swap32() might be better names.

Just how common are these anyway?

jasnell · 2016-03-15T18:28:13Z

I can live with swap16 and swap32.

The swap16 is particularly useful when dealing with utf16 big-endian encoded buffer content that needs to be converted. The swap32 is less common but included for completeness. There are certain formats (such as TIFF) that can be big-endian or little-endian and it's common when working with those to have to swap the order to meet expectations. I wouldn't say that these are critical by any stretch, but they are useful and building the implementation directly into node_buffer is cheap.

jasnell · 2016-03-15T18:40:06Z

Methods renamed to swap16/swap32

trevnorris · 2016-03-15T18:53:10Z

src/node_buffer.cc

@@ -1094,6 +1130,8 @@ void SetupBufferJS(const FunctionCallbackInfo<Value>& args) {
  env->SetMethod(proto, "hexWrite", HexWrite);
  env->SetMethod(proto, "ucs2Write", Ucs2Write);
  env->SetMethod(proto, "utf8Write", Utf8Write);
+  env->SetMethod(proto, "swap16", SwapShort);
+  env->SetMethod(proto, "swap32", SwapLong);


While case differs, generally the names are the same between c++ and js. Would Swap{16,32} work?

jasnell · 2016-03-15T19:51:19Z

@trevnorris ... ok, updated! PTAL

jasnell · 2016-03-15T20:17:43Z

CI: https://ci.nodejs.org/job/node-test-pull-request/1935/

Fishrock123 · 2016-03-15T21:39:31Z

Does this belong in core?

trevnorris · 2016-03-15T21:54:30Z

src/node_buffer.cc

+
+  if (ts_obj_length % sizeof(uint16_t) != 0)
+    return env->ThrowRangeError(
+      "Buffer length must be a multiple of 16-bits");


nit: wrap multi-line statements in conditionals with {}, and double indent with multi-line statements. another thing lint should be catching.

trevnorris · 2016-03-15T22:02:46Z

@Fishrock123 Asked myself the same thing. I believe it may just fit within the parameters of what functionality belongs in core. For example:

// Read in utf16be file
const file = fs.readFileSync(path).swap16().toString('utf16le');
// Write it back to disk
fs.writeFileSync(path, Buffer(file, 'utf16le').swap16());

Though additional input is welcome.

@jasnell have a nit, but LGTM otherwise. Let's leave this open though for at least a couple days for others to respond.

jasnell · 2016-03-15T22:07:04Z

@Fishrock123 ... it is certainly possible to do in userland but there's a bit of a performance penalty if it's done in pure javascript... I'll post some benchmark numbers in a couple of minutes that compares the performance of swap16/swap32 to equivalent pure javascript code. I went back and forth on this one also but I landed in the same @trevnorris did... that it likely fits just inside that boundary.

jasnell · 2016-03-15T22:27:27Z

Benchmarks ... the swap(16|21) is the impl in node_buffer, while the htons/htonl is a pure javascript impl... note that at the low end of the range the javascript impl's blow away the native impls, but that reverses once the buffer size grows. Given that, I may change this up a bit to do the swap in javascript if the length is below a certain threshold.

4     - swap16/htons - 15881312.97675 / 56764416.88469 - htons
64    - swap16/htons - 11189303.45557 / 18467670.86474 - htons
1024  - swap16/htons - 2019124.50116  / 1546034.86159  - swap16
2056  - swap16/htons - 1070521.29584  / 732798.39598   - swap16
4096  - swap16/htons - 591436.23889   / 383412.43533   - swap16

4     - swap32/htonl - 15827403.80753 / 78215981.07361 - htonl
64    - swap32/htonl - 11970720.57513 / 20776537.63779 - htonl
1024  - swap32/htonl - 2255585.06207  / 1886075.54128  - swap32
2056  - swap32/htonl - 1242583.71829  / 1036100.06795  - swap32
4096  - swap32/htonl - 621999.48227   / 511625.91432   - swap32

williamkapke · 2016-03-15T23:08:26Z

benchmark/buffers/buffer-swap.js

+    buf[i - 1] = i;
+  bench.start();
+  for (i = 0; i < n; i += 1)
+    method.apply(buf);


Maybe avoid using apply inside the bench start/end?

In buffer-iterate.js it just calls off the instance...

const name = conf.method; //... for (i = 0; i < n; i += 1) method[name](buf);

jasnell · 2016-03-15T23:49:52Z

@trevnorris @williamkapke ... nits addressed.

I also updated the implementation such that if Buffer length is below a certain threshold, a pure javascript version of the swap will be used rather than dropping down to the native layer. Based on the benchmarks, when the Buffer is below a certain size, it's far more efficient to swap in js. We may be able to further tweak the threshold but the current limits appear to be within the margin of error.

williamkapke · 2016-03-16T02:00:17Z

lib/buffer.js

+  if (len < 512) {
+    if (len % 2 !== 0)
+      throw new RangeError('Buffer length must be a multiple of 16-bits');
+    for (var i = 0, n = 0; i < this.length; i += 2) {


you already have len above... you might as well use it here too! :D
(Same for swap32)

doh! missed that one. good catch :-)

williamkapke · 2016-03-16T02:16:42Z

If we're doing this, what about (also) including a more generic buffer.swap(a,b) that swap16 and swap32 could leverage? Maybe that would tank perf on the native side?

I think it makes the JS side cleaner...

Buffer.prototype.swap = function swap(a, b) {
  var x = this[a];
  this[a] = this[b];
  this[b] = x;
  return this;
};

Buffer.prototype.swap32 = function swap32() {
  for (var i = 0; i < buf.length; i += 2) {
    this.swap(i++, i + 1);
    this.swap(i++, i + 1);
  }
  return this;
};

Buffer.prototype.swap16 = function swap16() {
  for (var i = 0; i < buf.length; i++) {
    this.swap(i, ++i);
  }
  return this;
};

const buf = Buffer([0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7, 0x8]);
console.log(buf.swap(1,2));
//or
console.log(buf.swap16());
//or
console.log(buf.swap32());

jasnell · 2016-03-16T03:58:48Z

Having a swap function makes sense but I wouldn't expose it as part of the Buffer API.

Nits addressed!

williamkapke · 2016-03-16T05:51:22Z

heh- that's how I originally wrote it and then decided to put swap on the prototype ;)

LGTM!

trevnorris · 2016-03-16T20:49:55Z

benchmark/buffers/buffer-swap.js

+  bench.start();
+  for (i = 0; i < n; i += 1)
+    buf[method]();
+  bench.end(n);


mind throwing these (the for loop) in their own function? when profiling it's easier to look for execution of a named function that only contains the code actually being profiled.

jasnell · 2016-03-17T00:47:14Z

@trevnorris ... done! also squashed the commits and rebased to pick up the buffer api changes also

ronkorving · 2016-03-18T03:42:45Z

doc/api/buffer.markdown

+Interprets the `Buffer` as an array of unsigned 32-bit integers and swaps
+the byte-order *in-place*. Throws a `RangeError` if the `Buffer` length is
+not a multiple of 32 bits. The method returns a reference to the Buffer, so
+calls can be chained


Missing closing period.

ofrobots · 2016-03-22T15:52:30Z

What are the semantics when the size of the buffer is not a multiple of 2 / 4?

EDIT: found the answer by looking at the test-case.

jasnell · 2016-03-22T16:30:28Z

It's also covered in the documentation addition. A RangeError is thrown.

trevnorris · 2016-03-22T20:09:00Z

lib/buffer.js

@@ -17,6 +17,48 @@ var poolSize, poolOffset, allocPool;


 binding.setupBufferJS(Buffer.prototype, bindingObj);
+
+const swap16n = Buffer.prototype.swap16;
+const swap32n = Buffer.prototype.swap32;


Why are we doing this instead of attaching the native implementation to binding.swap*()?

Just an artifact of how it was originally written. You're right, attaching directly to binding.swap*() is better.

trevnorris · 2016-03-22T20:11:46Z

Nice tests. Left two comments. Other than that LGTM.

jasnell · 2016-03-22T20:20:19Z

@trevnorris .. thanks .. updated to address those nits!

trevnorris · 2016-03-23T06:06:19Z

@jasnell Thanks much. LGTM

ronkorving · 2016-03-23T06:33:17Z

lib/buffer.js

+  // dropping down to the native code is faster.
+  const len = this.length;
+  if (len % 2 !== 0)
+    throw new RangeError('Buffer length must be a multiple of 16-bits');


Incredible nitpicking, but nonetheless I would opt for one of these:

Buffer size must be a multiple of 16 bits

or:

Buffer length must be a multiple of 2

I hope you understand the nuance difference I'm getting at.

Will change to Buffer size must be a multiple of 16 bits when I land.

Adds Buffer.prototype.swap16() and Buffer.prototype.swap32() methods that mutate the Buffer instance in-place by swapping the 16-bit and 32-bit byte-order. Example: ```js const buf = Buffer([0x1, 0x2, 0x3, 0x4]); buf.swap16(); console.log(buf); // prints Buffer(0x2, 0x1, 0x4, 0x3); buf.swap32(); console.log(buf); // prints Buffer(0x3, 0x4, 0x1, 0x2); ``` PR-URL: #5724 Reviewed-By: Trevor Norris <trev.norris@gmail.com>

jasnell · 2016-03-23T15:54:23Z

Landed in 7d73e60. Thanks all!

Adds Buffer.prototype.swap16() and Buffer.prototype.swap32() methods that mutate the Buffer instance in-place by swapping the 16-bit and 32-bit byte-order. Example: ```js const buf = Buffer([0x1, 0x2, 0x3, 0x4]); buf.swap16(); console.log(buf); // prints Buffer(0x2, 0x1, 0x4, 0x3); buf.swap32(); console.log(buf); // prints Buffer(0x3, 0x4, 0x1, 0x2); ``` PR-URL: #5724 Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Notable changes: * buffer: * make byteLength work with ArrayBuffer & DataView (Jackson Tian) [#5255](#5255) * backport --zero-fill-buffers command line option (James M Snell) [#5744](#5744) * backport new buffer constructor APIs (James M Snell) [#5763](#5763) * add swap16() and swap32() methods (James M Snell) [#5724](#5724) * fs: add the fs.mkdtemp() function. (Florian MARGAINE) [#5333](#5333) * net: emit host in lookup event (HUANG Wei) [#5598](#5598) * node: --no-browser-globals configure flag (Fedor Indutny) [#5853](#5853) * npm: Upgrade to v3.8.3. (Forrest L Norvell) * repl: support standalone blocks (Prince J Wesley) [#5581](#5581) * src: override v8 thread defaults using cli options (Tom Gallacher) [#4344](#4344)

Notable changes: * buffer: * make byteLength work with ArrayBuffer & DataView (Jackson Tian) [#5255](#5255) * backport --zero-fill-buffers command line option (James M Snell) [#5744](#5744) * backport new buffer constructor APIs (James M Snell) [#5763](#5763) * add swap16() and swap32() methods (James M Snell) [#5724](#5724) * fs: add the fs.mkdtemp() function. (Florian MARGAINE) [#5333](#5333) * net: emit host in lookup event (HUANG Wei) [#5598](#5598) * node: --no-browser-globals configure flag (Fedor Indutny) [#5853](#5853) * npm: Upgrade to v3.8.3. Fixes a security flaw in the use of authentication tokens in HTTP requests that would allow an attacker to set up a server that could collect tokens from users of the command-line interface. Authentication tokens have previously been sent with every request made by the CLI for logged-in users, regardless of the destination of the request. This update fixes this by only including those tokens for requests made against the registry or registries used for the current install. (Forrest L Norvell) [npm#6](npm#6) * repl: support standalone blocks (Prince J Wesley) [#5581](#5581) * src: override v8 thread defaults using cli options (Tom Gallacher) [#4344](#4344)

Notable changes: * buffer: * make byteLength work with ArrayBuffer & DataView (Jackson Tian) [#5255](#5255) * backport --zero-fill-buffers command line option (James M Snell) [#5744](#5744) * backport new buffer constructor APIs (James M Snell) [#5763](#5763) * add swap16() and swap32() methods (James M Snell) [#5724](#5724) * fs: add the fs.mkdtemp() function. (Florian MARGAINE) [#5333](#5333) * net: emit host in lookup event (HUANG Wei) [#5598](#5598) * node: --no-browser-globals configure flag (Fedor Indutny) [#5853](#5853) * npm: Upgrade to v3.8.3. Fixes a security flaw in the use of authentication tokens in HTTP requests that would allow an attacker to set up a server that could collect tokens from users of the command-line interface. Authentication tokens have previously been sent with every request made by the CLI for logged-in users, regardless of the destination of the request. This update fixes this by only including those tokens for requests made against the registry or registries used for the current install. (Forrest L Norvell) [npm#6](npm#6) * repl: support standalone blocks (Prince J Wesley) [#5581](#5581) * src: override v8 thread defaults using cli options (Tom Gallacher) [#4344](#4344) PR-URL: #5970

jasnell added buffer Issues and PRs related to the buffer subsystem. semver-minor PRs that contain new features and should be released in the next minor version. labels Mar 15, 2016

jasnell changed the title ~~buffer: add swapShort() and swapLong() methods~~ buffer: add swap16() and swap32() methods Mar 15, 2016

trevnorris reviewed Mar 15, 2016
View reviewed changes

williamkapke reviewed Mar 15, 2016
View reviewed changes

williamkapke reviewed Mar 16, 2016
View reviewed changes

trevnorris reviewed Mar 16, 2016
View reviewed changes

jasnell force-pushed the buffer-byte-swap branch 2 times, most recently from 74769f0 to 26b4ffd Compare March 17, 2016 00:46

ronkorving reviewed Mar 18, 2016
View reviewed changes

jasnell modified the milestone: 6.0.0 Mar 22, 2016

Fishrock123 removed this from the 6.0.0 milestone Mar 22, 2016

trevnorris reviewed Mar 22, 2016
View reviewed changes

Address feedback from @trevnorris

9838523

ronkorving reviewed Mar 23, 2016
View reviewed changes

jasnell closed this Mar 23, 2016

evanlucas mentioned this pull request Mar 31, 2016

v5.10.0 Release Proposal #5970

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buffer: add swap16() and swap32() methods #5724

buffer: add swap16() and swap32() methods #5724

jasnell commented Mar 15, 2016

mscdex commented Mar 15, 2016

jasnell commented Mar 15, 2016

jasnell commented Mar 15, 2016

trevnorris Mar 15, 2016

jasnell commented Mar 15, 2016

jasnell commented Mar 15, 2016

Fishrock123 commented Mar 15, 2016

trevnorris Mar 15, 2016

trevnorris commented Mar 15, 2016

jasnell commented Mar 15, 2016

jasnell commented Mar 15, 2016

williamkapke Mar 15, 2016

jasnell commented Mar 15, 2016

williamkapke Mar 16, 2016

jasnell Mar 16, 2016

williamkapke commented Mar 16, 2016

jasnell commented Mar 16, 2016

williamkapke commented Mar 16, 2016

trevnorris Mar 16, 2016

jasnell Mar 16, 2016

jasnell commented Mar 17, 2016

ronkorving Mar 18, 2016

ofrobots commented Mar 22, 2016

jasnell commented Mar 22, 2016

trevnorris Mar 22, 2016

jasnell Mar 22, 2016

trevnorris commented Mar 22, 2016

jasnell commented Mar 22, 2016

trevnorris commented Mar 23, 2016

ronkorving Mar 23, 2016

jasnell Mar 23, 2016

jasnell commented Mar 23, 2016

buffer: add swap16() and swap32() methods #5724

buffer: add swap16() and swap32() methods #5724

Conversation

jasnell commented Mar 15, 2016

Pull Request check-list

Affected core subsystem(s)

Description of change

mscdex commented Mar 15, 2016

jasnell commented Mar 15, 2016

jasnell commented Mar 15, 2016

Choose a reason for hiding this comment

jasnell commented Mar 15, 2016

jasnell commented Mar 15, 2016

Fishrock123 commented Mar 15, 2016

Choose a reason for hiding this comment

trevnorris commented Mar 15, 2016

jasnell commented Mar 15, 2016

jasnell commented Mar 15, 2016

Choose a reason for hiding this comment

jasnell commented Mar 15, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

williamkapke commented Mar 16, 2016

jasnell commented Mar 16, 2016

williamkapke commented Mar 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented Mar 17, 2016

Choose a reason for hiding this comment

ofrobots commented Mar 22, 2016

jasnell commented Mar 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trevnorris commented Mar 22, 2016

jasnell commented Mar 22, 2016

trevnorris commented Mar 23, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented Mar 23, 2016