make enums containing ByteBuffer fit in three words #349

weissi · 2018-04-23T21:12:30Z

only really makes sense together with #351

Motivation:

NIO's dynamic pipeline comes with one performance caveat: If your type
isn't specialised in NIOAny (only types in the NIO module can be) and
it's over 3 words (24 bytes on 64-bit platforms), you'll suffer an
allocation for every time you box it in NIOAny. Very unfortunately, both
ByteBuffer and FileRegion were exactly 3 words wide which means that any
enum containing those would be wider than 3 words. Worse, both
HTTP{Request,Response}Head contained types that were wider than 3 words.
The best solution to this problem is to shrink ByteBuffer and FileRegion
to just under 3 words and that's exactly what this PR is doing. That is
slightly tricky as ByteBuffer was already bit packed fairly well
(reader/writer indices & slice begin/end were stored as a UInt32). The
trick we employ now is to store the slice beginning in a UInt24 and the
file region reader index in a UInt56. That saves one byte for both
ByteBuffer and FileRegion with very moderate tradeoffs: The reader index
in a file now needs to be within 64 PiB (peta bytes) and a byte buffer
slice beginning must start within 16 MiB (mega bytes). Note: The
reader/writer indices as well as slice ends are not affected and can
still be within 4 GiB. Clearly no one would care about the restrictions
for FileRegions in the real world but we might hit the ByteBuffer slice
beginning limit in which case the slice would be copied out. But
given that slices are mostly used to slice off headers in network
protocols, 16 MiB should be plenty.

Norman was kind enough to measure the perf differences:

master (before this PR):

$ wrk -c 256 -d 10s -t 4 -s ~/Downloads/pipeline-many.lua -H "X-Host: SomeValue" -H "Host: swiftnio.io" -H "ThereAreEvenMoreHeaders: AndMoreValues" http://localhost:8888
Running 10s test @ http://localhost:8888
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    93.02ms  158.13ms   1.97s    93.07%
    Req/Sec   112.79k    26.05k  258.75k    84.50%
  4501098 requests in 10.04s, 214.63MB read
  Socket errors: connect 0, read 111, write 0, timeout 89
Requests/sec: 448485.61
Transfer/sec:     21.39MB

after this PR:

$ wrk -c 256 -d 10s -t 4 -s ~/Downloads/pipeline-many.lua -H "X-Host: SomeValue" -H "Host: swiftnio.io" -H "ThereAreEvenMoreHeaders: AndMoreValues" http://localhost:8888
Running 10s test @ http://localhost:8888
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   107.53ms  206.56ms   1.99s    90.55%
    Req/Sec   124.15k    26.56k  290.41k    89.25%
  4952904 requests in 10.03s, 236.17MB read
  Socket errors: connect 0, read 161, write 0, timeout 22
Requests/sec: 493852.65
Transfer/sec:     23.55MB

so we see a nice 10% improvement and allocations go down by 2, too:

23:21:44 info: 1000_reqs_1_conn: allocations not freed: 0
23:21:44 info: 1000_reqs_1_conn: total number of mallocs: 67832

from

19:08:57 info: 1000_reqs_1_conn: allocations not freed: 0
19:08:57 info: 1000_reqs_1_conn: total number of mallocs: 69834

Modifications:

shrank ByteBuffer by 1 byte, making it 23 bytes in total
shrank FileRegion by 1 byte, making it 23 bytes in total
CoW boxed parts of HTTP{Request,Response}Head to make them fit within 2 words
added @_inlineable to NIOAny where appropriate to not suffer boxed existentials in different places
added tests for the new edge cases

Result:

more speed
fewer allocations

normanmaurer · 2018-04-24T07:43:52Z

Sources/NIO/ByteBuffer-core.swift

@@ -29,6 +29,86 @@ let sysFree: @convention(c) (UnsafeMutableRawPointer?) -> Void = free
 }
 #endif

+struct SadUInt24: ExpressibleByIntegerLiteral {


Haha... Sad... ;)

normanmaurer · 2018-04-24T07:44:26Z

Sources/NIO/ByteBuffer-core.swift

+ }
+
+ init?(_ value: UInt32) {
+ guard value & 0xff_00_00_00 == 0 else {


TIL that you can use _ to make it more readable

normanmaurer

Looks great so far... Just some random comments

normanmaurer · 2018-04-24T07:45:18Z

Sources/NIOHTTP1/HTTPTypes.swift

@@ -19,11 +19,52 @@ let headerSeparator: StaticString = ": "

 /// A representation of the request line and header fields of a HTTP request.
 public struct HTTPRequestHead: Equatable {
+ private class _Storage {


I guess it not really makes a difference but use final (for all internal classes) ?

normanmaurer · 2018-04-24T07:45:38Z

Sources/NIOHTTP1/HTTPTypes.swift

+ }
+
+ func copy() -> _Storage {
+ return _Storage(method: self.method, rawURI: self.rawURI, version: self.version)


Can you either always use .init or always _Storage just to make it consistent ?

normanmaurer · 2018-04-24T14:40:12Z

Sources/NIO/ByteBuffer-core.swift

+
+ guard sliceBeginIndex <= ByteBuffer.Slice.maxSupportedLowerBound else {
+ // the slice's begin is past the maximum supported slice begin value (16 MiB) so the only option we have
+ // is copy the slice into a fresh buffer and the slice begin will then be at index 0.


nit: s/and the/. The/

Lukasa

Can we pull the changes to the HTTP types to a separate PR? It'll make reviewing the changes quite a bit easier. 😄

Lukasa · 2018-04-24T14:45:25Z

Sources/NIO/ByteBuffer-core.swift

+ init(_ value: UInt32) {
+ precondition(value & 0xff_00_00_00 == 0, "value \(value) too large for _UInt24")
+ self.b12 = UInt16(truncatingIfNeeded: (value & 0xff_ff) >> 0 )
+ self.b3 = UInt8(truncatingIfNeeded: (value & 0xff_ff_00) >> 16)


You can save yourself a masking operation here. You already precondition that the top byte is zero, so with that knowledge in place you can just shift right by 16 bits and store into UInt8.

Also as noted in the chat, the mask isn't really right anyway.

oops! thanks

Lukasa · 2018-04-24T14:46:03Z

Sources/NIO/ByteBuffer-core.swift

@@ -29,6 +29,97 @@ let sysFree: @convention(c) (UnsafeMutableRawPointer?) -> Void = free
 }
 #endif

+@_versioned
+struct _UInt24: ExpressibleByIntegerLiteral {


Can we get a doc comment here please?

Lukasa · 2018-04-24T14:46:13Z

Sources/NIO/ByteBuffer-core.swift

+}
+
+@_versioned
+struct _ByteBufferSlice {


Similarly, can we get a doc comment here please?

Lukasa · 2018-04-24T14:48:29Z

Sources/NIO/ByteBuffer-core.swift

@@ -486,8 +577,19 @@ public struct ByteBuffer {
 }
 let index = _toIndex(index)
 let length = _toCapacity(length)
+ let sliceBeginIndex = self._slice.lowerBound + index
+
+ guard sliceBeginIndex <= ByteBuffer.Slice.maxSupportedLowerBound else {


Nit: the word "begin" here is odd. Start might be better.

Lukasa · 2018-04-24T14:49:42Z

Sources/NIO/FileRegion.swift

@@ -12,6 +12,68 @@
 //
 //===----------------------------------------------------------------------===//

+struct _UInt56: ExpressibleByIntegerLiteral {


Might be nice to consolidate our helper integers into a single source file.

fair enough, will do that

Lukasa · 2018-04-24T14:50:04Z

Sources/NIO/FileRegion.swift

+ precondition(value & 0xff_00_00_00_00_00_00_00 == 0)
+ self.init(b1234: UInt32(truncatingIfNeeded: (value & 0xff_ff_ff_ff) >> 0 ),
+ b56: UInt16(truncatingIfNeeded: (value & 0xff_ff_00_00_00_00) >> 32),
+ b7: UInt8(truncatingIfNeeded: (value & 0xff_00_00_00_00_00_00) >> 48))


Given your earlier precondition, this mask is unnecessary.

normanmaurer · 2018-04-24T14:41:35Z

Sources/NIO/FileRegion.swift

+
+extension _UInt56 {
+ init(_ value: UInt64) {
+ precondition(value & 0xff_00_00_00_00_00_00_00 == 0)


should all of these precondition be assert as this is not really something the user can trigger but only ourselves if our code has a bug.

normanmaurer · 2018-04-24T16:51:57Z

Sources/NIO/IntegerTypes.swift

+
+extension _UInt24 {
+ init(_ value: UInt32) {
+ precondition(value & 0xff_00_00_00 == 0, "value \(value) too large for _UInt24")


@weissi should this be an assert as there is no way for the user to actual construct this ? I think we usually use assert for things that would only happen because of a bug in NIO itself

normanmaurer · 2018-04-24T16:52:17Z

Sources/NIO/IntegerTypes.swift

+
+extension _UInt56 {
+ init(_ value: UInt64) {
+ precondition(value & 0xff_00_00_00_00_00_00_00 == 0)


@weissi should this be an assert as there is no way for the user to actual construct this ? I think we usually use assert for things that would only happen because of a bug in NIO itself

Also consider adding a message as well

Lukasa

One minor nit: feel free to either address it or not depending on how you feel about it.

Lukasa · 2018-04-24T17:07:44Z

Tests/NIOTests/ByteBufferTest.swift

+
+ let offset = Int(_UInt24.max) + 1
+ let expectedReadablebytes = totalBufferSize - offset
+ let slice = buf.getSlice(at: offset, length: expectedReadablebytes)!


Minor nit: it might be better if we sliced the same size as the test above, to isolate the test differences to only being the slice start index, rather than the slice sizes.

addressed and improved test (by marking some 'important' positions with different byte values)

normanmaurer

LGTM after @Lukasa's nit is addressed... Great work!

Motivation: NIO's dynamic pipeline comes with one performance caveat: If your type isn't specialised in NIOAny (only types in the NIO module can be) and it's over 3 words (24 bytes on 64-bit platforms), you'll suffer an allocation for every time you box it in NIOAny. Very unfortunately, both ByteBuffer and FileRegion were exactly 3 words wide which means that any enum containing those would be wider than 3 words. Worse, both HTTP{Request,Response}Head contained types that were wider than 3 words. The best solution to this problem is to shrink ByteBuffer and FileRegion to just under 3 words and that's exactly what this PR is doing. That is slightly tricky as ByteBuffer was already bit packed fairly well (reader/writer indices & slice begin/end were stored as a UInt32). The trick we employ now is to store the slice beginning in a UInt24 and the file region reader index in a UInt56. That saves one byte for both ByteBuffer and FileRegion with very moderate tradeoffs: The reader index in a file now needs to be within 64 PiB (peta bytes) and a byte buffer slice beginning must start within 16 MiB (mega bytes). Note: The reader/writer indices as well as slice ends are _not_ affected and can still be within 4 GiB. Clearly no one would care about the restrictions for FileRegions in the real world but we might hit the ByteBuffer slice beginning limit in which case the slice would be copied out. But given that slices are mostly used to slice off headers in network protocols, 16 MiB should be _plenty_. Norman was kind enough to measure the perf differences: master (before this PR): ``` $ wrk -c 256 -d 10s -t 4 -s ~/Downloads/pipeline-many.lua -H "X-Host: SomeValue" -H "Host: swiftnio.io" -H "ThereAreEvenMoreHeaders: AndMoreValues" http://localhost:8888 Running 10s test @ http://localhost:8888 4 threads and 256 connections Thread Stats Avg Stdev Max +/- Stdev Latency 93.02ms 158.13ms 1.97s 93.07% Req/Sec 112.79k 26.05k 258.75k 84.50% 4501098 requests in 10.04s, 214.63MB read Socket errors: connect 0, read 111, write 0, timeout 89 Requests/sec: 448485.61 Transfer/sec: 21.39MB ``` after this PR: ``` $ wrk -c 256 -d 10s -t 4 -s ~/Downloads/pipeline-many.lua -H "X-Host: SomeValue" -H "Host: swiftnio.io" -H "ThereAreEvenMoreHeaders: AndMoreValues" http://localhost:8888 Running 10s test @ http://localhost:8888 4 threads and 256 connections Thread Stats Avg Stdev Max +/- Stdev Latency 107.53ms 206.56ms 1.99s 90.55% Req/Sec 124.15k 26.56k 290.41k 89.25% 4952904 requests in 10.03s, 236.17MB read Socket errors: connect 0, read 161, write 0, timeout 22 Requests/sec: 493852.65 Transfer/sec: 23.55MB ``` so we see a nice 10% improvement Modifications: - shrank ByteBuffer by 1 byte, making it 23 bytes in total - shrank FileRegion by 1 byte, making it 23 bytes in total - added @_inlineable to NIOAny where appropriate to not suffer boxed existentials in different places - added tests for the new edge cases Result: - more speed - fewer allocations

normanmaurer · 2018-04-24T17:31:32Z

@weissi please merge once CI completes

weissi force-pushed the jw-bananas-2 branch from f3f7f04 to c578feb Compare April 23, 2018 21:16

normanmaurer reviewed Apr 24, 2018

View reviewed changes

normanmaurer requested changes Apr 24, 2018

View reviewed changes

weissi force-pushed the jw-bananas-2 branch 3 times, most recently from b35aa27 to 25fae62 Compare April 24, 2018 14:24

weissi changed the title ~~[EXPERIMENT] [DO-NOT-MERGE] shrink ByteBuffer and reduce allocations~~ make enums containing ByteBuffer fit in three words Apr 24, 2018

weissi force-pushed the jw-bananas-2 branch from 25fae62 to 62a8c8b Compare April 24, 2018 14:26

weissi requested review from normanmaurer and Lukasa April 24, 2018 14:26

weissi force-pushed the jw-bananas-2 branch from 62a8c8b to 7f15f16 Compare April 24, 2018 14:33

normanmaurer reviewed Apr 24, 2018

View reviewed changes

weissi force-pushed the jw-bananas-2 branch from 7f15f16 to b3625e4 Compare April 24, 2018 14:42

Lukasa requested changes Apr 24, 2018

View reviewed changes

weissi force-pushed the jw-bananas-2 branch 2 times, most recently from d7f91d4 to 1f36f2d Compare April 24, 2018 16:08

normanmaurer requested changes Apr 24, 2018

View reviewed changes

weissi force-pushed the jw-bananas-2 branch 2 times, most recently from a445186 to 6c91e27 Compare April 24, 2018 17:03

weissi mentioned this pull request Apr 24, 2018

shrink HTTP{Request,Response}Head by CoW boxing them #351

Merged

Lukasa approved these changes Apr 24, 2018

View reviewed changes

normanmaurer approved these changes Apr 24, 2018

View reviewed changes

weissi force-pushed the jw-bananas-2 branch from 6c91e27 to c6303d1 Compare April 24, 2018 17:28

weissi merged commit db61cfe into apple:master Apr 24, 2018

weissi deleted the jw-bananas-2 branch April 24, 2018 17:51

weissi added the semver/patch No public API change. label Apr 24, 2018

weissi added this to the 1.6.0 milestone Apr 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make enums containing ByteBuffer fit in three words #349

make enums containing ByteBuffer fit in three words #349

weissi commented Apr 23, 2018 •

edited

Loading

normanmaurer Apr 24, 2018

normanmaurer Apr 24, 2018

normanmaurer left a comment

normanmaurer Apr 24, 2018

normanmaurer Apr 24, 2018

normanmaurer Apr 24, 2018

Lukasa left a comment

Lukasa Apr 24, 2018

Lukasa Apr 24, 2018

weissi Apr 24, 2018

Lukasa Apr 24, 2018

Lukasa Apr 24, 2018

Lukasa Apr 24, 2018

Lukasa Apr 24, 2018

weissi Apr 24, 2018

Lukasa Apr 24, 2018

normanmaurer Apr 24, 2018

normanmaurer Apr 24, 2018

normanmaurer Apr 24, 2018

Lukasa left a comment

Lukasa Apr 24, 2018

weissi Apr 24, 2018

normanmaurer left a comment

normanmaurer commented Apr 24, 2018

make enums containing ByteBuffer fit in three words #349

make enums containing ByteBuffer fit in three words #349

Conversation

weissi commented Apr 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

normanmaurer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lukasa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lukasa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

normanmaurer left a comment

Choose a reason for hiding this comment

normanmaurer commented Apr 24, 2018

weissi commented Apr 23, 2018 •

edited

Loading