Efficient communication of data between isolates

[leafpetersen]

A common pattern in flutter apps is to offload computationally intensive work (e.g. json decoding) to a separate isolate to avoid blocking the main UI thread. This breaks down when the isolate needs to return large amounts of data to the UI thread, since just the time to transfer the data back to the main isolate can block the main isolate for too long.

[leafpetersen]

cc @mraleph @jacob314

[jmesserly]

This would also be a huge help to the IDE too. Often they're written to take advantage of parallelism, so they can respond quickly to some requests (e.g. autocomplete) while taking more time to respond to others.

(C#'s IDE for example is designed to take advantage of .NET's Task Parallel Library)

[mraleph]

/cc @aam who is investigating transferrables.

[ramsestom]

I think that the dart-lang team should focus on providing a real threads support, with shared memory, for Dart rather than trying to improve the communication between Isolates which would just be a sticking-plaster solution (with many limitations like the immutability of shared objects between Isolates). Isolates should simply be used for isolated tasks, as their name suggests, requiring little or no data transfer between processes. For the rest Dart should offer the possibility to use real threads, like any modern programming language worthy of the name

[ping996]

I send message to the other isolate from the main isolate takes almost 8ms, it takes too long time of port communication.
2019-12-12 14:38:03.707 7699-7718/com.example.flutter_app I/flutter: >>>>sendMessage action: EXECUTE_JS 2019-12-12 14:38:03.717 7699-7730/com.example.flutter_app I/flutter: receive message from main.......

[modulovalue]

Is anybody working on this issue?

--

I've provided a rudimentary benchmark to measure skipped frames due to inefficient communication between isolates in dart-lang/sdk#40653

This is the result of a single run (json encoding and decoding):

13985982 bytes, please wait ~10 seconds.
 • Isolate Decode
   - Took          : 196
   - SkippedFrames : 152     (Frame 241 to 393)

 • Isolate Encode
   - Took          : 187
   - SkippedFrames : 6     (Frame 482 to 488)

 • Decode
   - Took          : 47
   - SkippedFrames : 47     (Frame 488 to 535)

 • Encode
   - Took          : 29
   - SkippedFrames : 30     (Frame 535 to 565)

This shows that it's impossible to have a smooth UI when a big json string has to be decoded and consumed entirely by the main isolate.

[mraleph]

We are actively working on implementing improvements to the concurrency story in the VM. The design space is vast, so we are focused mainly on implementing underlying infrastructure. One of the first approaches we are going to try, once the infrastructure is fully in place, is "send-and-exit" approach (tracked by dart-lang/sdk#37835).

[mkustermann]

The core of this issue seems to be

This breaks down when the isolate needs to return large amounts of data to the UI thread, since just the time to
transfer the data back to the main isolate can block the main isolate for too long.

This is now addressed with a newly exposed Isolate.exit() API (was recently added as part of dart-lang/sdk#47164) that can be used to hand ownership of large amounts of data to another isolate without making the receiver having a O(n) cost. There's a few follow-up items in dart-lang/sdk#37835 though the main functionality is implemented now.

Even when not using Isolate.exit() but rather send data using SendPort.send() one will transitively copy the mutable part of the message, though this happens on the sender side. So even in this case, receiving - in normal circumstances - is only a fixed O(1) cost.

What does remain is that isolates coordinate on garbage collections - so certain pauses due to GC are shared across isolates. Though even single-isolate scenarios pay this pause cost.

Closing this issue since most planned work has been implemented as part of dart-lang/sdk#36097

[MadOPcode]

Can't help but ask. What prevents from adding support for (almost) 0-cost transfer between isolates without copying of typed data and Lists, Maps, Sets that contain only primitive types and are not referenced by anything?
It could look something like this:

// Keyword "transient" prevents the object from being referenced
// from inside any other object or from being assigned to any variable
// and forces the object's type parameters for List, Map, etc to be primitives.
transient Uint8List? bytes = Uint8List(1 << 10);
sendPort.send(bytes);
// at this point bytes == null
bytes[0] = 1; // compile-time error

transient List<int>? listA = [0, 1, 2];
// for transient List, Map, etc. only primitive type parameters
// should be allowed as keys/values/elements
transient List<Object>? listB = []; // compile-time error
var listC = listA; // compile-time error
var map = { 'listA': listA }; // compile-time error

sendPort.send(listA);
// listA == null

void modifyList(transient List<int> list) {
  map['listA'] = listA; // compile-time error
  list.add(10); // ok
}

listA = [6, 7, 8, 9];
modifyList(listA); // see above
sendPort.send(listA);
listA.add(10); // compile-time error

[mkustermann]

What prevents from adding support for (almost) 0-cost transfer between isolates without copying of typed data and Lists, Maps, Sets that contain only primitive types and are not referenced by anything?

For 0-cost / sharing of objects, one has to choose between a) mutable b) immutable (and maybe c) transfer).

So far we always avoided the mutable option because it would allow one isolate to see side-effects (mutations of objects of any kinds) from other isolates. This is unexpected for the programmer and usually one needs to provide synchronization/locking primitives to allow a programmer to work safely in such an environment.

So we have chosen the immutable option for now. So transitively immutable objects (e.g. any constant, non-constant strings) are shared. For allowing other data structures (e.g. typed lists as you suggest) to be immutable one has to do more checking at runtime or have type-system guarantees (as you suggest with transient keyword).

The runtime option we have so far avoided due to cost in performance and code-size. The language option we have not explored yet: It is a long process to make language changes and one would want to make it generally applicable (ideally also for composites).

[MadOPcode]

@mkustermann what about doing it in a Javascript-like way at least for typed lists?

var dwords = Uint32List(0x100);
var bytes = Uint8List.view(dwords.buffer, 0x10);
print(bytes.buffer.lengthInBytes); // 1024
// SendPort.transfer sets the length of the source ByteBuffer to 0,
// takes the underlying memory buffer referenced by this ByteBuffer
// and uses it to create ByteBuffer and corresponding typed list
// to be used in the other isolate.
sendPort.transfer(dwords);
print(bytes.buffer.lengthInBytes); // 0
print(bytes.offsetInBytes); // 16
print(bytes[0]); // runtime error

The only drawback of this approach that I currently see is that this allows for the existence of typed lists which have offsets greater than the underlying buffer's length. But perhaps the existing range checks will deal with this anyway?

[mkustermann]

For typed list specifically

One can have multiple views on top of one backing store. So transferring one view would transfer the backing store, rendering all other views unusable. This is suboptimal - as it can lead to very obscure bugs in programs.

The mechanism of transferring will effectively make the typed list have length 0 and error on read/write access. The fact that the length can change means our optimizing compiler can no longer do certain optimizations, making byte access slower. So we would pay for better inter-isolate typed data support with normal byte access speed. This is a hard sell - but maybe we'll push for it in the future.

@MadOPcode In the meantime I can offer you an (unsafe) escape hatch: You can use dart:ffi to allocate memory on the C heap. The resulting buffer (of type Pointer<Uint8>) can be viewed as a typed list (via Pointer<Uint8>.asTypedList(<length>) - see api docs). You can send the address of that pointer to another isolate and use asTypedList(), then you'll have access to the same bytes from multiple isolates. Though you won't have strong memory ordering garantees if you have read-write or write-write conflicts. You'll also need to free the memory manually (ensuring that there's no more references to it.

Please consider opening a new issue if you have other requests, as this issue is specific to blocking UI thread when receiving data from helper isolates, which we have fixed.

[rchoi1712]

The mechanism of transferring will effectively make the typed list have length 0 and error on read/write access. The fact that the length can change means our optimizing compiler can no longer do certain optimizations, making byte access slower. So we would pay for better inter-isolate typed data support with normal byte access speed. This is a hard sell - but maybe we'll push for it in the future.

@mkustermann What if instead of changing the length, the ByteBuffer backing would just go away (be set to null), or point to an empty Buffer, once the list is transferred? Otherwise, perhaps allowing the creation of moveable ByteBuffers & views that support transferability - at the cost of a null-pointer check per access - will work here?

The runtime option we have so far avoided due to cost in performance and code-size. The language option we have not explored yet: It is a long process to make language changes and one would want to make it generally applicable (ideally also for composites).

I have a proposal for a language-level solution that relies on being able to perform a recursive move/transfer of the underlying data in collections - #3298 - and am looking to get some expert/team feedback on it

Please consider opening a new issue if you have other requests, as this issue is specific to blocking UI thread when receiving data from helper isolates, which we have fixed.

While Isolate.exit solves the issue of child->parent communication on exit, proposal #3298 would handle the case of two-way communication between long-lived isolates - i.e., dart-lang/sdk#52577, but not limited to TypedData.