os dialogs fail when called via dart:ffi on macOS (thread pinning Dart standalone)

[mit-mit]

Repro steps:

• git clone https://github.com/AndrewBelt/osdialog
• Add this to the Makefile:

> dylib: osdialog.o osdialog_mac.o
> 	$(CC) $(CFLAGS) -dynamiclib -undefined suppress -flat_namespace $(LDFLAGS) osdialog.o osdialog_mac.o -o osdialog.dylib

• Run make ARCH=mac dylib
• Run the following .dart program:

import 'dart:ffi' as ffi;

// char *osdialog_file(osdialog_file_action action, const char *path,
// const char *filename, osdialog_filters *filters);
typedef OSDialogFileC = ffi.Pointer Function(
    ffi.Int32, ffi.Pointer, ffi.Pointer, ffi.Pointer);
typedef OSDialogFileDart = ffi.Pointer Function(
    int, ffi.Pointer, ffi.Pointer, ffi.Pointer);

main() {
  final dylib = ffi.DynamicLibrary.open('osdialog.dylib');
  final OSDialogFileDart osdialog = dylib
      .lookup<ffi.NativeFunction<OSDialogFileC>>('osdialog_file')
      .asFunction();

  osdialog(0, ffi.nullptr, ffi.nullptr, ffi.nullptr);
}

Result:

2019-09-11 10:33:32.334 dart[49968:894229] WARNING: NSWindow drag regions should only be invalidated on the Main Thread! This will throw an exception in the future. Called from (
	0   AppKit                              0x00007fff29deb607 -[NSWindow(NSWindow_Theme) _postWindowNeedsToResetDragMarginsUnlessPostingDisabled] + 378
	1   AppKit                              0x00007fff29de89f7 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 1479
	2   AppKit                              0x00007fff29ea7d95 -[NSPanel _initContent:styleMask:backing:defer:contentView:] + 50
	3   AppKit                              0x00007fff29de842a -[NSWindow initWithContentRect:styleMask:backing:defer:] + 45
	4   AppKit                              0x00007fff29ea7d4a -[NSPanel initWithContentRect:styleMask:backing:defer:] + 64
	5   AppKit                              0x00007fff2a4bf4f5 -[NSSavePanel initWithContentRect:styleMask:backing:defer:] + 592
	6   AppKit                              0x00007fff2a0b9dec +[NSSavePanel _crunchyRawUnbonedPanel] + 550
	7   osdialog.dylib                      0x00000001084567bc osdialog_file + 124
	8   ???                                 0x00000001087c41ee 0x0 + 4437328366
	9   ???                                 0x00000001094acd03 0x0 + 4450864387
)

...

[sjindel-google]

The problem seems to be that the mutator isn't running on the system's "main" thread.

Since you're running from the standalone runtime, it's seems natural that the mutator for the main isolate should run on the main thread.

However, this sort of problem is more challenging to solve for embedders like Flutter, because the embedder can run any isolate on whatever thread it wants. I believe plugin code gets executed on the main thread of Flutter apps, and scheduling Dart code (or native code called by Dart) to run on the same thread would challenging.

/cc @chinmaygarde

[chinmaygarde]

Right. In Flutter, all Dart code runs on either an engine or VM managed thread. This thread is never the platforms main thread. The platform channels mechanism works around this limitation by forwarding all platform messages to the main thread and then sending the responses from native code back to the originating thread running Dart code.

In this specific case, the limitation is the that the the AppKit method is only safe to be called from the main thread. I suspect this is the case for all UI toolkits (whether they warn on unsafe use or not) used by the linked project. Instead of calling osdialog_file directly, the project should call a method that dispatches the call to the main thread (using something like libdispatch).

[davewhit3]

@danrubel @chinmaygarde @mit-mit any updates on this issue?

[chinmaygarde]

As I mentioned towards the end of my last comment, it is up to the native code to schedule the task on the appropriate thread. This is working as intended. In the case of osdialog_file, libdispatch may be used to schedule the task on the main thread.

[davewhit3]

A good idea!
Everything is clear now. Thank you very much!

[liamappelbe]

We're working on solving this in flutter (flutter/flutter#136314), but for dart command line programs this isn't something we can solve in the VM. As far as macOS is concerned, the "main thread" doesn't exist in pure dart apps, because there is no thread running the OS event loop (there's no DispatchQueue). So even if we did provide something like PlatformIsolates or thread pinning in the Dart VM, it wouldn't help with this use case.

A 3rd party package could solve this problem by writing some native code that initialises all the required OS event loop infrastructure on some thread (which would then become the "main thread" by definition), and they could interact with that native code through FFI. Such a package could also use the Dart C API to spawn an isolate on that thread, and provide a way for users to run code on that isolate, if necessary. This is outside of the scope of what we can do from within the VM though.

[mraleph]

This is outside of the scope of what we can do from within the VM though.

FWIW we fully control the embedder used by the dart binary so I don't see any reason why we can't actually start event loop when it is requested.

[matanlurey]

I ran into this issue while working on a (just for fun, not work related) dummy/prototype of using SDL or SDL-like libraries from pure (non-Flutter) Dart. Instead of explaining, I created a (very small) sample project (made easy thanks to the great work by @dcharkes and @knopp): https://github.com/matanlurey/dart_ffi_createwindow.

As it stands (and I'm not an expert, so I will gladly be wrong) it seems to me it might be mechanically impossible for, within the standalone Dart embedder, reliably use the main thread where required for FFI reasons (i.e. windowing). In my above example, I have (IMO) very simple code that tries to compute the result of 1 + 2 on the (explicit) main thread:

(I used Rust, if someone wanted to do this in C or Obj-C I expect it to work the same)

// Bindings around GrandCentralDispatch, i.e. libdispatch.
// See https://crates.io/crates/dispatch.
// See https://developer.apple.com/library/mac/documentation/Performance/Reference/GCD_libdispatch_Ref/index.html.
use dispatch::Queue;

#[no_mangle]
pub extern "C" fn ffi_sum_on_current_thread(a: i32, b: i32) -> i32 {
    a + b
}

#[no_mangle]
pub extern "C" fn ffi_sum_on_main_thread(a: i32, b: i32) -> i32 {
    Queue::main().exec_sync(|| a + b)
}

And the Dart side:

import 'dart:ffi';

@Native<Int32 Function(Int32, Int32)>()
external int ffi_sum_on_current_thread(int a, int b);

@Native<Int32 Function(Int32, Int32)>()
external int ffi_sum_on_main_thread(int a, int b);

---

My understanding is the only real way I could work around this is by creating a custom embedder, i.e. a small C, Rust, or Go program that starts (on the main thread), does the things I want (on the main thread), and then starts the Dart VM and executes code. I'm really wary of that approach, because at some point it just feels like I'm writing "Flutter 2, but bad", so I'm hoping this example nerd-snipes someone into helping me :)

[knopp]

This is probably relevant

flutter/flutter#150525

In theory it is possible to get platform thread and UI thread merged, @jonahwilliams did the work for iOS, I have a prototype working on macOS (it needs some work on pumping the microtask queue). I haven't had time recently to work on it, hopefully I'll get back to it soon. I'm fairly confident I can get this working on other desktop embedders too.

[matanlurey]

To be clear I'm thinking about the standalone Dart embedder and not use within Flutter.

[knopp]

My bad, somehow missed the standalone dart part. This would needs some threading consideration and system run loop integration and would need to be done per platform. You're right, you would need to build a minimal embedder for this, unless tighter integration with platform event loops is in scope for dart, which could get hairy (i.e. glib dependency in linux)...

[knopp]

Main thread is only one part of this. Running the platform event loop is another. The async code is quite convoluted, could there maybe be a way in future to override task scheduling in pure dart, so for example on macOS the Dart could override task default scheduling to post the task to current run loop, and then just call CFRunLoopRun()?

Similarly on Windows the dart could could create HWND based msg loop, override task scheduling to post the tasks to that HWND and then run the standard message loop?

[matanlurey]

I know it's overly simplistic but I just want a --start-on-main-thread-like-python flag 😄

[knopp]

I think main thread would be easy. Main run loop out of the box on the other hand would need to tie dart on linux to a particular implementation (i.e. glib). Not sure if that dependency is acceptable?

[liamappelbe]

@matanlurey What is the definition of "main thread" in this case? Typically the main thread is defined by the OS as whichever thread is running the OS/window event loop.

For example, if I'm writing a C++ app from scratch, I could create a window or event loop directly on that first thread, or I could spawn a different thread and create the window there (in which case the second thread would be the "main thread"). My understanding is that a program that doesn't interact with the OS's window system, like a pure Dart program in the standalone embedder, doesn't really have a main thread in that sense.

Presumably you could designate any arbitrary thread as the main thread using FFI by starting an event loop there (just need to be mindful of the fact that the embedder sometimes changes the thread an Isolate is running on). Someone could write a Dart package that does this.

I'm curious what Queue::main() does in your example. I would have thought you'd need to call an API function like Queue::startMainEventLoopOnThisThread() that would effectively define which thread is the main thread. Maybe it hangs because it sends a message to a non-existent main thread and never gets a response?

[matanlurey]

@liamappelbe:

Hey thanks for the response Liam. I also saw your comments on #52106 which seem to overlap here (basically, thanks for being helpful in general).

I'm curious what Queue::main() does in your example. I would have thought you'd need to call an API function like Queue::startMainEventLoopOnThisThread() that would effectively define which thread is the main thread. Maybe it hangs because it sends a message to a non-existent main thread and never gets a response?

Yeah this might just be my misunderstanding and it's totally possible for this to work.

Queue::main() is a bit of Rust wrapper code around libdispatch:

pub fn dispatch_get_main_queue() -> dispatch_queue_t {
    unsafe { &_dispatch_main_q as *const _ as dispatch_queue_t }
}

It's plausible I misunderstand. Let's assume I created a function to start an event loop:

#[no_mangle]
pub extern "C" fn ffi_start_event_loop() {
    winit::event_loop::EventLoop::new().unwrap();
}

When I call it I get the following output:

% dart --enable-experiment=native-assets run example/main_thread.dart 
ffi_sum_on_current_thread(1, 2) = 3
thread '<unnamed>' panicked at /Users/matan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/winit-0.30.5/src/platform_impl/macos/event_loop.rs:225:14:
on macOS, `EventLoop` must be created on the main thread!

This also occurs within a new thread:

#[no_mangle]
pub extern "C" fn ffi_start_event_loop() {
    // Start a thread to make it the main thread.
    std::thread::spawn(|| {
        let event_loop = EventLoop::new().unwrap();
    });
}

My understanding is that a program that doesn't interact with the OS's window system, like a pure Dart program in the standalone embedder, doesn't really have a main thread in that sense.

I wonder if that's true on MacOS ARM. I assume Dart is similar to the JVM in this regard:

https://stackoverflow.com/questions/28149634/what-does-the-xstartonfirstthread-vm-argument-do-mean, and Dart would need to provide something like -XstartOnFirstThread and I couldn't do it myself.

[liamappelbe]

Interesting. Sounds like on Mac the definition of "main thread" is whatever thread it runs the native main() on. That's annoying. We can't really add a -XstartOnFirstThread either, without thread pinning infrastructure (otherwise even if we start the main isolate on thread 0, it might switch threads), or rewriting the standalone embedder to work like the flutter embedder with a custom event loop for the main isolate.

Maybe a lighter-weight solution would be for the standalone embedder to expose the main thread to FFI somehow?

[matanlurey]

Maybe a lighter-weight solution would be for the standalone embedder to expose the main thread to FFI somehow?

It sort of feels terribly hacky, but yeah something like:

@Native<Int32 Function(Int32, Int32)>(runOnMainThread: true)
external int ffi_sum_on_main_thread(int a, int b);

I am sure this comes with its own problems 😅

[knopp]

On macOS the main thread is the first application thread (for which pthread_main_np() returns 1). There are multiple assumptions about this inside CFRunLoop and GCD and there is no way around this sadly.

[munrocket]

How to solve this issue in objective-c wrapper? Synchronous dispatch freezes, and asynchronous does not update values.

int ffi_sum_on_current_thread(int a, int b) {
  return a + b;
}

int ffi_sum_on_ui_thread(int a, int b) {
  __block int result = 0;
  dispatch_async(dispatch_get_main_queue(), ^{ //sync freezes
    result = a + b;
  });
  [NSThread sleepForTimeInterval: 3.0];
  return result; //result not updated
}

Repo for C: https://github.com/munrocket/dart-ffi-mac-ui

Same issue for Flutter? I wan't to develop my FFI bindings on MacOS.