[vm/ffi] Support asynchronous callbacks

[sjindel-google]

Update Sept 6, 2023: Asynchronous callbacks with void returns can now be done with NativeCallable.listener from Dart 3.1.

Update March 12, 2020: Asynchronous callbacks can be done through the native ports mechanism. See the samples in samples/ffi/async and corresponding c code. In the future we would like to add a more concise syntax that requires less boilerplate code.

An asynchronous callback can be invoked outside the parent isolate and schedules a microtask there.

[dcharkes]

Asynchronous callbacks can be invoked on an arbitrary thread.

Synchronous callbacks cannot be invoked on an arbitrary thread because it would break the Dart language semantics: only one thread executing Dart per isolate.

Note that we need a solution for managing isolate lifetime when registering an asynchronous callback to be called from any thread in C. If the isolate died, and the callback is called, we have undefined behavior. A possible solution suggested by @mkustermann is to provide call-once callbacks, and keep the isolate alive until they are all called exactly once. Or we could have a counter or an arbitrary condition that we can check.

[dcharkes]

For inspiration, Kotlin requires the C library to be aware of Kotlin when doing non-main-thread-callbacks:

If the callback doesn't run in the main thread, it is mandatory to init the Kotlin/Native runtime by calling kotlin.native.initRuntimeIfNeeded().

Source: https://github.com/JetBrains/kotlin-native/blob/master/INTEROP.md#callbacks

[sjindel-google]

It's not the thread which the callback is tied to, it's the isolate. So our callbacks may be invoked on any thread so long as that thread has called Dart_EnterIsolate.

[dcharkes]

It's not the thread which the callback is tied to, it's the isolate. So our callbacks may be invoked on any thread so long as that thread has called Dart_EnterIsolate.

I presume Dart_EnterIsolate also (could) prevent isolates from being shutdown or killed.

[sjindel-google]

Another solution to managing lifetime is that the asynchronous callbacks need to be explicitly de-registered, and the isolate lives until all callbacks are de-registered.

Another challenge is knowing which Isolate the callback should run on. In AOT we have only one trampoline per target method, and there may be indefinitely many isolates from the same source at runtime.

[derolf]

Does it mean that calling back into Dart works as long as Dart_EnterIsolate is called before invoking the callback? How does the native code know that?

[dcharkes]

We have not made a design for this feature yet, we'll explore the possibilities when creating a design.

[derolf]

Okay, we have a workaround for it:

• Native: Thread X wants to run callback
• Native: Thread X adds the callback to a global list of pending callbacks
• Native: Thread X send sigusr1 to the process
• Dart: Main Isolate watches sigusr1 (using ProcessSignal.watch) and gets woken up
• Dart: Main Isolate calls via FFI into a function that executes all pending callbacks

[dcharkes]

@derolf nice!

Would you like to provide a minimal sample and contribute it as a PR for others to learn from for the time being?

[derolf]

It didn’t work so far because Flutter itself is using SIGUSR1/2 for hot reloading.

However, I saw that a NativePort can be sent through FFI to the c-layer. Do you have any example how to send something to that nativePort from C?

[dcharkes]

@derolf, to my understanding the NativePort solution only works if you own the Dart embedder yourself.

cc @mkustermann

Linking flutter/flutter#46887.

[derolf]

Okay, I implemented a way that works reliable and clean.

The queue:

On the native side, you need a threadsafe queue that allows to:
-- Enqueue callbacks from some thread that you want to execute in the main isolate (enqueue)
-- Have a function exposed through ffi that blocks until there are callbacks in the queue (wait)
-- Have a function exposed through ffi that executes pending callbacks (execute)

The dance:

Now, you spawn a slave isolate that waits on the queue and then sends a message to the main isolate, and waits again, ... (forever loop)

The main isolate receives this message and calls into ffi to execute all pending callbacks.

If some thread wants to dispatch a callback, you enqueue it instead of calling it. This will wakeup the slave isolate and that will send the said message to main and main will deliver the callback.

That's it. Less than 100 LOC to do it all.

[mkustermann]

Have a function exposed through ffi that blocks until there are callbacks in the queue (wait)

That is slightly problematic in an event-loop based system. The synchronous blocking will prevent the isolate from processing any other messages (timers, socket i/o, ...).

We'll write an example on how this can be done already now with our dart_native_api.h and ports.

[derolf]

@mkustermann The problem is that none of the dart_native_api.h functions are available/exported in Flutter, so I can't use them. Happy to see your example how you make that work!

The "slave" isolate's one and only job is to do this waiting on the queue. It's doing nothing else. Here's its code (ripped out of the codebase):

class _SlaveIsolateMessage {
  _SlaveIsolateMessage(this.port, this.isolate);
  final SendPort port;
  final int isolate;
}

void _slaveIsolate(_SlaveIsolateMessage msg) {
    print("_slaveIsolate running for ${msg.isolate}");
    while (_dart_ffi_wait_for_callbacks(msg.isolate) != 0) {
      print("_slaveIsolate has callbacks for ${msg.isolate}");
      msg.port.send(1);
    }
    print("_slaveIsolate done for ${msg.isolate}");
    msg.port.send(0);
  }

final _dart_ffi_wait_for_callbacks = dylib.lookupFunction<Int32 Function(Int32), int Function(int)>('dart_ffi_wait_for_callbacks');

The isolate int used to distinguish different native callback queues as we plan to have more than one "main" isolate.

[dcharkes]

The samples have landed, see referenced commits.

[insinfo]

Some status of when this will be available, I saw that the Realm team is just waiting for this feature to be able to launch a Realm for Flutter

[dcharkes]

Asynchronous callbacks can be today done through the native ports mechanism as illustrated in samples/ffi/async.

In the future we would like to add a more concise syntax that requires less boilerplate code.

[katyo]

Asynchronous callbacks can be today done through the native ports mechanism as illustrated in samples/ffi/async.

In the future we would like to add a more concise syntax that requires less boilerplate code.

As I understand we still depends from APIs declared by dart_native_api.h on the native side so this technique still does not usable with flutter.

[mkustermann]

As I understand we still depends from APIs declared by dart_native_api.h on the native side
so this technique still does not usable with flutter.

The dart:ffi library exposes the native api symbols now to Dart code via NativeApi.postCObject (as well as NativeApi.closeNativePort, NativeApi.newNativePort). Dart code can make a ReceivePort, obtain it's native port via receivePort.sendPort.nativePort, give that native port as well as the NativeApi.postCObject function pointer to C code. The C code can then post messages on the port, which Dart code can react on.

@katyo Would that work for you?

[provokateurin]

@mkustermann Can you give a small example with the method you're mentioning? I need this for my app, because it's basically a deal breaker and I don't exactly understand how your method should work in code.
Thanks in advance

[insinfo]

@dcharkes

Another idea we had was to compile a subset of Dart that does not interact with the GC and the VM features...

That would be very interesting, and I think it has a lot to do with what I requested here. dart-lang/language#1758

I think having a strict mode in dart is somewhat close to C# unsafe which allows running parts of code outside the garbage collector and doing raw pointer operations with a better syntax

https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/unsafe

I think it would be very positive to have this strict mode in dart to be able to optimize parts of a code and maybe it would also open up the possibility of having multithreading dart-lang/language#333 in this strict mode, that would be fantastic to not need to leave dart to do certain things, without needing the complexity having to rely on another language or cross-platform software development stack with rust or c++

[Keithcat1]

How would I keep track of which Dart object my callback was bound to? Can I use a closure?

[liamappelbe]

How would I keep track of which Dart object my callback was bound to? Can I use a closure?

@Keithcat1 At the moment you can only create FFI callbacks from static functions, but we're planning to add closure support soon. The infrastructure we needed to build for async callbacks will also let us support closures: #52689

[LEggcookies]

And would you need to do anything in Dart with the same data structures besides processing the callbacks?

@dcharkes
I encountered an obstacle.

As shown in the diagram above, device->onIncoming does not execute within the thread called by connectDevice, causing the receiveCallback to fail. I need to send all these parameters back to Flutter for processing. I tried using ReceivePorts, but I don't know how to wrap so much content in a Dart_CObject, and how to extract the corresponding parameters from the message in Flutter. I hope to get your help and would be greatly appreciative!

[dcharkes]

Asynchronous callbacks with void returns can now be done with NativeCallable.listener from Dart 3.1.

[nmfisher]

It seems these aren't usable via dart2wasm yet. Any suggestions for callbacks from native code to calling an async Dart function from native code compiled to wasm?

[mraleph]

Any suggestions for callbacks from native code to an async Dart function when compiled to wasm?

You seem to be asking the question in the opposite direction. So just to clarify you have a Dart function that returns a Future and you want to call it from native code and pretend that this call is synchronous? Can you maybe post an example code of what you are trying to achieve.

[nmfisher]

Exactly what you suggested - calling an async Dart function from native code and either treating the result as synchronous, or at least some other way of propagating a result back to native code that's compatible with wasm. My first attempt was using native Dart ports, but these seem not to be supported under wasm.

e.g. something like the following:

native_code.c (compiled to wasm)

// function pointer to the Dart function loadResource 
// assume we've already set this appropriately   
void* (*_load_resource_dart)(const char* path); 

// this is a native function that will call an async Dart function
void load_resource_from_flutter(const char* asset) {
    void* flutterAssetData = load_resource_dart(asset);
}

Dart

Future<Pointer<Void>> loadResource(Pointer<Char> path) async {
    var data = await rootBundle.load(path.toDartString());
    var dataPtr = ... // allocate memory & copy into it, etc
    return dataPtr;
}

[mraleph]

@nmfisher well, that's not going to work. Consider the following: to call asynchronous function synchronously you need to pause Wasm execution until async function completes then resume it later.

What you can do is to restructure your native code to use callbacks and thread some additional context through them manually:

// function pointer to the Dart function loadResource 
// assume we've already set this appropriately   
void* (*_load_resource_dart)(const char* path, void* ctx, void (*loaded)(void* ctx, void* data)); 

// this is a native function that will call an async Dart function
void load_resource_from_flutter(const char* asset) {
    void* ctx = ...; // Some context that needs threading through async code.
    load_resource_dart(asset, ctx, &flutter_resource_loaded);
}

void flutter_resource_loaded(void* ctx, void* flutterAssetData) {
   // Asset loaded.
}

[nmfisher]

Thanks @mraleph - that's the approach I've been using so far, I was just having issues with concurrency on the native side when using dart2js and then function pointers when using dart2wasm, so I was wondering if there was a Dart-first approach that was better.

I'll keep plugging away and see if I can get it to work as expected.

[nmfisher]

Actually I suppose this is a good a place as any to ask - is it even currently possible to use Pointer.fromFunction via dart2wasm?

e.g.

import "dart:ffi";

void foo() {
  print("foo");
}

void main() {
  var ptr = Pointer.fromFunction<Void Function()>(foo);
  print(ptr);
}

dart --enable-asserts pkg/dart2wasm/bin/dart2wasm.dart foo.dart foo.wasm
d8 --experimental-wasm-gc --experimental-wasm-type-reflection pkg/dart2wasm/bin/run_wasm.js -- foo.mjs foo.wasm

fails with:

NoSuchMethodError: method not found: '_nativeCallbackFunction'
Receiver: null
Arguments: []
    at NoSuchMethodError._throwUnimplementedExternalMemberError (wasm://wasm/0008dd2a:wasm-function[188]:0x13763)
    at _nativeCallbackFunction (wasm://wasm/0008dd2a:wasm-function[589]:0x1d0f6)
    at _#ffiCallback0 (wasm://wasm/0008dd2a:wasm-function[585]:0x1d0be)
    at main (wasm://wasm/0008dd2a:wasm-function[56]:0xfb89)
    at main tear-off trampoline (wasm://wasm/0008dd2a:wasm-function[58]:0xfbab)
    at _invokeMain (wasm://wasm/0008dd2a:wasm-function[67]:0x101de)
    at Module.invoke (/tmp/foo.mjs:168:28)
    at main (pkg/dart2wasm/bin/run_wasm.js:352:21)

[dcharkes]

This issue is currently tracking all kinds of asynchronous use cases.

The basic use case of an asynchronous call is now supported via https://api.dart.dev/stable/3.2.4/dart-ffi/NativeCallable/NativeCallable.listener.html.

Further work for other use cases is tracked in more specific issues:

• Callbacks on the platform thread: Platform Isolate flutter/flutter#136314
• Callbacks into already running isolates: Shared Memory Multithreading language#3531
• Callbacks into temporary isolates [vm/ffi] NativeCallable.onTemporaryIsolate #54530

(Side note: all of the above use cases are synchronous from the native code point of view, but asynchronous from the spawning Dart isolate point of view.)

I'll close this issue. Please open a new issue if your use case is not covered by one of the remaining open issues.

Thanks @liamappelbe for implementing NativeCallables! 🎉

[insinfo]

Can you use NativeCallable for system audio API callbacks or miniaudio libraries like PortAudio?

[maks]

@insinfo you beat me to asking that :-)
though I think the answer is no given that NativeCallable is still async via a Port and maybe this is more aligned with the work being done in #136314 ?

[Keithcat1]

I doubt it. AFAIK NativeCallable won't wait for Dart to run your callback on the main thread, so from Miniaudio's point of view, you it assumes that since the callback has already finished, you aren't using the data anymore and it is free to do something else with it. NativeCallable will give you pointers to Miniaudio buffers, but not whatever data was in those buffers (the buffers can change before you get a chance to read it!). The Miniaudio device event APIs would work, though and Miniaudio Osaudio (miniaudio/extras/osaudio) can be used to record or play audio data without using threads, it'll just block until the ammount you asked for is ready.

[insinfo]

It's a little sad and discouraging that we still don't have a pure Dart solution for Audio API callbacks, Dart ffi won't be complete without this, the team behind Dart is very talented and I hope that a solution will be implemented in the future

[dcharkes]

Can you use NativeCallable for system audio API callbacks or miniaudio libraries like PortAudio?

I'm assuming the callbacks (1) come from another thread than the mutator thread of the isolate, and (2) need to be blocking and return an answer. If this is the case one of the following:

• Callbacks into already running isolates: Shared Memory Multithreading language#3531
• Callbacks into temporary isolates [vm/ffi] NativeCallable.onTemporaryIsolate #54530

The first one is being discussed. The second one is currently on hold because the first is a more general solution.

It's a little sad and discouraging that we still don't have a pure Dart solution for Audio API callbacks, Dart ffi won't be complete without this, the team behind Dart is very talented and I hope that a solution will be implemented in the future.

The Dart concurrency system (isolates) is designed to make isolated code easy to reason about. However, that doesn't align so well with interoperating with languages which do shared memory multi-threading. We're aware of this and trying to find a good solution. It might take a while to find a good solution, so please be patient. And if you're interested feel free to contribute to the discussion about shared memory multithreading. (P.S. With shared memory multi threading we're going to have things like ConcurrentModificationException etc. 😅 There's a reason why Dart doesn't have shared memory multi-threading until now.)

[maks]

Thanks @dcharkes for the pointer to the new proposal in the Shared Memory Multithreading PR, very exciting to see it being worked on!

And yes you are correct, most audio APIs seem to follow the model of application code needing to register a callback with the audio system, that then gets called back by a thread (which thread is determined by the audio system) which passes in a pointer to a buffer which it expects to be filled in with required number of audio samples by the time that callback function returns.

So from that proposal PRs doc, this:

• dealing with APIs which want to call back from native into Dart on an arbitrary thread.

is exactly the case for audio APIs. @dcharkes is commenting on that markdown doc in the PR the best way to contribute to the discussion?

[dcharkes]

@dcharkes is commenting on that markdown doc in the PR the best way to contribute to the discussion?

Yes!

[sanjaygholap]

Hi can you please help me to resolved this issue "cannot invoke a native callback outside of an isolate" Now what happened in the C++ DLL file. Internally, they started a separate thread with a callback. so how to handle this issue.https://stackoverflow.com/questions/78404125/cannot-invoke-a-native-callback-outside-of-an-isolate?noredirect=1#comment138225228_78404125

[liamappelbe]

@sanjaygholap That error message means you're using a sync callback (either NativeCallable.isolateLocal or Pointer.fromFunction). You need to use NativeCallable.listener instead.

[sanjaygholap]

same issue "Cannot invoke a native callback outside of an isolate".Within the C++ DLL file. A separate thread with a callback was started internally.

[liamappelbe]

@sanjaygholap It's not possible to hit that error message unless you're using a sync callback. Async callbacks follow this code path instead, which always returns before that error. Async callbacks are specifically designed to work from any thread, so they should work even if your DLL spawned a new thread. Double check that your callback is a NativeCallable.listener.

If you're sure that you have a NativeCallable.listener that is hitting that error, that would be a new bug. Open a separate issue with repro steps and I'll take a look.

[sanjaygholap]

@liamappelbe sir, we get same issue "Cannot invoke a native callback outside of an isolate".Within the C++ DLL file. A separate thread with a callback was started internally.
I create new issue please check "#55626"