SurfaceTexture NDK bindings

[lattice0]

Are there plans to support OpenGL?

[dvc94ch]

there is gl, glutin and probably a dozen other rust crates that provide opengl bindings. I doubt it's in scope for this project, as it's not android specific.

[lattice0]

I've seen one pure rust glutin example that works on Android. However, if I have a Kotlin SurfaceTexture and want to render into it with native code, I need the NDK bindings so I can do stuff like

    egl.eglCreateWindowSurface(eglDisplay, eglConfig, surfaceTexture, null);

that is, pass the surfaceTexture pointer back and forth.

[MarijnS95]

eglCreateWindowSurface is an EGL binding, not an NDK binding. As David said, GL is a generic API that has nothing to do with the specifics of the NDK; would you mind updating this issue and title to be more specific about whatever problem you may be facing?

For starters you could look at how glutin uses eglCreateWindowSurface:

https://github.com/rust-windowing/glutin/blob/279b5a406878ce3638e1b80527ae07958eb6c12b/glutin/src/api/egl/mod.rs#L540-L545

Then, is there any reference code (ie. written in Kotlin, Java or C++) that converts a Kotlin SurfaceTexture to an EGLSurface? It seems that a SurfaceTexture wraps / streams to an OpenGL ES Texture; if you have the texname of this texture (and the appropriate OpenGL context) you might be able to interact with it directly.

[lattice0]

@MarijnS95 in flutter, it creates a SurfaceTexture for me, and it looks like there's no way to get the texName from inside a SurfaceTexture, amd I right?

[lattice0]

Don't I need at least https://developer.android.com/ndk/reference/group/surface-texture? I think I need to at least attach this SurfaceTexture to the opengl context. The problem is, how do I get a SurfaceTexture on the Rust side given only the texId...

ASurfaceTexture_fromSurfaceTexture(JNIEnv *env, jobject surfacetexture) is out of question because I want to call Rust code directly from Flutter, so no JVM, the JVM is only used to create the SurfaceTexture using Flutter's API

[MarijnS95]

I this hinges on ASurfaceTexture_acquireANativeWindow as ANativeWindow is what can be passed into eglCreateWindowSurface.

You'll have to ask @dvc94ch for Flutter support, as I don't have the slightest clue. If Flutter hands you a SurfaceTexture and is implemented in native code, is there a possibility that it simply wraps ASurfaceTexture which you can use to get access to the aforementioned API?

Again, you should rewrite this issue and title to document what you want to achieve, rather than requesting something opaque.

[dvc94ch]

when building a flutter-rs plugin for video playback, flutter had a Texture widget. So you'd create an opengl texture, render the paced frames to the texture and pass the texture id to the Texture widget on the flutter side. When it comes to embedding flutter in a native android ui, don't really know. Don't have any experience with native android ui's.

[lattice0]

@dvc94ch ok, changed the title. I don't want to embed a flutter view on Android, I want to render to a flutter texture in Rust side. It turns out that to create a texture on Flutter you need to call kotlin/java code:

  override fun onMethodCall(@NonNull call: MethodCall, @NonNull result: Result) {
    if (call.method == "getPlatformVersion") {
      result.success("Android ${android.os.Build.VERSION.RELEASE}")
    } else if (call.method=="create_texture") {
      Log.d(LOG_TAG, "createVideoRenderer called")
      val entry: TextureRegistry.SurfaceTextureEntry = texture_registry.createSurfaceTexture();
      val surfaceTexture: surfaceTexture = entry.surfaceTexture();
      //Store the surfaceTexture somewhere for rendering later

It gives me a SurfaceTexture, and to render to it in native side the only way I can see is through https://developer.android.com/ndk/reference/group/surface-texture#group___surface_texture_1ga8928719bfd3fc8a26595c7c8b0040aa2, or at least I though it helped me, if I could access the texture id. How??

Anyways, there is still a major problem: I want to call Rust code through Dart's FFI, which completely ignores the JVM, but ASurfaceTexture_fromSurfaceTexture(JNIEnv *env, jobject surfacetexture) obviously uses JVM. I could call it from the Kotlin side, but will the returned ASurfaceTexture * be valid on the Rust side?

[dvc94ch]

ah I see. looks like what you're missing is some jni bindings to the android flutter embedder. your dart application uses the ffi to call into your rust code, which then uses the jni to create a surface texture. you can then render to the surface texture using opengl.

[lattice0]

@dvc94ch I'm a bit lost. If I load the rust .so through Flutter/Dart, then it wouldn't have access to the JVM, so I cannot call the android embedder, which is Kotlin/Java, right? Or is the embedder C++ and just called through Kotlin/Java?

[dvc94ch]

all android applications are java applications. your activity initializes the flutter android embedder, which initializes the flutter engine which loads your dart code. you need to initialize the ndk-context from your activity and then your dart code can call your rust code, which can use the ndk-context to create the surface texture. if you're asking how cargo-apk helps you with this, the answer is it doesn't. You need to use xbuild or roll your own.

[lattice0]

Yes, the Kotlin code:

  override fun onMethodCall(@NonNull call: MethodCall, @NonNull result: Result) {
    if (call.method == "getPlatformVersion") {
      result.success("Android ${android.os.Build.VERSION.RELEASE}")
    } else if (call.method=="create_texture") {
      Log.d(LOG_TAG, "createVideoRenderer called")
      val entry: TextureRegistry.SurfaceTextureEntry = texture_registry.createSurfaceTexture();
      val surfaceTexture: surfaceTexture = entry.surfaceTexture();
      //Store the surfaceTexture somewhere for rendering later
      //here

which is called from Flutter as a plugin method call, initializes a SurfaceTexture bound to a Flutter texture. Technically I can call C++ on line here on code above and get a pointer to the C++ ASurfaceTexture, then transform this pointer into a java long and pass back through the Flutter method call to Dart code, and then pass it through dart to Rust code, and then Rust code can call

int ASurfaceTexture_attachToGLContext(
ASurfaceTexture *st,
uint32_t texName
)

to attach whatever texture it creates, and also call things like ASurfaceTexture_updateTexImage(ASurfaceTexture *st) and ASurfaceTexture_attachToGLContext(ASurfaceTexture *st, uint32_t texName) to update/render the images.

Is this what you meant? There's still the issue of having to call https://github.com/flutter/engine/blob/8ab4124eeff5b259153e56fadfaabf670655687b/shell/platform/android/io/flutter/embedding/engine/renderer/FlutterRenderer.java#L417 every time a new frame should be rendered, which would be hard to do from Rust as it's Java code. Also, I find it very ugly and unsafe to use Kotlin to call C++ to get the pointer, then pass as a number to Flutter, which passes to Rust, which casts to ASurfaceTexture (by the way, I'd need ASurfaceTexture in Rust which I think should be mapped by this lib, as it's from the NDK).

Is there something I'm missing here?

[dvc94ch]

yep, we can add bindings for ASurfaceTexture. everything else is out of scope. so ndk-sys already contains ASurfaceTexture, you'd just need to add a small high level wrapper to ndk or use ndk-sys directly if you don't want to bother.

[lattice0]

@dvc94ch it'd be nice for me to do a PR to get high level support for ASurfaceTexture. On which file should I add it? https://github.com/rust-windowing/android-ndk-rs/tree/master/ndk/src

[dvc94ch]

I'd create a new one called surface_texture.rs to match the documentation

[lattice0]

@dvc94ch is it ok? #267

[lattice0]

Do you think my plan is ok? It's very ugly to pass pointers around as numbers between JVM/C++ and then Rust, over Flutter. Also, everytime the rust side writes a new frame to a texture, it needs to call Flutter's markTextureFrameAvailable which I can only do by having a JVM pointer, but Rust does not have a JVM pointer as it's a .so loaded by Dart. I could pass one from Kotlin to Flutter as a number also, but I find all of this very wrong.

[dvc94ch]

Why don't you dlopen your rust so and initialize ndk-context from your kotlin activity?

[lattice0]

Because if I open the rust .so from kotlin then whenever I want to pass a message from flutter to rust it needs to go through a platform channel which is very slow and requires me to create maps and deep copy data, etc. Therefore I have to open the rust side from dart but create the surface texture from kotlin, but also call markTextureAvailable's kotlin function from Dart every time there's a new frame.

…

-------- Mensagem Original -------- Ativo 22 de abr. de 2022 23:57, David Craven escreveu:

Why don't you dlopen your rust so and initialize ndk-context from your kotlin activity? — Reply to this email directly, [view it on GitHub](#262 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABSTHALSTIG45XTAZG7GQUTVGNRK3ANCNFSM5TTMI74Q). You are receiving this because you authored the thread.Message ID: ***@***.***>

[dvc94ch]

You should be able to dlopen it from java and from flutter

[maxammann]

If I understood correctly, #267 adds support to get a native window from a surface texture, right? That is awesome!
Its a feature missing in winit afaik

[dvc94ch]

So wasn't 100% sure how multiple dlopens to the same library work:

If the same shared object is opened again with dlopen(), the same object handle is returned. The dynamic linker maintains reference counts for object handles, so a dynamically loaded shared object is not deallocated until dlclose() has been called on it as many times as dlopen() has succeeded on it. Constructors (see below) are called only when the object is actually loaded into memory (i.e., when the reference count increases to 1).

https://man7.org/linux/man-pages/man3/dlopen.3.html

[lattice0]

@maxammann I didn't do for this purpose but it looks like yes, it's possible :)

[MarijnS95]

@lattice0 Did you succeed opening the library and initializing the pointers?

If you can call C++ from Kotlin and perform whatever operation you happen to need there, you can also do that directly with Rust.

[lattice0]

@MarijnS95 gonna try soon

[MarijnS95]

Relevant links for this issue:

• Using a Surface == NativeWindow directly, instead of going through SurfaceTexture: Render directly to lower-level graphics surface maplibre/maplibre-rs#53 (comment)
• That suggestion actually working: Postponed: Experimentally render to an Android surface maplibre/maplibre-rs#99
• Tying that functionality into glutin to support any NativeWindow and not just ndk_glue::native_window(): Use glutin to render to Surface/NativeWindow without winit/ndk_glue rust-windowing/glutin#1324

[lattice0]

I think you're saying that it's possible to open twice and get it loaded as just one lib, so any pointer or static var in kotlin is also accessible in dart. I was thinking that dart could be loading it as a secondary lib, I don't know exactly how dynamic library loading works for Dart on Android, gotta research but this seems promising if true. So I should load twice at startup and then I can store the surface pointer in a rust static map or something, so it can be acessed in both sides. One strange thing though is that I'd have to have JNI stuff embedded into the rust .so but I think that's ok. 🤔🤔🤔🤔🤔🤔

…

-------- Mensagem Original -------- Ativo 23 de abr. de 2022 00:11, David Craven escreveu:

You should be able to dlopen it from java and from flutter — Reply to this email directly, [view it on GitHub](#262 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABSTHAKA64ZHTKLPWKWRMJ3VGNS6LANCNFSM5TTMI74Q). You are receiving this because you authored the thread.Message ID: ***@***.***>