Render directly to lower-level graphics surface

[ntadej]

Like maplibre-gl-native, this library could support rendering to lower-level graphics surfaces, ideally

• OpenGL framebuffer
• Metal surface
• DirectX surface

This would allow easier integration to custom apps or other libraries like Qt.

[maxammann]

Possibly related to #47 and #28.

The surface is handled through: https://github.com/rust-windowing/raw-window-handle

All surfaces supported by it can be supported using wgpu.

[MarijnS95]

Thinking about this, winit currently implements RawWindowHandle on their own structure where it forwards ndk's NativeWindow: should the NDK perhaps implement HasRawWindowHandle directly on NativeActivity NativeWindow?

Will that help you out at least a little bit here?

[maxammann]

Will that help you out at least a little bit here?

Not really, the code to turn a NativeActivity into a RawWindowHandle is simple (and unsafe :D) https://github.com/rust-windowing/winit/blob/master/src/platform_impl/android/mod.rs#L785

We actually need to avoid using NativeActivity in maplibre-rs completely.

[MarijnS95]

Sorry, I meant NativeWindow (wrote it correct once, wrong the second time).

[maxammann]

Sorry, I meant NativeWindow (wrote it correct once, wrong the second time).

And I did read too quickly :D Actually NativeWindow/android.view.Surface sounds like the thing we need! I never worked with the NDK directly, so how would we get a NativeWindow from an android Surface? I would imaging that the surface gets created in Kotlin. After that it need to be passed to the NDK somehow.

EDIT: I think we somehow can pass a Surface directly via JNI? https://stackoverflow.com/a/24313036
EDIT: Here is an example:

• https://github.com/android/ndk-samples/blob/87ae9bff54a9264e8b92515bfbda055d4ebe28bc/native-media/app/src/main/java/com/example/nativemedia/NativeMedia.java#L347
• https://github.com/android/ndk-samples/blob/87ae9bff54a9264e8b92515bfbda055d4ebe28bc/native-media/app/src/main/cpp/native-media-jni.c#L507

[maxammann]

@MarijnS95 So, yes I suspect we could already do this by passing a surface to Rust and then just implementing an unsafe RawWindowHandle on top of it. Maybe no changes in the NDK are required for that. The difficult part will probably be to handle inputs. winit depends on the NativeActivity for that.

Status Quo: Rendering to a surface probably works already fine on iOS and Android. Handling input is a different story though.

[MarijnS95]

It looks like you can get that info straight from #28 / rust-mobile/ndk#262 (comment): apparently you can use that for interop with OpenGL textures, though the Surface docs also point to https://developer.android.com/reference/android/opengl/EGL14#eglCreateWindowSurface(android.opengl.EGLDisplay,%20android.opengl.EGLConfig,%20java.lang.Object,%20int[],%20int). This should at least work on a NativeWindow, but that's far out of scope for the android.view.Surface docs...

EDIT: I think we somehow can pass a Surface directly via JNI? https://stackoverflow.com/a/24313036 EDIT: Here is an example:

Yes, as long as you can get hold of a SurfaceTexture in Kotlin you can use the aforementioned NDK PR to wrap the JNI pointer and acquire a native window on it, use that as normal.

• https://github.com/android/ndk-samples/blob/87ae9bff54a9264e8b92515bfbda055d4ebe28bc/native-media/app/src/main/java/com/example/nativemedia/NativeMedia.java#L347
• https://github.com/android/ndk-samples/blob/87ae9bff54a9264e8b92515bfbda055d4ebe28bc/native-media/app/src/main/cpp/native-media-jni.c#L507

Ah yes, you can apparently also go from a JNI Surface directly to a NativeWindow without the above - hopefully that also means not having to go through ASurfaceTexture_updateTexImage which seems to incur an extra copy. Not sure if that poses limitations on being able to import it to GL though; Android UI should already be implemented in GL so that may not be a problem.
The NDK Doesn't have this function yet, but I have the missing functionality in progress somewhere.

I haven't used the APIs for myself yet so we're just piecing together function return types into function arguments at this point, seems there are multiple ways of doing this. In either scenario it seems that this needs to go through NativeWindow.

Status Quo: Rendering to a surface probably works already fine on iOS and Android. Handling input is a different story though.

I don't think an input queue can be attached to a non-NativeActivity activity, and it somehow has to "deactivate" itself when not interacting with the surface wrapped here. You're probably on your own reimplementing that if you can come up with some common API to forward Android and iOS input events to Rust... :(

[MarijnS95]

Ah yes, the docs for ASurfaceTexture_acquireANativeWindow() state:

Returns a reference to an ANativeWindow (i.e. the Producer) for this SurfaceTexture. This is equivalent to Java's: Surface sur = new Surface(surfaceTexture);

So there's no point going through a SurfaceTexture if you can get hold of a Surface, albeit I don't remember if you can get hold of the latter in Java/Kotlin view APIs. Then it's up to preference whether to pass a Surface to the NDK and turn it into a NativeWindow, or pass a SurfaceTexture into the NDK and turn it into a SurfaceTexture there, subsequently acquire the NativeWindow for it.

[maxammann]

I don't think an input queue can be attached to a non-NativeActivity activity, and it somehow has to "deactivate" itself when not interacting with the surface wrapped here. You're probably on your own reimplementing that if you can come up with some common API to forward Android and iOS input events to Rust... :(

You are probably right. I would actually be quite happy to remove the winit dependency. Using native gesture handlers on iOS and Android would also probably increase the "smothness" of inputs. But it's very tool that the surface topic is not that hard.

We should create an experiment for iOS and Android soon.

[MarijnS95]

I haven't fiddled with winits events too much (still have to do that in one of our own crates) but it's probably better to have access to the raw finger/touch API for creating the smoothest experience possible. I still have some C++ code lying around for smooth panning and zooming with one or more fingers - no matter if the user accidentally touches the screen with >2 fingers - and it's unclear whether I can port that to winit in the same capacity.

Let us/me know if you need anything to change or be added in the NDK. I went ahead and added the missing JNI android.view.Surface support in rust-mobile/ndk#272 before you'd need it, perhaps together with the SurfaceTexture PR that's also going to come in handy.

[maxammann]

Successfully rendered to an Android Surface :)

[maxammann]

Closing in favor of this one: #28