Lwt and OCaml 5

[Lupus]

OCaml 5 is just around the corner. Are there any plans/vision for Lwt in OCaml 5 world, featuring multicore and effect handlers? Will Lwt be evolved further, or it will be left at support mode, leaving future innovation to effect-native libraries like upcoming eio?

[raphael-proust]

@Sudha247 has already started some of this work with #860. It's a new module Lwt_domain which adds a function detach : ('a -> 'b) -> 'a -> 'b Lwt.t which detaches the call to the provided function to a separate domain (from Domainslib). It's a specific integration with a specific scope/purpose, but I'd encourage you to have a look.

As for a more general trend, this is not quite decided yet. In any system, mixing two separate event loops or scheduler is complicated. Which makes it non-obvious to integrate Lwt with libraries that provide I/O in other ways. One of the things that is being looked into is to have Lwt running on one domain and then some other libraries written in direct style running on other, dedicated domains. This needs some tricks to allow other domains to interrupt the sleeping Lwt scheduler and it's not sure how well it would work if at all.

Another possibility is to make an effects-based library which mimicks the scheduling behaviour of Lwt. It's not clear we can have an exact replica of the scheduling behaviour, but we can possibly get close enough that it's possible to swap from one to the other in most applications 🤞 . Basically, it'd mean making an interface-and-semantics-compatible effect-based replacement for Lwt.

On a more pragmatic side, I want to experiment with effects and Lwt and document does-and-donts and maybe make a tutorial for how to use some of the effects.

[Lupus]

making an interface-and-semantics-compatible effect-based replacement for Lwt

This would be really promising. Lwt is very popular within the OCaml ecosystem, and I expect effects to quickly raise in popularity, which would require tough choices from someone willing to consume this ecosystem if Lwt is no way compatible with effect-based applications.

[raphael-proust]

Of interest to the conversation: @talex5 's experimental Lwt-Eio bridge https://github.com/talex5/lwt_eio

[talex5]

Yes, that seems to be working well. I just used it to convert Dream to use a direct-style API, while continuing to use Lwt libraries internally (https://github.com/aantron/dream/pull/194/files).