Realm Quiescence

[lightsighter]

There are currently several issues with determining when it is safe to shutdown Realm. Currently, Realm has a shutdown method that should be invoked by the client when it is safe to shutdown the runtime. Users can pass in precondition events to the shutdown method to ensure that it is only invoked once all outstanding Realm operations (e.g., tasks, copies, etc.) have completed. However, there are some things that the client can ask Realm to do for which there is no way at the moment to safely know that the "effects" of such operations have finished running and it is safe to shutdown. For example:

1. Instance destruction: if invoked on a remote node, it requires an active message to be sent to the owner node to do the deletion and there's no way for the user to know when this message has been received and handled.
2. Sparsity map destruction: can be invoked on a remote node, and also trigger an active message. Further active messages can be sent to the subscriber nodes to clean up and then acknowledge messages sent back to the owner. All those need to be completed before it is safe to invoke shutdown.
3. Reservation deletion: might need to "pointer chase" around to find the current owner node of the reservation to actually destroy it.

There might be others that I'm missing, but those are at least a few different examples of things the client can't currently take into account when determining when it is safe to invoke shutdown. There are at least three different ways to address this:

1. Realm could explicitly return an Event from each of those different kinds of destruction operations that will encapsulate all the effects that need to be done to carry out the destruction. Then the user can be sure to chain those events into the precondition for the shutdown method. This places the burden mostly on the user and also means that they have to be tracking all those events long before the application might ever reach shutdown. It also adds additional event triggering overhead when it might not be necessary.
2. Realm can do custom tracking for each of these kinds of special shutdown conditions internally. This might involving have special internal counters for all the "effects" of these destructions to ensure that all the effects are done after shutdown has been invoked. Note that this also requires tracking all these effects even shutdown is invoked in case they are still in flight when shutdown is invoked which means you're also incurring the overhead all the time. It also feels like different modules in Realm might be prone to forgetting to add this tracking if they have special resources too.
3. Realm could develop a quiescence algorithm that would determine when all outstanding operations were complete including their effects and then shutdown automatically. This approach is probably the most tricky to implement as it's equivalent to developing hang detection since quiescence is effectively a kind of hang where there are no outstanding things still in flight. It would be general though and probably incur lower overhead during the actual execution of the program, especially if detection of quiescence doesn't even begin until at least one node has called wait_for_shutdown.

I think I'm fine with choosing any of these as solutions although I feel like option 3 is the most elegant and least prone to the bugs and mistakes.