Controller-Runtime running using a state over GRPC via StateClient crashes when it loses connection to the server

[utkuozdemir]

Observed it when using https://github.com/siderolabs/omni + https://github.com/siderolabs/omni-infra-provider-bare-metal.

The moment I stop Omni (ran via make docker-compose-up WITH_DEBUG=true) by CTRL+C, the provider's controller runtime crashes with:

Error: failed to run provider: failed to run components: controller runtime watch error: rpc error: code = Internal desc = server closed the stream without sending trailers
2024/12/19 14:05:27 failed to run: failed to run provider: failed to run components: controller runtime watch error: rpc error: code = Internal desc = server closed the stream without sending trailers

Let me know if more info is needed.

[utkuozdemir]

And it sometimes fails with this error instead of the one above:

controller runtime watch error: rpc error: code = Unavailable desc = error reading from server: read memu 1734683825790992862->grpc-conn: EOF