Data receiving actor goes to Terminated state due to possible cause: channel flooding
#6925
-
|
Hi! Akka version. Environment.
Problem.
Bad solution. Question. The recipient processes messages one at a time (no more than one message in his queue), while the sender sends thousands of messages. Why might this happen? Another attempts. NOTE! |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 15 replies
-
|
@ArtyomAfanasovBS what would cause the actors to shut down? Actors running over Akka.Remote can't do that without end-user code forcing them to even if there's a connectivity problem. Are you using Akka.IO to send these messages over the network? |
Beta Was this translation helpful? Give feedback.
-
|
@Aaronontheweb , thanks for the quick response! |
Beta Was this translation helpful? Give feedback.
-
|
After that, my code executes the logic to recreate the actor system. The call to recreate the actor system is called if connection between the sender and receiver is lost |
Beta Was this translation helpful? Give feedback.
-
|
I still don't see any indication of actors terminating, but I do see that the Akka.Remote connection has become quarantined (which renders it unusable.) That's caused by Since you're using WIFI you're going to be running at a higher per-message latency due to the inherent lossy-ness of the connection. Your fundamental problem is a backpressure issue, which I explain in broad terms in this video:https://www.youtube.com/watch?v=0KYoIvrM9VY What I would do in your case: you need some way to emit a batch of events to the receiver and then force the sender to wait until the receiver has processed them. The best way to implement this is through having the sender deliver messages back to the sender + have the sender use actor behavior-switching to toggle between sending / waiting for receiver to ACK behaviors. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! I'll try it! |
Beta Was this translation helpful? Give feedback.
-
|
The problem is that messages reach the recipient "one at a time" rather than "many" at once. Therefore, the recipient processes his mailbox slowly - there is always 1 message in it. But the sender sends thousands of messages to the recipient. |
Beta Was this translation helpful? Give feedback.
-
|
In the |
Beta Was this translation helpful? Give feedback.
-
|
The configuration does not contain settings that limit the mailbox to a maximum of one message: |
Beta Was this translation helpful? Give feedback.
-
I'm not sure about Akka.IO. How can I know if Akka.IO is being used? I send messages between actors via |
Beta Was this translation helpful? Give feedback.
-
|
Good afternoon! :) I added batch messaging to the configuration, but it didn't help. The recipient's mailbox is still very limited. Sometimes it holds 2-3 messages (rather than one as I said earlier), but the vast majority of the time it only holds one message. Of course, I could try to use the |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the I guess "tiny capacity of recipient mailbox" is a topic for a separate question? |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps there is some kind of TCP problem? |
Beta Was this translation helpful? Give feedback.
-
|
The mailbox has an unlimited size - it's just that actors tend to instantly process messages when they receive them, so seeing a queue backlog build up should only happen under rare circumstances when things are extremely busy. You're mistaking the low queue length for "low capacity" which isn't the case at all. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I got the terms mixed up, thank you). I also did an experiment. I artificially increased the processing time of received messages. As a result, the messages accumulated in the recipient's mailbox, even if the participants were on different machines and communicating over Wi-Fi. Thus, to answer the question of my topic:
|
Beta Was this translation helpful? Give feedback.
-
|
Excellent, glad you resolved it @ArtyomAfanasovBS - please let us know if there's anything else we can help you with. |
Beta Was this translation helpful? Give feedback.
I still don't see any indication of actors terminating, but I do see that the Akka.Remote connection has become quarantined (which renders it unusable.) That's caused by
/systemmessages not being able to get processed / acknowledged on-time by Akka.Remote's reliable delivery system and that happens when the network becomes excessively busy or destabilizes.Since you're using WIFI you're going to be running at a higher per-message latency due to the inherent lossy-ness of the connection.
Your fundamental problem is a backpressure issue, which I explain in broad terms in this video:https://www.youtube.com/watch?v=0KYoIvrM9VY
What I would do in your case: you need some way to emit a batch o…