Cadence behind loadbalancer not working properly #4737
Unanswered
tarun-discovery
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have deployed cadence in Kubernetes, the pods are running behind load balancer. The workers try to initialize cadence clients using namespace. service way since both are in same k8 cluster.
However when I have more than 1 cadence instance running, the task list is not stable, as below you see I have actually 4 pollers running, but it shows randomly sometimes 2 sometimes 1 and decision and activity handles also are not stable, it feels like due to load balancer the DNS keeps returning different IP and that's what causes this issue.
This issue resolves when I only have 1 cadence instance running. How do I resolve this issue and keep more than one cadence instances ?
I have enable global domain set as true and boot strap mode is dns, I have following as ringpop seeds: api-gtp-cadence.api-gtp-cadence:7933,0.0.0.0:7933,api-gtp-cadence.api-gtp-cadence:7934,0.0.0.0:7934,api-gtp-cadence.api-gtp-cadence:7935,0.0.0.0:7935,api-gtp-cadence.api-gtp-cadence:7939,0.0.0.0:7939
Beta Was this translation helpful? Give feedback.
All reactions