Cadence cluster networking configuration #4356

longquanzheng · 2021-08-13T17:33:03Z

longquanzheng
Aug 13, 2021

NOTE: Please leave your questions below and we can improve on this topic

Here uses configuration in 0.21.0 as example. But the other versions should be the same.

Single server cluster without replication

Ringpop configuration

Ringpop is to connect all the nodes within the same cluster.

Usually bootstrapMode should be DNS, see example. Using DNS is recommended because if any of the IP addresses of bootstrap is not up, hosts can't join the cluster. This could be a problem for IP changes or restarting or scaling up.
bootstrapHosts can be DNS of any service, or even all of them like the example.
If you use IP mode, make sure that the host with the IP is always up.
Ringpop requires all bootstrap IPs to be up during startup(within maxJoinDuration) This could be a chicken-egg problem. To break this circle the IPs needs to be published even the nodes are not ready. In Kubernetes, this is achieved by publishNotReadyAddresses
We probably can remove the restriction in the future, but that could have some risk of network partition issue.

To verify the DNS, you should be able to use nslookup command to see all the IPs behind it.

publicClient configuration

It's for server worker service to connect to frontend.

HostPort should be the DNS address of the frontend service.
To verify the DNS, you should be able to use nslookup command to see all the IPs behind it.
If using Kubernetes, ClusterIP should work as a DNS, see helmchart values and template
See related discussion
Alternatively, you can use LoadBalancer. This would allow your client workers to access frontend service from outside of K8s.

`clusterMetadata.clusterInformation.<currentClusterName>.rpcAddress` configuration

It must be exactly the same as publicClient.hostPort
This is duplicated, and will be improved in Unify publicClient with clusterMetadata.clusterInformation.<currentClusterName>.rpcAddress configuration #3984

Service.RPC configuration

service.rpc is to configure the service to listen on certain IP and port.

Only bindOnIP should be used in a cluster. bindOnLocalHost is only for local testing server
The IP address will be used for not only accepting requests, but also discovery address in Ringpop. See how Ringpop does the node discovery.
Therefore, the IP needs to be the IP of the instance. In K8s setup, it's a PodID.

Multiple Server clusters with XDC(global domain) replication group

XDC(global domain) is a replication group of multiple cadence clusters.
Based on the "Single server cluster without replication", clusterMetadata needs to be updated to configure multiple clusters to connect to each other.

Supposed we have two clusters, clusterA and clusterB

Config of clusterA should be:

clusterMetadata:
  enableGlobalDomain: true
  failoverVersionIncrement: 10
  masterClusterName: "clusterA" 
  currentClusterName: "clusterA"
  clusterInformation:
    clusterA:
      enabled: true
      initialFailoverVersion: 1
      rpcName: "cadence-frontend"
      rpcAddress: "<frontend DNS of clusterA>:7933"
    clusterB:
      enabled: true
      initialFailoverVersion: 0
      rpcName: "cadence-frontend"
      rpcAddress: "<frontend DNS of clusterB>:7933"

Config of clusterB should be:

clusterMetadata:
  enableGlobalDomain: true
  failoverVersionIncrement: 10
  masterClusterName: "clusterA" 
  currentClusterName: "clusterB"
  clusterInformation:
    clusterA:
      enabled: true
      initialFailoverVersion: 1
      rpcName: "cadence-frontend"
      rpcAddress: "<frontend DNS of clusterA>:7933"
    clusterB:
      enabled: true
      initialFailoverVersion: 0
      rpcName: "cadence-frontend"
      rpcAddress: "<frontend DNS of clusterB>:7933"

Some rules of the config:

currentClusterName must the different for each cluster. This means which this deployment IS.
masterClusterName should be the same in both cluster. This decides which cluster can register new domains and update domain information.
Failover can be made from any cluster. It's NOT decided by masterClusterName
failoverVersionIncrement decides the maximum number clusters in this replication groups
initialFailoverVersion is the identifier of each cluster. 0<= the value < failoverVersionIncrement
the value of initialFailoverVersion and failoverVersionIncrement should be exactly the same across all clusters.
rpcAddress is for two clusters talking to each other. If the clusters are in a different Kubernetes clusters, you may have to use LoadBalancer because ClusterIP will not work across two K8s.

Client worker

Client worker is the worker that is running your workflow and activity code.

Recommended to use DNS to connect to frontend service

Related discussions

#4143
#3150
#3054

banzaicloud/banzai-charts#1278

ceearrashee · 2021-08-14T07:52:19Z

ceearrashee
Aug 14, 2021

Hello to continue of discussion:
You ask me about domain lookup:

/paser-staging-cadence-frontend/~$: nslookup paser-staging-cadence-frontend
Server:         10.152.183.10
Address:        10.152.183.10:53

** server can't find paser-staging-cadence-frontend.cluster.local: NXDOMAIN

** server can't find paser-staging-cadence-frontend.svc.cluster.local: NXDOMAIN

** server can't find paser-staging-cadence-frontend.cluster.local: NXDOMAIN

*** Can't find paser-staging-cadence-frontend.paser-staging.svc.cluster.local: No answer

** server can't find paser-staging-cadence-frontend.svc.cluster.local: NXDOMAIN

Name:   paser-staging-cadence-frontend.paser-staging.svc.cluster.local
Address: 10.152.183.33

Domain name resolved properly because domain description works.

/paser-staging-cadence-frontend/~$: cadence d desc
Name: paser
UUID: dcc71fdb-44eb-41f7-940d-96df1c1e59d3
Description: 
OwnerEmail: 
DomainData: map[]
Status: REGISTERED
RetentionInDays: 365
EmitMetrics: false
ActiveClusterName: active
Clusters: active
HistoryArchivalStatus: DISABLED
VisibilityArchivalStatus: DISABLED
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+

Other services also work properly. A problem occurs only while I try to update the domain (in other cases, not tested). At the beginning (around two years ago), we created a cluster with retention days 365, but now the database is too big, and our team decided to decrease retention days to 30 as a maximal value for new clusters.

5 replies

longquanzheng Aug 14, 2021
Author

What is the error you run into when update the retention?

Do you use helm chart and updated recently? There is a bug and they just released a bug fix. banzaicloud/banzai-charts#1275 (comment)

Basically the chart changed the cluster name while the domain still use the older names.

ceearrashee Aug 16, 2021

Hello! I started the problem discussed in the slack thread: https://uber-cadence.slack.com/archives/CL22WDF70/p1627974339017100

Yes, I used the latest chart, 0.20.3, but an error is still present. Actual docker.yaml after cluster update:

log:
  stdout: true
  level: "debug,info"
  levelKey: "level"

persistence:
  defaultStore: default
  visibilityStore: visibility
  numHistoryShards: 512
  datastores:
    default:
      sql:
        pluginName: mysql
        databaseName: cadence
        connectAddr: "paser-staging-cadence-mysql.paser-staging.svc.cluster.local:3306"
        connectProtocol: "tcp"
        user: cadence
        password: cadence
    visibility:
      sql:
        pluginName: mysql
        databaseName: cadence_visibility
        connectAddr: "paser-staging-cadence-mysql.paser-staging.svc.cluster.local:3306"
        connectProtocol: "tcp"
        user: cadence
        password: cadence

ringpop:
  name: cadence
  bootstrapMode: dns
  bootstrapHosts:
    - paser-staging-cadence-frontend-headless:7833
    - paser-staging-cadence-frontend-headless:7933
    - paser-staging-cadence-history-headless:7834
    - paser-staging-cadence-history-headless:7934
    - paser-staging-cadence-matching-headless:7835
    - paser-staging-cadence-matching-headless:7935
    - paser-staging-cadence-worker-headless:7939
  maxJoinDuration: 30s

services:
  frontend:
    rpc:
      grpcPort: 7833
      port: 7933
      bindOnIP: 10.1.207.112
    metrics:
      tags:
        type: frontend
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

  history:
    rpc:
      grpcPort: 7834
      port: 7934
      bindOnIP: 10.1.207.112
    metrics:
      tags:
        type: history
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

  matching:
    rpc:
      grpcPort: 7835
      port: 7935
      bindOnIP: 10.1.207.112
    metrics:
      tags:
        type: matching
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

  worker:
    rpc:
      port: 7939
      bindOnIP: 10.1.207.112
    metrics:
      tags:
        type: worker
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

clusterMetadata:
  enableGlobalDomain: false
  failoverVersionIncrement: 10
  masterClusterName: "primary"
  currentClusterName: "primary"
  clusterInformation:
    primary:
      enabled: true
      initialFailoverVersion: 0
      rpcName: "cadence-frontend"
      rpcAddress: cadence:7933

dcRedirectionPolicy:
  policy: selected-apis-forwarding
  toDC: ""

archival:
  status: "disabled"

publicClient:
  hostPort: "paser-staging-cadence-frontend:7933"

dynamicConfigClient:
  filepath: "/etc/cadence/config/dynamicconfig/config.yaml"
  pollInterval: "10s"

An error occurred while I tried to update retention days:

/etc/cadence $ cadence --do paser --address paser-staging-cadence-frontend:7933 
d up --rd 30
Error: Operation UpdateDomain failed.
Error Details: BadRequestError{Message: Invalid cluster name: active}
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack(0xd, 0x0, 0x0)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/uber/cadence/tools/cli.printError(0x2502edd, 0x1e, 0x2a61660, 0xc000312330)
        /cadence/tools/cli/util.go:536 +0x2ad
github.com/uber/cadence/tools/cli.ErrorAndExit(0x2502edd, 0x1e, 0x2a61660, 0xc000312330)
        /cadence/tools/cli/util.go:547 +0x49
github.com/uber/cadence/tools/cli.(*domainCLIImpl).UpdateDomain(0xc00030ce80, 0xc000102840)
        /cadence/tools/cli/domainCommands.go:276 +0x457
github.com/uber/cadence/tools/cli.newDomainCommands.func2(0xc000102840)
        /cadence/tools/cli/domain.go:82 +0x48
github.com/urfave/cli.HandleAction(0x203b6e0, 0x25ac418, 0xc000102840, 0xc000102840, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:528 +0x127
github.com/urfave/cli.Command.Run(0x24d0694, 0x6, 0x0, 0x0, 0xc000433080, 0x2, 0x2, 0x2505f3c, 0x1f, 0x0, ...)
        /go/pkg/mod/github.com/urfave/[email protected]/command.go:174 +0x528
github.com/urfave/cli.(*App).RunAsSubcommand(0xc00036f500, 0xc000102160, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:407 +0x882
github.com/urfave/cli.Command.startApp(0x24cfb54, 0x6, 0x0, 0x0, 0xc000107ac0, 0x1, 0x1, 0x24ee32c, 0x16, 0x0, ...)
        /go/pkg/mod/github.com/urfave/[email protected]/command.go:373 +0x845
github.com/urfave/cli.Command.Run(0x24cfb54, 0x6, 0x0, 0x0, 0xc000107ac0, 0x1, 0x1, 0x24ee32c, 0x16, 0x0, ...)
        /go/pkg/mod/github.com/urfave/[email protected]/command.go:102 +0x9d6
github.com/urfave/cli.(*App).Run(0xc00036f340, 0xc00003a090, 0x9, 0x9, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:279 +0x731
main.main()
        /cadence/cmd/tools/cli/main.go:37 +0x4f

longquanzheng Aug 16, 2021
Author

The clusterMeta section needs to be

clusterMetadata:
  enableGlobalDomain: true
  failoverVersionIncrement: 10
  masterClusterName: "active"
  currentClusterName: "active"
  clusterInformation:
    active:
      enabled: true
      initialFailoverVersion: 0
      rpcName: "cadence-frontend"
      rpcAddress: cadence:7933

so that you can continue to update the domain

I’m not very sure how to make that happen in the chart . One easy way is just hardcoded and value in the template in template/server-configmap.yaml

longquanzheng Aug 18, 2021
Author

I just found the the latest fix is still having problem: https://github.com/banzaicloud/banzai-charts/pull/1282/files

It's hardcoded to "primary" as the name which is not compatible with the existing deployment. Like I said in banzaicloud/banzai-charts#1274

@pregnor is working on banzaicloud/banzai-charts#1278 which would allow passing in any cluster name.

@ceearrashee for now, I think you should hardcode the value back to "active" like previous https://github.com/banzaicloud/banzai-charts/blob/24278f1aa6d09d59082e40f2c0a61d786627abe1/cadence/templates/server-configmap.yaml#L143

pregnor Aug 18, 2021

@ceearrashee

We have just released the chart version 0.21.0 where you are able to configure the chart according to your existing Cadence clusters.

You are going to need to configure the chart similarly to this example, because the active cluster name indicates that your Cadence cluster was created from chart 0.16 or below.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cadence cluster networking configuration #4356

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Cadence cluster networking configuration #4356

longquanzheng Aug 13, 2021

Single server cluster without replication

Ringpop configuration

publicClient configuration

clusterMetadata.clusterInformation.<currentClusterName>.rpcAddress configuration

Service.RPC configuration

Multiple Server clusters with XDC(global domain) replication group

Client worker

Related discussions

Replies: 1 comment · 5 replies

ceearrashee Aug 14, 2021

longquanzheng Aug 14, 2021 Author

ceearrashee Aug 16, 2021

longquanzheng Aug 16, 2021 Author

longquanzheng Aug 18, 2021 Author

pregnor Aug 18, 2021

longquanzheng
Aug 13, 2021

`clusterMetadata.clusterInformation.<currentClusterName>.rpcAddress` configuration

Replies: 1 comment 5 replies

ceearrashee
Aug 14, 2021

longquanzheng Aug 14, 2021
Author

longquanzheng Aug 16, 2021
Author

longquanzheng Aug 18, 2021
Author