REDIS_URL for sentinel mode on k8s

[yuzhouliu9]

I've deployed Redis using the bitnami helm-chart in sentinel mode.

It deploys 3 k8s Service objects:

• redis - type ClusterIP with endpoints pointing to all replicas. Due to round robin routing to backend pods, this can be used as a read endpoint but not as a write endpoint (correct me if wrong)
• redis-headless - headless service with endpoints to all replicas. Can this be used for REDIS_URL, and how?
• redis-master- Service that points to the current master and is updated on failovers. This currently works with REDIS_URL=redis-master:6379

Using REDIS_TYPE=single and REDIS_URL=redis-master:6379 service works. But on failover, it takes a few seconds for it complete and for redis-master to point to newly elected master. During this time these errors appear in ratelimit logs:

caught error during call: failed to decode pipeline CmdAction 'INCRBY' with keys [ratelimit_authority_…]: EOF”

The docs suggest using REDIS_TYPE=sentinel with a comma delimited REDIS_URL=<master>,<hostname>:<port>,<hostname>:<port> but it's not clear to me how to populate that list.

What is the correct setup for REDIS_URL in sentinel mode?
Is the failover downtime of a few seconds expected?

[yuzhouliu9]

Found the answer, there are 2 options:
Use redis-headless with this config

        - name: REDIS_TYPE
          value: "sentinel"
        - name: REDIS_URL
          value: mymaster,ratelimit-redis-node-0.ratelimit-redis-headless.routing.svc.cluster.local:26379,ratelimit-redis-node-1.ratelimit-redis-headless.routing.svc.cluster.local:26379,ratelimit-redis-node-2.ratelimit-redis-headless.routing.svc.cluster.local:26379

Or use redis-master that redis will update during failover

        - name: REDIS_TYPE
          value: "single"
        - name: REDIS_URL
          value: ratelimit-redis-master:6379

Both result in the same failover recovery time of 4 seconds.
REDIS_TYPE=sentinel with headless does generate a lot more log noise with these repeated

time="2024-10-31T18:00:18Z" level=error msg="creating redis connection error : dial tcp 100.99.95.184:6379: connect: connection refused"

Leaving the ticket open if anyone can say which is the preferred option.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.