404 error connecting to socket channel on K8S cluster

[iasbm]

hello we used this HELM Chart to deploy our RASA BOT image with credentials.yaml configured for socket channel.
Of course we are able to get it all working before K8S deployment

Now we deployed the bot in K8S cluster using this helm chart and the bot is responding with "Hello from Rasa: 3.3.1" when we visit https://my-bot.domain.com.. But we are getting the below error when we are connecting to socket.io channel.

https://my-bot.domain.com./socket.io/?EIO=4&transport=polling&t=OUz5vHV 404
Error: xhr poll error
    at n.value (index.js:99:957)
    at n.<anonymous> (index.js:193:227277)
    at r.emit (index.js:84:263874)
    at n.value (index.js:193:225378)
    at index.js:193:224880

we followed the configuration here and tried below combinations and still the same above error

We didn't deploy RasaX /Enterprise HELM chart. SO its just RASA Opensource and Action server Helm Charts we deployed.

1. set applicationSettings.credentials=false ( the idea here is to let the RASA use the credentials.yaml from our image)
2. Then changed the rasa-values.yaml as follows...

credentials:
      # -- Enable credentials configuration for channel connectors
      enabled: true

      # -- Additional channel credentials which should be used by Rasa to connect to various
      # input channels
      ## See: https://rasa.com/docs/rasa/messaging-and-voice-channels
      additionalChannelCredentials: {}
      #rest:
      #  facebook:
      #    verify: "rasa"
      #    secret: "<SECRET>"
      #    page-access-token: "<PAGE-ACCESS-TOKEN>"
   
      socketio:
        user_message_evt: user_uttered 
        bot_message_evt: bot_uttered 
        session_persistence: true

can you help where / what we are missing