Opensearch pod are not re-started after editing S3 secrets and all pods needed manual re-start:

[mohania17]

Describe the bug

We have observed that the opensearch pods are not restarted automatically when we change the s3 secrets; it needs manual restart

As per the below article; It says that Changes to the keystore contents are not automatically applied to the running Elasticsearch node. Re-reading settings requires a node restart.
https://www.elastic.co/guide/en/elasticsearch/reference/8.15/secure-settings.html#reloadable-secure-settings
https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html

So looks like it requires manual restarts only which is not acceptable; If opensearch requires manual re-start, then proper documentation is required.

Related component

Plugins

To Reproduce

1. Install opensearch
2. Deploy S3 secrets
3. Change secrets; the pods will not restart

Expected behavior

The pods must restart automatically once we update the S3 secrets

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[prudhvigodithi]

[Triage]
In Kubernetes Ideal way is to have the pods notice the change of secret/configMap and do a rolling restart or in this case since OpenSearch supports reload_secure_settings https://opensearch.org/docs/latest/api-reference/nodes-apis/nodes-reload-secure/, we have a job or use opensearchLifecycle to re-load the settings, @mohania17 can you please take a look and see if you can implement a solution ?
Thank you.

[mohania17]

Hi @prudhvigodithi
I tried to add the above setting reload_secure_settings in the dashboard it didnt worked
The job or opensearchLifecycle; I am unable to implement in opensearch installation via Opster(operator)