Updated Capture/Viewer Nodes to pull config files from Param Store

[chelma]

Description

• Began creating Capture/Viewer Config Files in-Python, uploading them to Parameter Store, and pulling them during container startup rather than embedding them into the container images

Tasks

• Enable Dynamic Container Configuration #76

Testing

• Ran create-cluster to update the containers and add-vpc to set up capture. Checked the container logs and the Viewer to ensure everything still worked.

Capture Startup Logs:

============================================================
Cluster: MyCluster3
Role: Capture Node
Arkime Config INI Path: /arkime/clusters/MyCluster3/capture-ini
Arkime Additional File Paths: ["/arkime/clusters/MyCluster3/capture-files/default.rules"]
AWS Region: us-east-2
Bucket Name: mycluster3-capturebucket-capturebucket66d28918-1x30yu8fzm6m3
LB Healthcheck Port: 4242
OpenSearch Endpoint: vpc-arkimedomain872-1nzuztrqm7dl-avy5u6ma7nfnxuye4vggd5uvaa.us-east-2.es.amazonaws.com
OpenSearch Secret Arn: arn:aws:secretsmanager:us-east-2:XXXXXXXXXXXX:secret:OpenSearchPassword9643DC3D-nRRSpQndpaAR-ECCqlS
S3 Storage Class: STANDARD
============================================================
Processing File in Param: /arkime/clusters/MyCluster3/capture-files/default.rules
File Name: default.rules
File Path Prefix: /opt/arkime/etc
Written to: /opt/arkime/etc/default.rules
Processing config.ini in Param: /arkime/clusters/MyCluster3/capture-ini
File Name: config.ini
File Path Prefix: /opt/arkime/etc
Written to: /opt/arkime/etc/config.ini
Configuring /opt/arkime/etc/config.ini ...
Successfully configured /opt/arkime/etc/config.ini
Testing connection/creds to OpenSearch domain vpc-arkimedomain872-1nzuztrqm7dl-avy5u6ma7nfnxuye4vggd5uvaa.us-east-2.es.amazonaws.com ...

Viewer Startup Logs:

============================================================
Cluster: MyCluster3
Role: Viewer Node
Arkime Config INI Path: /arkime/clusters/MyCluster3/viewer-ini
Arkime Additional File Paths: []
AWS Region: us-east-2
Bucket Name: mycluster3-capturebucket-capturebucket66d28918-1x30yu8fzm6m3
OpenSearch Endpoint: vpc-arkimedomain872-1nzuztrqm7dl-avy5u6ma7nfnxuye4vggd5uvaa.us-east-2.es.amazonaws.com
OpenSearch Secret Arn: arn:aws:secretsmanager:us-east-2:XXXXXXXXXXXX:secret:OpenSearchPassword9643DC3D-nRRSpQndpaAR-ECCqlS
Viewer Port: 8005
Viewer Password Secret Arn: arn:aws:secretsmanager:us-east-2:968674222892:secret:ViewerPasswordA5AFB5A1-51N3qqM57M2D-YOzI5v
Viewer User: admin
============================================================
Processing config.ini in Param: /arkime/clusters/MyCluster3/viewer-ini
File Name: config.ini
File Path Prefix: /opt/arkime/etc
Written to: /opt/arkime/etc/config.ini
Configuring /opt/arkime/etc/config.ini ...
Successfully configured /opt/arkime/etc/config.ini
Testing connection/creds to OpenSearch domain vpc-arkimedomain872-1nzuztrqm7dl-avy5u6ma7nfnxuye4vggd5uvaa.us-east-2.es.amazonaws.com ...

License

I confirm that this contribution is made under an Apache 2.0 license and that I have the authority necessary to make this contribution on behalf of its copyright owner.

[awick]

1. The variable names I think should be clarified if they are parameter store path or disk path
2. I would like to see if we could not have config be special and just have a list of all files. Then use a type key (or something) to say this is ini, do the seds, this is raw, etc.
3. capture/viewer should have -c $configfilepath without that they assume /opt/arkime/etc/config.ini
4. should ArkimeFile just have a full path instead of basepath and filename, also this might be the parameter store path

[chelma]

5. The Viewer's Run Script should reference $configfilepath instead of the hardcoded path value

[chelma]

I think I've addressed everything except (2), which I split into a separate task - #80

[awick]

LGTM