-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #109 from ccremer/stiebeleltron-stack
Add stiebeleltron-stack chart
- Loading branch information
Showing
47 changed files
with
2,834 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Patterns to ignore when building packages. | ||
| # This supports shell glob matching, relative path matching, and | ||
| # negation (prefixed with !). Only one pattern per line. | ||
| .DS_Store | ||
| # Common VCS dirs | ||
| .git/ | ||
| .gitignore | ||
| .bzr/ | ||
| .bzrignore | ||
| .hg/ | ||
| .hgignore | ||
| .svn/ | ||
| # Common backup files | ||
| *.swp | ||
| *.bak | ||
| *.tmp | ||
| *.orig | ||
| *~ | ||
| # Various IDEs | ||
| .project | ||
| .idea/ | ||
| *.tmproj | ||
| .vscode/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| dependencies: | ||
| - name: stiebeleltron-exporter | ||
| repository: https://ccremer.github.io/charts | ||
| version: 0.1.0 | ||
| - name: influxdb2 | ||
| repository: https://helm.influxdata.com | ||
| version: 2.0.4 | ||
| digest: sha256:46ad65dd0a4eca266c542dc6ccf4c9f3f7f73c34cb6d85e2888b9dbbf4500c4b | ||
| generated: "2021-12-12T16:23:18.747836615+01:00" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| apiVersion: v2 | ||
| name: stiebeleltron-stack | ||
| description: A Helm chart for installing stiebeleltron-exporter with long-term storage | ||
|
|
||
| # A chart can be either an 'application' or a 'library' chart. | ||
| # | ||
| # Application charts are a collection of templates that can be packaged into versioned archives | ||
| # to be deployed. | ||
| # | ||
| # Library charts provide useful utilities or functions for the chart developer. They're included as | ||
| # a dependency of application charts to inject those utilities and functions into the rendering | ||
| # pipeline. Library charts do not define any templates and therefore cannot be deployed. | ||
| type: application | ||
|
|
||
| # This is the chart version. This version number should be incremented each time you make changes | ||
| # to the chart and its templates, including the app version. | ||
| # Versions are expected to follow Semantic Versioning (https://semver.org/) | ||
| version: 0.1.0 | ||
|
|
||
| # This is the version number of the application being deployed. This version number should be | ||
| # incremented each time you make changes to the application. Versions are not expected to | ||
| # follow Semantic Versioning. They should reflect the version the application is using. | ||
| # It is recommended to use it with quotes. | ||
| #appVersion: "1.16.0" | ||
|
|
||
| sources: | ||
| - https://github.com/ccremer/stiebeleltron-exporter | ||
|
|
||
| dependencies: | ||
| - name: stiebeleltron-exporter | ||
| version: 0.1.1 | ||
| repository: https://ccremer.github.io/charts | ||
| alias: stiebeleltron | ||
| condition: stiebeleltron.enabled | ||
| - name: influxdb2 | ||
| alias: influxdb | ||
| version: 2.0.4 | ||
| repository: https://helm.influxdata.com | ||
| condition: influxdb.enabled |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| > ⚠️ **WARNING**: See [data retention](#data-retention) first! | ||
| ## About | ||
|
|
||
| This Helm chart installs the stiebeleltron-exporter along with long-term storage and a Grafana dashboard. | ||
| The goal is to be able to do statistics of the photovoltaic installation. | ||
|
|
||
| I wanted to keep stiebeleltron-exporter Helm chart minimal in case you're only interested in the exporter or you're bringing your own metrics stack. | ||
|
|
||
| ## Features | ||
|
|
||
| * stiebeleltron-exporter chart | ||
| * InfluxDB 2 chart | ||
| - Post-Install hook to configure archival bucket with downsampling | ||
| * Grafana datasource | ||
| * Grafana dashboard | ||
|
|
||
| ## Minimal required parameters | ||
|
|
||
| You must configure following parameters in the minimum: | ||
|
|
||
| ```yaml | ||
| influxdb: | ||
| adminUser: | ||
| token: changeme | ||
| password: changeme | ||
|
|
||
| stiebeleltron: | ||
| exporter: | ||
| isgUrl: http://your.isg.device.or.ip/ | ||
| telegraf: | ||
| influxdb: | ||
| token: changeme # should be the same as `influxdb.adminUser.token`! | ||
| globalTags: | ||
| site: my-home | ||
| ``` | ||
| ## Data retention | ||
| Before installing the chart, think about data retention. | ||
| By default, the chart installs InfluxDB 2.0 with an initial bucket that keeps metrics for 14 days. | ||
| Additionally, the chart features a one-time post-install hook to create an archival bucket that keeps data forever. | ||
| It also creates a tasks that periodically downsamples the high-precision data to 10-minute-averages into the archival bucket. | ||
| This bucket is meant to hold data for years. | ||
| You can adjust retention with the following parameters: | ||
| ```console | ||
| influxdb.adminUser.retention_policy | ||
| archival.* | ||
| ``` | ||
|
|
||
| > ℹ️ **NOTE**: After installation, any changes in bucket namings, retention or downsampling settings will NOT be applied anymore. | ||
| > You'd need to manually change this with InfluxDB CLI within the pod. | ||
| ## Grafana integration | ||
|
|
||
| The chart comes with a Grafana datasource and dashboard ConfigMaps enabled by default. | ||
| Please ensure Grafana sidecar is enabled and it searches in the release namespace, or set `grafana.*.namespace`. | ||
|
|
||
| The dashboard features some panels that calculate per-day metrics. | ||
| Make sure the stiebeleltron ISG device, InfluxDB and Grafana dashboard are roughly in the same timezone. | ||
| The stiebeleltron ISG console includes counters like "energy per day" that reset at midnight. | ||
|
|
||
| ## Migrating from netdata plugin | ||
|
|
||
| If you have used the stiebeleltron plugin that is part of netdata, you can migrate the data to the exporter variant. | ||
|
|
||
| As this step is done manually once, it's not automated, but here's roughly how I did it: | ||
|
|
||
| 1. Install this chart but don't start the exporter and influxdb yet (0 replicas each). | ||
| Also disable the archival feature for now. | ||
| 2. Modify the InfluxDB 2.x StatefulSet, so that you can migrate using the Docker approach. | ||
| You'd need to set `DOCKER_INFLUXDB_INIT_MODE=upgrade` and add the volume mounts from the old DB. | ||
| Set `INFLUXD_ENGINE_PATH=/var/lib/influxdb2/migration` (we appended `migration`, otherwise there will be some permission errors). | ||
| Set `INFLUXD_BOLT_PATH=/var/lib/influxdb2/migration/influxd.bolt`. | ||
| 3. You can now scale the replicas to 1. | ||
| 4. Once the InfluxDB 2.x is running and the old DB upgraded, enter its shell and execute the following: | ||
| ``` | ||
| influx backup /var/lib/influxdb2/backup | ||
| ``` | ||
| 5. Exit shell and scale replica to 0. | ||
| 6. Reset `DOCKER_INFLUXDB_INIT_MODE`, `INFLUXD_ENGINE_PATH` and `INFLUXD_BOLT_PATH` to default values and set `replicas=1`. | ||
| 7. Enter shell again and execute a restore: | ||
| ``` | ||
| influx bucket delete --name stiebeleltron_live | ||
| influx restore /var/lib/influxdb2/backup | ||
| influx bucket create --name stiebeleltron_archive --org stiebeleltron | ||
| rm -rf /var/lib/influxdb2/migration /var/lib/influxdb2/backup | ||
| ``` | ||
| 8. You should have now `netdata/autogen`, `stiebeleltron_live` and `stiebeleltron_archive` bucket. | ||
| The last 2 should be empty for now. | ||
| 9. Migrate & downsample the raw data from `netdata/autogen` bucket using the [scripts provided](files/influxdb/migration). | ||
| You might need to edit the query a bit if you are deviating from the defaults. | ||
| You can execute the scripts with `kubectl`, e.g. | ||
| ``` | ||
| cat migration/<the-file>.flux | kubectl -n stiebeleltron exec -i stiebeleltron-influxdb-0 -- influx query -o stiebeleltron | ||
| ``` | ||
| 10. Each migration script can take a few minutes. | ||
| Check the Grafana dashboard if the values make sense. | ||
| 11. Scale up the exporter and verify that values are stored in `stiebeleltron_live` bucket. | ||
| 12. Create the downsampling task for `stiebeleltron_archive`, check out the ConfigMap in the Helm chart that would installed on first installation. | ||
| 13. Delete the `netdata/autogen` bucket via CLI. | ||
|
|
||
| <!--- | ||
| Common/Useful Link references from values.yaml | ||
| --> | ||
| [resource-units]: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes | ||
| [prometheus-operator]: https://github.com/coreos/prometheus-operator | ||
| [prom-relabel-config]: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#relabelconfig |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,167 @@ | ||
| # stiebeleltron-stack | ||
|
|
||
|   | ||
|
|
||
| A Helm chart for installing stiebeleltron-exporter with long-term storage | ||
|
|
||
| ## Installation | ||
|
|
||
| ```bash | ||
| helm repo add ccremer https://ccremer.github.io/charts | ||
| helm install stiebeleltron-stack ccremer/stiebeleltron-stack | ||
| ``` | ||
| > ⚠️ **WARNING**: See [data retention](#data-retention) first! | ||
| ## About | ||
|
|
||
| This Helm chart installs the stiebeleltron-exporter along with long-term storage and a Grafana dashboard. | ||
| The goal is to be able to do statistics of the photovoltaic installation. | ||
|
|
||
| I wanted to keep stiebeleltron-exporter Helm chart minimal in case you're only interested in the exporter or you're bringing your own metrics stack. | ||
|
|
||
| ## Features | ||
|
|
||
| * stiebeleltron-exporter chart | ||
| * InfluxDB 2 chart | ||
| - Post-Install hook to configure archival bucket with downsampling | ||
| * Grafana datasource | ||
| * Grafana dashboard | ||
|
|
||
| ## Minimal required parameters | ||
|
|
||
| You must configure following parameters in the minimum: | ||
|
|
||
| ```yaml | ||
| influxdb: | ||
| adminUser: | ||
| token: changeme | ||
| password: changeme | ||
|
|
||
| stiebeleltron: | ||
| exporter: | ||
| isgUrl: http://your.isg.device.or.ip/ | ||
| telegraf: | ||
| influxdb: | ||
| token: changeme # should be the same as `influxdb.adminUser.token`! | ||
| globalTags: | ||
| site: my-home | ||
| ``` | ||
| ## Data retention | ||
| Before installing the chart, think about data retention. | ||
| By default, the chart installs InfluxDB 2.0 with an initial bucket that keeps metrics for 14 days. | ||
| Additionally, the chart features a one-time post-install hook to create an archival bucket that keeps data forever. | ||
| It also creates a tasks that periodically downsamples the high-precision data to 10-minute-averages into the archival bucket. | ||
| This bucket is meant to hold data for years. | ||
| You can adjust retention with the following parameters: | ||
| ```console | ||
| influxdb.adminUser.retention_policy | ||
| archival.* | ||
| ``` | ||
|
|
||
| > ℹ️ **NOTE**: After installation, any changes in bucket namings, retention or downsampling settings will NOT be applied anymore. | ||
| > You'd need to manually change this with InfluxDB CLI within the pod. | ||
| ## Grafana integration | ||
|
|
||
| The chart comes with a Grafana datasource and dashboard ConfigMaps enabled by default. | ||
| Please ensure Grafana sidecar is enabled and it searches in the release namespace, or set `grafana.*.namespace`. | ||
|
|
||
| The dashboard features some panels that calculate per-day metrics. | ||
| Make sure the stiebeleltron ISG device, InfluxDB and Grafana dashboard are roughly in the same timezone. | ||
| The stiebeleltron ISG console includes counters like "energy per day" that reset at midnight. | ||
|
|
||
| ## Migrating from netdata plugin | ||
|
|
||
| If you have used the stiebeleltron plugin that is part of netdata, you can migrate the data to the exporter variant. | ||
|
|
||
| As this step is done manually once, it's not automated, but here's roughly how I did it: | ||
|
|
||
| 1. Install this chart but don't start the exporter and influxdb yet (0 replicas each). | ||
| Also disable the archival feature for now. | ||
| 2. Modify the InfluxDB 2.x StatefulSet, so that you can migrate using the Docker approach. | ||
| You'd need to set `DOCKER_INFLUXDB_INIT_MODE=upgrade` and add the volume mounts from the old DB. | ||
| Set `INFLUXD_ENGINE_PATH=/var/lib/influxdb2/migration` (we appended `migration`, otherwise there will be some permission errors). | ||
| Set `INFLUXD_BOLT_PATH=/var/lib/influxdb2/migration/influxd.bolt`. | ||
| 3. You can now scale the replicas to 1. | ||
| 4. Once the InfluxDB 2.x is running and the old DB upgraded, enter its shell and execute the following: | ||
| ``` | ||
| influx backup /var/lib/influxdb2/backup | ||
| ``` | ||
| 5. Exit shell and scale replica to 0. | ||
| 6. Reset `DOCKER_INFLUXDB_INIT_MODE`, `INFLUXD_ENGINE_PATH` and `INFLUXD_BOLT_PATH` to default values and set `replicas=1`. | ||
| 7. Enter shell again and execute a restore: | ||
| ``` | ||
| influx bucket delete --name stiebeleltron_live | ||
| influx restore /var/lib/influxdb2/backup | ||
| influx bucket create --name stiebeleltron_archive --org stiebeleltron | ||
| rm -rf /var/lib/influxdb2/migration /var/lib/influxdb2/backup | ||
| ``` | ||
| 8. You should have now `netdata/autogen`, `stiebeleltron_live` and `stiebeleltron_archive` bucket. | ||
| The last 2 should be empty for now. | ||
| 9. Migrate & downsample the raw data from `netdata/autogen` bucket using the [scripts provided](files/influxdb/migration). | ||
| You might need to edit the query a bit if you are deviating from the defaults. | ||
| You can execute the scripts with `kubectl`, e.g. | ||
| ``` | ||
| cat migration/<the-file>.flux | kubectl -n stiebeleltron exec -i stiebeleltron-influxdb-0 -- influx query -o stiebeleltron | ||
| ``` | ||
| 10. Each migration script can take a few minutes. | ||
| Check the Grafana dashboard if the values make sense. | ||
| 11. Scale up the exporter and verify that values are stored in `stiebeleltron_live` bucket. | ||
| 12. Create the downsampling task for `stiebeleltron_archive`, check out the ConfigMap in the Helm chart that would installed on first installation. | ||
| 13. Delete the `netdata/autogen` bucket via CLI. | ||
|
|
||
| <!--- | ||
| Common/Useful Link references from values.yaml | ||
| --> | ||
| [resource-units]: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes | ||
| [prometheus-operator]: https://github.com/coreos/prometheus-operator | ||
| [prom-relabel-config]: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#relabelconfig | ||
|
|
||
| ## Source Code | ||
|
|
||
| * <https://github.com/ccremer/stiebeleltron-exporter> | ||
|
|
||
| ## Requirements | ||
|
|
||
| | Repository | Name | Version | | ||
| |------------|------|---------| | ||
| | https://ccremer.github.io/charts | stiebeleltron(stiebeleltron-exporter) | 0.1.1 | | ||
| | https://helm.influxdata.com | influxdb(influxdb2) | 2.0.4 | | ||
|
|
||
| ## Values | ||
|
|
||
| | Key | Type | Default | Description | | ||
| |-----|------|---------|-------------| | ||
| | archival.bucket | string | `"stiebeleltron_archive"` | Name of the archival bucket to create after installation. | | ||
| | archival.enabled | bool | `true` | Whether long-term archival is enabled. Note: Disabling archival after installation (when enabled) does not remove the archival bucket. | | ||
| | archival.retention | string | `"0s"` | Retention of the archival bucket. `0s` means forever. | | ||
| | archival.window | string | `"10m"` | Fixed windows of time in which metrics are averaged. | | ||
| | fullnameOverride | string | `""` | | | ||
| | grafana.dashboard.enabled | bool | `true` | | | ||
| | grafana.dashboard.labels | object | `{"grafana_dashboard":"1"}` | The labels which the sidecar is filtering for dashboards. | | ||
| | grafana.dashboard.namespace | string | `""` | Override the namespace where the ConfigMap is installed, defaults to release namespace. | | ||
| | grafana.datasource.enabled | bool | `true` | | | ||
| | grafana.datasource.labels | object | `{"grafana_datasource":"1"}` | The labels which the sidecar is filtering for data sources. | | ||
| | grafana.datasource.namespace | string | `""` | Override the namespace where the ConfigMap is installed, defaults to release namespace. | | ||
| | influxdb.adminUser.bucket | string | `"stiebeleltron_live"` | | | ||
| | influxdb.adminUser.organization | string | `"stiebeleltron"` | | | ||
| | influxdb.adminUser.retention_policy | string | `"14d"` | | | ||
| | influxdb.enabled | bool | `true` | | | ||
| | influxdb.pdb.create | bool | `false` | | | ||
| | influxdb.persistence.size | string | `"2Gi"` | | | ||
| | influxdb.resources.limits.memory | string | `"256Mi"` | | | ||
| | influxdb.resources.requests.cpu | string | `"50m"` | | | ||
| | influxdb.resources.requests.memory | string | `"128Mi"` | | | ||
| | nameOverride | string | `"stiebeleltron"` | | | ||
| | stiebeleltron.enabled | bool | `true` | | | ||
| | stiebeleltron.nameOverride | string | `"exporter"` | | | ||
| | stiebeleltron.telegraf.enabled | bool | `true` | | | ||
| | stiebeleltron.telegraf.globalTags.site | string | `"home"` | The name of the site or environment. | | ||
| | stiebeleltron.telegraf.influxdb.bucket | string | `"stiebeleltron_live"` | The high-precision bucket name, needs to be equal to `influxdb.adminUser.bucket`. | | ||
| | stiebeleltron.telegraf.influxdb.token | string | `""` | The token to connect to InfluxDB, needs to be equal to `influxdb.adminUser.token`. | | ||
| | stiebeleltron.telegraf.influxdb.url | string | `"http://stiebeleltron-influxdb"` | | | ||
| | stiebeleltron.telegraf.interval | string | `"5s"` | Interval of sending metrics to InfluxDB. | |
Oops, something went wrong.