Metrics and Dashboard Panel Exporting & Loading

[breezewish]

Feature Request

Is your feature request related to a problem? Please describe:

Previously raised in #730, this feature request makes it more specific and feature-rich.

Describe the feature you'd like:

A. Grafana Exporting

• Support exporting Grafana dashboard panel data, which can later be transferred manually to PingCAP R&D or DBA.

• Provide an online UI to easily import these exported Grafana dashboard data.

Grafana exporting generally produce lightweight data. For example, exporting 1 week data for TiKV-Details panel takes about 10MB. However the exported data only contains current, calculated metrics, i.e. viewers cannot modify expressions, or view more subtle metrics.

This feature needs to support both current deployments and new deployments. For current deployments, a browser script and a short manual is provided to users, guiding them how to export through the browser DevTools. For new deployments, user only need to click a simple button in the UI of TiDB Dashboard to export data.

B. Prometheus Exporting

• Support dumping Prometheus raw data, which can later be transferred manually to PingCAP R&D or DBA.

• Provide a command-line utility to bootstrap a Prometheus instance (optionally bootstrap a Grafana server as well) from the dumped data, so that the dumped data can be easily viewed.

Prometheus exporting is very heavyweight compared to Grafana Exporting. It may produce >100MB or even >GB data for long term metrics. These dumped metrics may not be well compressed as well. On the other side, based on the exported data, viewers can view metrics that is not listed in Grafana panels, or easily modify expressions.

This feature needs to support both current deployments and new deployments. For current deployments, a CLI utility is provided to dump Prometheus TSDB data in a specific time range. For new deployments, user only need to click a simple button in the UI of TiDB Dashbaord to dump and transfer data back to the browser desktop.

Describe alternatives you've considered:

Implementation roadmaps (draft):

A. Grafana Exporting (for current deployment)

• A front-end script to be run in DevTools, which can export current Grafana Dashboard with data embedded to a JSON file: https://gist.github.com/breeswish/a8c2f766178853dfbc8335b9dfe2a3dd

• A website to introduce export instructions (like how to open DevTools), provide a button to copy front-end script by one-click

• Add import UI to website: drop JSON to the UI to restore Grafana dashboards (maybe powered by a modified version of Grafana front-end?)

• I18N

A. Grafana Exporting (for new deployment)

• TBD

B. Prometheus Exporting (for current deployment)

• A command line utility to export Prometheus TSDB data: prometheus-dumper

  • Prototype implementation via snapshot + direct compaction (compaction is used to filter out data outside time range).

  • Verify whether it works. If not, we may need to write tombstone flags first and then compact.

  • Check whether it's worth to compress the exported data.

  • Maybe contribute this feature to Prometheus official repository?

  • Maybe clean up the dump directory to not leave some garbage data when dump failed.

• A command line utility to load Prometheus TSDB data and present Grafana dashboard UI: tidb-metrics-viewer

  • Prototype implementation to start a Prometheus query API server over an existing dumped TSDB data

  • Verify Prometheus query API server works, supports all necessary interfaces required by Grafana

  • Start a Grafana (maybe at the same port) instance as well

    • A standalone Grafana server that can be compiled and run directly without the need to supply frontend assets: grafana-standalone

    • Currently this standalone Grafana server still requires a default config file to be exist in the working directory. Maybe we'd better remove the dependency.

    • Not tested whether Prometheus data source works. Maybe need some further works.

    • Integrate into tidb-metrics-viewer, start the Grafana server from tidb-metrics-viewer , where Prometheus API is also started

  • Accept a TiDB version parameter, fetch corresponding Grafana dashboard description files and import it automatically into the embedded Grafana at start up.

  • Maybe change Grafana config defaults to write its database (grafana.db) to a temporary directory, and clean up before server is stopped, to make this utility totally lightweight and green.

  • Provide the instructions to dump and load Prometheus data in the website.

B. Prometheus Exporting (for new deployment)

• TBD

[IANTHEREAL]

Accept Plan A. Plan B not in plan now

[kennytm]

/assign

[ti-srebot]

Assign failed.

[kennytm]

For A, with existing solutions around:

• https://github.com/alexrudd/snapshot_grafana — the most promising solution, unfortunately abandoned and doesn't work on Grafana 5+ according to grafana 5 broken alexrudd/snapshot_grafana#3
• https://github.com/ascendcorp/grafana-snapshot — a simple wrapper around https://github.com/ascendcorp/grafana-snapshot which does not fill in the snapshotData.
• https://github.com/pavanmt/grafana-dashboard-backup — more complex wrapper but ditto
• https://github.com/AlexJHayward/snapshotter — written in Scala but ditto
• https://github.com/Whyeasy/grafana-snapshot-helper — capable of extracting the local snapshots but that's it

There are also grafana/grafana#7539 and grafana/grafana#10401 for native support.

[breezewish]

This is now covered by the new collection tool.