[KYUUBI #5834] Add Grafana dashboard template #5147

zhaohehuhu · 2023-08-09T08:31:27Z

Why are the changes needed?

This PR adds a basic Grafana Dashboard template, also updates the metrics docs to guide users to use Prometheus and Grafana to monitor the Kyuubi server.

The Grafana Dashboard template is exported from the Grafana OSS v11.4.0

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate

Run test locally before make a pull request

zhaohehuhu · 2023-08-09T08:37:07Z

plz help review @pan3793 @zwangsheng . Thanks all.😄

codecov-commenter · 2023-08-09T09:48:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (b265ccb) to head (f6fc2d7).
Report is 2 commits behind head on master.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #5147   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         687     687           
  Lines       42463   42463           
  Branches     5796    5796           
======================================
  Misses      42463   42463

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bowenliang123 · 2023-08-09T13:25:03Z

Could you append a full screenshot to the desciption for demonstrating the dashboard?
And any further detailed explanation for datasource settings, metric group and etc.

zwangsheng · 2023-08-10T01:41:59Z

We should set max as 100.
Don't know why threshold is needed here, let's remove those operation-num thresholds.

zhaohehuhu · 2023-08-10T02:00:17Z

Could you append a full screenshot to the desciption for demonstrating the dashboard? And any further detailed explanation for datasource settings, metric group and etc.

Added screenshots. Thanks.

zhaohehuhu · 2023-08-10T02:01:54Z

Yup. We can remove the threshold if it's not required. @zwangsheng

zhaohehuhu · 2023-09-05T09:15:05Z

plz review it @pan3793

pan3793 · 2023-09-22T07:47:22Z

grafana/Dockerfile

+#   under the grafana folder.
+#   1. docker build --build-arg PROMETHEUS_URL_ARG="" -t grafana:kyuubi -f Dockerfile .
+#   Options:
+#     --build-arg the url to access promethues datasource


what's the value should be like? could you please add an example in the description?

pan3793 · 2023-09-22T07:49:04Z

grafana/Dockerfile

+# limitations under the License.
+#
+
+# Usage:


please create a REAMDE.md in the same folder, and move this guidance to README.md

appreciate it. I already provided REAMDE.md to guide users.

pan3793 · 2023-09-22T07:51:20Z

Currently, the default value of kyuubi.metrics.reporters is JSON, as PROMETHEUS becomes more popular today, how do you think changing it to PROMETHEUS in 1.8?

@yaooqinn @bowenliang123

zwangsheng · 2023-10-25T07:15:10Z

grafana/Dockerfile

+ENV PROMETHEUS_URL=$PROMETHEUS_URL_ARG
+COPY ./dashboard.yml /etc/grafana/provisioning/dashboards/
+COPY ./datasource.yml  /etc/grafana/provisioning/datasources/
+COPY ./*.json /var/lib/grafana/dashboards/


We should add empty line at end of file.

zwangsheng · 2023-10-25T07:15:25Z

grafana/dashboard.yml

+    editable: true
+    allowUiUpdates: true
+    options:
+      path: /var/lib/grafana/dashboards/


zwangsheng · 2023-10-25T07:15:29Z

grafana/datasource.yml

+    access: proxy
+    url: ${PROMETHEUS_URL}
+    isDefault: true
+    editable: false


zhaohehuhu · 2024-01-05T07:30:36Z

@pan3793 plz help review it

yaooqinn · 2024-01-05T07:37:39Z

Currently, the default value of kyuubi.metrics.reporters is JSON, as PROMETHEUS becomes more popular today, how do you think changing it to PROMETHEUS in 1.8?

@yaooqinn @bowenliang123

I'm OK with this, but it seems that we'd better to raise a discussion thread in dev

bowenliang123 · 2024-01-05T07:42:51Z

Currently, the default value of kyuubi.metrics.reporters is JSON, as PROMETHEUS becomes more popular today, how do you think changing it to PROMETHEUS in 1.8?
@yaooqinn @bowenliang123

I'm OK with this, but it seems that we'd better to raise a discussion thread in dev

The default metrics reporter has been set to Prometheus in #5344.

yaooqinn · 2024-01-05T07:47:34Z

Thank you @bowenliang123

sudohainguyen · 2024-06-22T13:57:39Z

hey folks, any blockers on this 🤔 ?

zhaohehuhu · 2024-07-04T02:53:13Z

hey folks, any blockers on this 🤔 ?

It still needs some time to be reviewed.

hakeedra · 2024-11-12T07:28:20Z

Hi, @zhaohehuhu some metrics do not include instance filtering, such as the expression in 'Thread Status'. When testing with Kyuubi version '1.9.1', metric names may depending on the Garbage Collector. For example, 'kyuubi_memory_usage_pools_PS_Eden_Space_used' may appear as 'kyuubi_memory_usage_pools_Eden_Space_used'.

zhaohehuhu · 2024-11-16T04:17:41Z

Hi, @zhaohehuhu some metrics do not include instance filtering, such as the expression in 'Thread Status'. When testing with Kyuubi version '1.9.1', metric names may depending on the Garbage Collector. For example, 'kyuubi_memory_usage_pools_PS_Eden_Space_used' may appear as 'kyuubi_memory_usage_pools_Eden_Space_used'.

Thanks. I'll update it.

pan3793

I'd like to merge this PR and polish the dashboard in follow-ups.

pan3793 · 2024-12-24T02:30:23Z

Thanks, merging to master

pan3793 added this to the v1.8.0 milestone Sep 1, 2023

pan3793 assigned zhaohehuhu Sep 1, 2023

pan3793 reviewed Sep 22, 2023

View reviewed changes

zwangsheng reviewed Oct 25, 2023

View reviewed changes

pan3793 modified the milestones: v1.8.0, v1.8.1 Nov 6, 2023

pan3793 mentioned this pull request Dec 8, 2023

[TASK][EASY] Add Grafana dashboard #5834

Closed

3 tasks

pan3793 changed the title ~~[Improvement]Add Grafana dashboard template~~ [KYUUBI #5834] Add Grafana dashboard template Dec 8, 2023

zhaohehuhu added 5 commits December 23, 2024 14:23

Add Grafana dashboard template

dbc2210

add docker file build a grafana image and load dashboards available

45e3ba3

add REAMDE.md to guide users

457085b

make it compatible with kyuubi 1.8

185f2cc

add status chart

4b2bd3d

zhaohehuhu and others added 2 commits December 23, 2024 14:23

add status chart

3fa2d23

update dashboard

465f054

pan3793 force-pushed the Improvement-0809 branch from d7f1553 to 465f054 Compare December 23, 2024 20:19

github-actions bot added module:kubernetes kind:documentation Documentation is a feature! kind:build labels Dec 23, 2024

pan3793 modified the milestones: v1.8.1, v1.11.0 Dec 23, 2024

pan3793 approved these changes Dec 23, 2024

View reviewed changes

fix style

f6fc2d7

pan3793 closed this in 7e8275b Dec 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KYUUBI #5834] Add Grafana dashboard template #5147

[KYUUBI #5834] Add Grafana dashboard template #5147

zhaohehuhu commented Aug 9, 2023 •

edited by pan3793

Loading

zhaohehuhu commented Aug 9, 2023 •

edited

Loading

codecov-commenter commented Aug 9, 2023 •

edited

Loading

bowenliang123 commented Aug 9, 2023 •

edited

Loading

zwangsheng commented Aug 10, 2023

zhaohehuhu commented Aug 10, 2023 •

edited

Loading

zhaohehuhu commented Aug 10, 2023 •

edited

Loading

zhaohehuhu commented Sep 5, 2023

pan3793 Sep 22, 2023

zhaohehuhu Sep 25, 2023

pan3793 Sep 22, 2023

zhaohehuhu Sep 25, 2023

pan3793 commented Sep 22, 2023

zwangsheng Oct 25, 2023

zwangsheng Oct 25, 2023

zwangsheng Oct 25, 2023

zhaohehuhu commented Jan 5, 2024

yaooqinn commented Jan 5, 2024

bowenliang123 commented Jan 5, 2024

yaooqinn commented Jan 5, 2024

sudohainguyen commented Jun 22, 2024

zhaohehuhu commented Jul 4, 2024

hakeedra commented Nov 12, 2024

zhaohehuhu commented Nov 16, 2024 •

edited

Loading

pan3793 left a comment

pan3793 commented Dec 24, 2024

[KYUUBI #5834] Add Grafana dashboard template #5147

[KYUUBI #5834] Add Grafana dashboard template #5147

Conversation

zhaohehuhu commented Aug 9, 2023 • edited by pan3793 Loading

Why are the changes needed?

How was this patch tested?

zhaohehuhu commented Aug 9, 2023 • edited Loading

codecov-commenter commented Aug 9, 2023 • edited Loading

Codecov Report

bowenliang123 commented Aug 9, 2023 • edited Loading

zwangsheng commented Aug 10, 2023

zhaohehuhu commented Aug 10, 2023 • edited Loading

zhaohehuhu commented Aug 10, 2023 • edited Loading

zhaohehuhu commented Sep 5, 2023

pan3793 Sep 22, 2023

Choose a reason for hiding this comment

zhaohehuhu Sep 25, 2023

Choose a reason for hiding this comment

pan3793 Sep 22, 2023

Choose a reason for hiding this comment

zhaohehuhu Sep 25, 2023

Choose a reason for hiding this comment

pan3793 commented Sep 22, 2023

zwangsheng Oct 25, 2023

Choose a reason for hiding this comment

zwangsheng Oct 25, 2023

Choose a reason for hiding this comment

zwangsheng Oct 25, 2023

Choose a reason for hiding this comment

zhaohehuhu commented Jan 5, 2024

yaooqinn commented Jan 5, 2024

bowenliang123 commented Jan 5, 2024

yaooqinn commented Jan 5, 2024

sudohainguyen commented Jun 22, 2024

zhaohehuhu commented Jul 4, 2024

hakeedra commented Nov 12, 2024

zhaohehuhu commented Nov 16, 2024 • edited Loading

pan3793 left a comment

Choose a reason for hiding this comment

pan3793 commented Dec 24, 2024

zhaohehuhu commented Aug 9, 2023 •

edited by pan3793

Loading

zhaohehuhu commented Aug 9, 2023 •

edited

Loading

codecov-commenter commented Aug 9, 2023 •

edited

Loading

bowenliang123 commented Aug 9, 2023 •

edited

Loading

zhaohehuhu commented Aug 10, 2023 •

edited

Loading

zhaohehuhu commented Aug 10, 2023 •

edited

Loading

zhaohehuhu commented Nov 16, 2024 •

edited

Loading