definition of 'task run time'

[cometta]

i triggered a spark job for a few minutes, but i see from Grafana dashboard showing 'task run time' of at least one hour. is the info on Grafana correct?

[LucaCanali]

It seems accurate and aligns with my own experience as well. The metrics you're observing are cumulative across all executed tasks and all executors. The main idea behind using Apache Spark is to parallelize execution, allowing multiple CPUs/tasks to work simultaneously. Additionally, the metric "Number of Active Tasks" indicates how many tasks are being executed in parallel.

[cometta]

for the case when i only need to know total duration = end time - start time of Spark job. any widget i can refer to?

[LucaCanali]

That's the kind of basic information you can easily get from the Spark Web UI, see https://spark.apache.org/docs/latest/web-ui.html