Validate the metrics

[kusumachalasani]

Below are list of steps which needs to be considered to validate the metric:

1. server_errors from prometheus side / web_errors from client side should be ideally zero.
2. The metric should have converged. Confidence Interval is calculated to understand the convergence of multiple data points collected. In an dedicated n/w, the convergence should be under 3% and it can be varied considering the multiple factors which can be.
   a. Shared network
   b. The application itself shows some inconsistencies all the times - which might be due to application itself , runtime used etc.
3. If the metric is not a startup time, the application should have warm up'ed enough before we collect the measurement metrics. This ensures for enough compilation time and optimizations required.
4. For a valid response time, the tail latencies should not be very high compared to average.