EM Metrics collection design

[bharathappali]

This discussion board is dedicated for the discussions over EM metrics collection part. The scripts are currently gathering metrics and feeding back to HPO module, This mechanism needs to be implemented in EM.

Key points to be discussed:

• Collection time (Scripts vs EM implementation)
• Aggregation of results to make up percentile info in absence of micrometer
• JVMTI Agent to collect percentile info (Experimental)
• Tree Distribution of obtained results for analysis

[bharathappali]

Implementation of EM Metric Map in EM Iteration Manager:

Current Implementation:

In the current EM implementation the EIM (EM Iteration Manager) the metrics are stored in a set of array list and at the time of returning/creating result JSON we parse the arraylist and generate the JSON structure which maps the input metrics.

ETD -> EIM -> EM iteration results -> (warmup / measurement lists irrespective of metrics - global metrics holder for trial)

Proposed Implementation:

Instead of storing in arraylist at a higher level of EIM we store the results in separate lists for each metric which will be mapped to the metric name and the map is linked to EIM, So when the EM pushes the result data it needs to select the particular list (warmup/measurement) for the metric name the data is collected and push it to that particular list. Caveats here are that EM needs to hold a MAP of LISTS which becomes a heavy chunk added to ETD (as EIM will be linked to ETD) but in the view of CPU vs MEM race I feel trading off memory for CPU is a better choice as we cannot parse the lists and generate the result map everytime the user asks for results so instead we maintain it as a metric map so that generating the result data wouldn't take more CPU.

ETD -> EIM -> EM Iteration Results -> Metrics Map (Number of keys = no. of metrics passed in input JSON, key = metric name, value = object of 3 lists (warmup/measurement/summarized))