Improve retry extension UX

[Evangelink]

Summary

Currently the retry extension has a basic/poor UX, we should onboard it to the new terminal reporter.

Background and Motivation

Inside each test run, we are using the new terminal reporter UX but we are not having a same quality ux for the retry levels.

Proposed Feature

Move from:

MSTest v3.6.3 (UTC 11/11/2024) [win-x64 - .NET 8.0.11]
Test Parallelization enabled for C:\src\mixed-mode\ProjectMixedMode\ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (Workers: 16, Scope: MethodLevel)
passed MTPTestMethod1 (3ms)
failed MTPTestMethod2 (2s 062ms)
  Assert.Fail failed.
    at ProjectWithMTP.MTPTest1.MTPTestMethod2() in ProjectWithMTP\MTPTest1.cs:17
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Test run summary: Failed! - ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (net8.0|x64)
  total: 2
  failed: 1
  succeeded: 1
  skipped: 0
  duration: 2s 201ms

=====================
Tests suite failed, total failed tests: 1, exit code: 2, attempt: 1/3
=====================

Test Parallelization enabled for C:\src\mixed-mode\ProjectMixedMode\ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (Workers: 16, Scope: MethodLevel)
failed MTPTestMethod2 (2s 042ms)
  Assert.Fail failed.
    at ProjectWithMTP.MTPTest1.MTPTestMethod2() in ProjectWithMTP\MTPTest1.cs:17
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Test run summary: Failed! - ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (net8.0|x64)
  total: 1
  failed: 1
  succeeded: 0
  skipped: 0
  duration: 2s 216ms

=====================
Tests suite failed, total failed tests: 1, exit code: 2, attempt: 2/3
=====================

Test Parallelization enabled for C:\src\mixed-mode\ProjectMixedMode\ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (Workers: 16, Scope: MethodLevel)
passed MTPTestMethod2 (2s 018ms)

Test run summary: Passed! - ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 2s 175ms

=====================
Tests suite completed successfully in 3 attempts
=====================

to something that feels more integrated to the new suite.

Suggestion 1:

MSTest v3.6.3 (UTC 11/11/2024) [win-x64 - .NET 8.0.11]
Test Parallelization enabled for C:\src\mixed-mode\ProjectMixedMode\ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (Workers: 16, Scope: MethodLevel)
passed MTPTestMethod1 (3ms)
failed MTPTestMethod2 (2s 062ms) - attempt 1
  Assert.Fail failed.
    at ProjectWithMTP.MTPTest1.MTPTestMethod2() in ProjectWithMTP\MTPTest1.cs:17
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
failed MTPTestMethod2 (2s 042ms) - attempt 2
  Assert.Fail failed.
    at ProjectWithMTP.MTPTest1.MTPTestMethod2() in ProjectWithMTP\MTPTest1.cs:17
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
passed MTPTestMethod2 (2s 026ms) - attempt 3

Test run summary: Failed! - ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (net8.0|x64)
  total: 2
  failed: 2
  succeeded: 2
  skipped: 0
  retried: 2
  duration: 6s 216ms

Suggestion 2:

MSTest v3.6.3 (UTC 11/11/2024) [win-x64 - .NET 8.0.11]
Test Parallelization enabled for C:\src\mixed-mode\ProjectMixedMode\ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (Workers: 16, Scope: MethodLevel)
passed MTPTestMethod1 (3ms)
failed MTPTestMethod2 (2s 062ms)
  Assert.Fail failed.
    at ProjectWithMTP.MTPTest1.MTPTestMethod2() in ProjectWithMTP\MTPTest1.cs:17
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Test run summary: Failed! - ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (net8.0|x64)
  total: 2
  failed: 1
  succeeded: 1
  skipped: 0
  duration: 2s 201ms

Starting attempt 2/3
Test Parallelization enabled for C:\src\mixed-mode\ProjectMixedMode\ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (Workers: 16, Scope: MethodLevel)
failed MTPTestMethod2 (2s 042ms)
  Assert.Fail failed.
    at ProjectWithMTP.MTPTest1.MTPTestMethod2() in ProjectWithMTP\MTPTest1.cs:17
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Test run summary: Failed! - ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (net8.0|x64)
  total: 1
  failed: 1
  succeeded: 0
  skipped: 0
  duration: 2s 216ms

Starting attempt 3/3
Test Parallelization enabled for C:\src\mixed-mode\ProjectMixedMode\ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (Workers: 16, Scope: MethodLevel)
passed MTPTestMethod2 (2s 018ms)

Test run summary: Passed! - ProjectWithMTP\bin\Debug\net8.0\ProjectWithMTP.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 2s 175ms

[nohwnd]

Nice, looks similar to hotreload ui. Is this purely ux or we have a way to describe the retries in trx as well?

[Evangelink]

At the moment, trx is producing different trx files. We are waiting from some input from AzDO team so we could define how to have them supporting our retry.

Maybe we can benefit from the work you will be doing in defining the schema + metadatas