Instrument app-lib for OpenTelemetry

[HauklandJ]

Description

Part of the Monitoring epic.

Now that Open Telemetry has been wired up, some automatic instrumentation is handled by the OpenTelemetry SDK and the instrumentation libraries.
It is time to add Manual Intrumentation to app-lib to make sense of the produced observability elements. This includes creating custom spans and metrics.

Custom spans can be created using the OpenTelemetry API:

using var tracer = TracerProvider.Default.GetTracer("YourTracerName");

using var span = tracer.StartActiveSpan("OperationName");
try
{
    // Your operation code
}
catch (Exception ex)
{
    span.RecordException(ex);
    throw;
}
finally
{
    span.End();
}

NOTE: Deciding on what to instrument is a part of this issue.

Additional Information

No response