Add OTLP log endpoint to convert log events to span annotations

[codefromthecrypt]

Feature

This would add a configurable log listener, which would convert log events to span annotations. It could make some heuristic decisions to say tell the difference between a normal log record and a log event. Also, it could make a decision on if it drops the bodies, or attaches them to the span as tag values.

This should be an option, not always on, as certainly some will be fine to use the existing approach of storing logs in ElasticSearch and using UI properties to link to a query.

Rationale

This helps as currently span events are migrating to log events, which dramatically impacts navigation in zipkin. For example, you would need to setup log correlation and jump screens to get to them, vs now when annotations are in the same UI.

Technically, the out-of-order concern isn't a problem in zipkin as out-of-order data has been a norm since day one. Specifically the problem of span annotations sent before or after the span isn't an issue.

Example Scenario

Notably, this is just now happening with OpenAI which moved from span events to log events.

open-telemetry/opentelemetry-python-contrib#3006

Prior Art

None so far as there's no work on something like this to my knowledge.