Bug: Monitoring Pipelines Not Capturing OpenAI Data in Versions > v0.3.35

[samuelpetermoshi]

Bug Report

Description:
After upgrading to OpenWebUI versions greater than v0.3.35 (currently testing on v0.4.6), I observed that data from built-in OpenAI models is no longer being sent to monitoring pipelines like Datadog and Langfuse. This issue does not occur in v0.3.35 and earlier versions.

Pipelines:

1. Datadog: https://github.com/open-webui/pipelines/blob/main/examples/filters/datadog_filter_pipeline.py
2. Langfuse: https://github.com/open-webui/pipelines/blob/main/examples/filters/langfuse_filter_pipeline.py

Steps to Reproduce:

1. Host OpenWebUI on a local machine running version v0.3.35.
2. Submit prompts to OpenWebUI using built-in OpenAI models.
3. Verify that monitoring data is successfully sent to both Datadog and Langfuse.
4. Upgrade OpenWebUI to a version greater than v0.3.35 (e.g., v0.4.6).
5. Submit prompts to OpenWebUI using built-in OpenAI models.
6. Check Datadog and Langfuse for the monitoring data.

Expected Behavior:
Monitoring data from built-in OpenAI models should continue to flow to Datadog and Langfuse across all versions, including those greater than v0.3.35.

Observed Behavior:

• Monitoring data is not sent to Datadog or Langfuse when using OpenWebUI versions greater than v0.3.35.
• This issue is specific to built-in OpenAI models, as data from other models or components may not be affected.

Environment Details:

• Local Setup:
  • Version: v0.3.35
  • Monitoring Data Status: Sent successfully to Datadog and Langfuse.
• On-Premise Setup:
  • Version: v0.4.6
  • Monitoring Data Status: Not sent to Datadog and Langfuse.

Attachments:
I have attached screenshots to demonstrate the issue:

1. Monitoring pipelines capturing data in v0.3.35.
2. No data captured in v0.4.6.
3. Example traces in Langfuse and Datadog. (Note: Redacted data is intentional for privacy reasons.)

Impact:
This bug directly affects the ability to monitor and trace activities involving OpenAI data, which is critical for observability and debugging.

Request:
Please investigate this issue and confirm if this behavior is due to changes in how OpenWebUI handles OpenAI model data in newer versions. Let me know if further details or testing support is needed.

Thank You!
Thank you for your continuous efforts and updates to OpenWebUI. Looking forward to the fix!

The images show how it worked on v0.3.35

cc: @ith , @maycszz , @SuvroBaner , @arnab55net

[rossm-mf]

We seem to experience the same however we run casperhansen/llama-3-70b-instruct-awq locally and this one seems to still work on Langfuse. Tried the latest Langfuse 2.9 then even the version rc 3.0.0 and update the langfuse filter pipeline

[rossm-mf]

Got a quick discussion on langfuse about it. It seems to be about the prefix we add

langfuse/langfuse#4504 (comment)