Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add connections_opened metric #1754

Merged
merged 1 commit into from
Sep 23, 2024

Conversation

rukai
Copy link
Member

@rukai rukai commented Sep 19, 2024

Opening new connections is quite expensive on kafka, not only do we have to reperform authentication, the new transform instance created for this new connection must fetch its own metadata from kafka.
To give visibility into this I have added this connections_opened metric which increments every time a new connection is opened.

The results of this metric on the windsock benchmark:
image

We can see that

  • we initially create 4 connections, presumably 2 for the producer and 2 for the consumer.
  • after that no more connections are created

So we can conclude that the performance issues is not caused by the client creating more connections.
I believe this metric should still be added as its helpful for ruling out such possibilities in the future.

Copy link

codspeed-hq bot commented Sep 19, 2024

CodSpeed Performance Report

Merging #1754 will not alter performance

Comparing rukai:add_connections_opened_metric (cf37951) with main (4693aab)

Summary

✅ 39 untouched benchmarks

@rukai rukai force-pushed the add_connections_opened_metric branch 2 times, most recently from e4047a6 to 0a9edce Compare September 19, 2024 07:25
@rukai rukai force-pushed the add_connections_opened_metric branch from 0a9edce to cf37951 Compare September 22, 2024 23:30
@rukai rukai merged commit 84189fc into shotover:main Sep 23, 2024
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants