[QUERY] Message completion slower with AMQP vs NetMessaging transport

[michaelmcmaster]

Library name and version

Azure.Messaging.ServiceBus 7.16.2

Query/Question

Background

I have an application that makes heavy use of ServiceBus sessions. I'm updating the application's ServiceBus client library from WindowsAzure.ServiceBus to Azure.Messaging.ServiceBus and testing is showing a reduction (roughly half as many messages/second) in performance using the newer Azure.Messaging.ServiceBus client. After significant tinkering and benchmarking, it appears the performance reduction is related to the "transport type" being used by the client; the AMQP transport takes roughly twice as long to perform the message Complete operation.

This "slow completion" (aka. message settlement) behavior is alluded to in #13920 as well, but doesn't have any resolution. My goal is to get the Azure.Messaging.ServiceBus client to the same performance level as the older WindowsAzure.ServiceBus client, but all the experiments I've tried... aside from scaling out the number of concurrent sessions... have proved fruitless. I've tried client-per-thread (instead of singleton), I've tried a range of Prefetch (0 to 100, settled on 10), I've tried using the Processor (instead of Receiver), and I've tested just a single session w/single thread... across all the experiments, the client's Complete method is significantly slower when using AMQP transport.

Benchmark Scenario

Setup

• Topic (duplicate detection enabled, message ordering enabled)
• Subscription (requires session enabled)
• Publish 1000 messages (10 sessions with 100 messages each) into Topic

Client Pseudocode

• Construct a single Subscription Client
  • Use default client options except increase PrefetchCount to 10
• Construct 10 threads which do the following
  • Using the (singleton) subscription client, Accept a message session
  • Using the (accepted) message session, enter the receive loop:
    • Receive a single message (using PeekLock)
    • Process the message (simple deserialize + log)
    • Complete the message
    • Repeat the receive loop until Receive returns null (no message)
• Capture (via stopwatch) the execution time for every ServiceBus library call

Benchmark Results

WindowsAzure.ServiceBus 7.0.1

TransportType: NetMessaging (client default):

Completed [10] sessions containing [1000] messages in [00:00:05.722] (174.735 msg/sec)
Receive  - Minimum:[0.000 ms], Average:[4.496 ms], Maximum:[148.000] ms
Process  - Minimum:[0.000 ms], Average:[0.501 ms], Maximum:[51.000] ms
Complete - Minimum:[33.000 ms], Average:[51.870 ms], Maximum:[241.000] ms

TransportType: Amqp (set via connection string):

Completed [10] sessions containing [1000] messages in [00:00:12.478] (80.135 msg/sec)
Receive - Minimum:[0.000 ms], Average:[0.819 ms], Maximum:[52.000] ms
Process - Minimum:[0.000 ms], Average:[0.355 ms], Maximum:[35.000] ms
Complete - Minimum:[63.000 ms], Average:[123.271 ms], Maximum:[398.000] ms

Azure.Messaging.ServiceBus 7.16.2

TransportType: Amqp (client default):

Completed [10] sessions containing [1000] messages in [00:00:11.846] (84.410 msg/sec)
Receive - Minimum:[0.000 ms], Average:[0.991 ms], Maximum:[86.000] ms
Process - Minimum:[0.000 ms], Average:[0.612 ms], Maximum:[61.000] ms
Complete - Minimum:[38.000 ms], Average:[116.760 ms], Maximum:[259.000] ms

Observation

Comparing the NetMessaging transport (WindowsAzure.ServiceBus) against the Amqp transport (both old and new) libraries, the old NetMessaging Receive average is slightly slower... but the Complete is much faster.

Question

I've tuned the client's options within the constraints of the application's design. Is there anything that can be done to improve the Azure.Messaging.ServiceBus client's Complete performance?

Environment

Windows 10, .NET Framework 4.8.9181.0, x64

[jsquire]

Hi @michaelmcmaster. Thank you for the detailed and well-researched issue; we appreciate the effort that you put into gathering data and it was helpful. This looks to be a protocol-level difference within the service based on the similar performance for AMQP between the two libraries. Unfortunately, this isn't something that the client has insight into nor can influence. From your description of the pattern that your application is using, we don't see any obvious suggestions for improvement.

This is something that we'll need the Service Bus service team to investigate and advise on. I've transferred this over to the Service Bus repository as #690, which will ensure that the service team has visibility and can assist.

[github-actions]

Hi @michaelmcmaster. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[michaelmcmaster]

This looks to be a protocol-level difference within the service based on the similar performance for AMQP between the two libraries. Unfortunately, this isn't something that the client has insight into nor can influence.

@jsquire I couldn't tell if this difference in behavior was client-side (ie. in the libraries) or server-side. Thank you for forwarding.

[github-actions]

Hi @michaelmcmaster, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.