Server's performance and instrumentation

[WolfganP]

The same way this discussion #1052 triggered the advances in multi-threading, I think it will be worthy to have a separate discussion on performance instrumentation.

Performance instrumentation I believe is one of the unexplored themes in Jamulus' development efforts. There are many threads discussing issues with ping going nuts, unpredictable performance on large clients/choirs' set, effects of multi-threading configs, one bad client ruining the mix for everyone, etc etc; but most of the cause/effect and/or proposed solutions I read don't seem to be based on firm & reproducible evidence, but mostly anecdotal.

So I wanted to open this discussion to focus on what can be added to Jamulus in terms of performance monitoring, logs, etc that may help to informed troubleshootings. The monitoring categories at the top of my head:

• Cyclic timer run off (at the decoding or mix-encoding threads), that may indicate incomplete audio processing cycles (CPU bound).
• Threads forking/closing (it may indicate process trashing)
• Receiving/Transmitting packets Queues' length per client
• Missing packets from X client between audio processing cycles.
• ???

Feel free to add your thoughts. I'll try to maintain the list above the include all contributions.

[gene96817]

I believe I am sometimes hearing a user's channel accumulate delay. My hypothesis is this occurs from successive buffer underruns. The question is how many underruns and how long does each underrun condition lasts. A relarted condition would be dropped packets (especially from buffer overruns).

[WolfganP]

Yeah, that may be a side effect of how QTimer works. We discussed that topic on #599 (comment) , but never went forward on actually measuring timer overruns.

It definitely behaves differently than a hard interrupt timer.

[gene96817]

When was the change made? How should the behavior have changed? Seems to me if the interrupt timer fires often enough (less than two packet time equivalent), the behavior wouldn't be observable.

never went forward on actually measuring timer overruns.

What is the meaning of timer overrun? I expected you to say buffer overrun.

[WolfganP]

With the Qtimer being triggered every 1ms for audio processing, by timer overrun I meant that the intended audio processing (opus decode, mix+encode) of the cycle didn't finish in the available slot and a new timer is triggered for the next cycle to start.

[gene96817]

ah, got it... thanks. There are definitely surprises to the coder if the
expected tasks don't complete within one timer tick. It is important to know timer overruns are happening. Back to:

but never went forward on actually measuring timer overruns.

What was the thinking that this was not important? It is natural to think it never happens. That makes knowing the "never happens" even more important to know it happened. This suggests to me the CPU clock speed is very important when we compare hardware. The CPU clock speed directly correlates to timer overruns.

[ghost]

I think the Buffer and Delay LEDs on the Client give some indications, useful or not, of how useful adding more metrics to Jamulus will be. Maybe a few (very few) Server metrics would be just as useful.
Someone should experiment with system monitoring while Jamulus is running. There must be server monitoring software by now (at least there is 'top').

Note that if you decide to have a high interrupt rate for some reason, it should be a cli option. I see these metrics and fast interrupts similarly as I see the -T option. Jamulus should work exceptionally well with a 10 millisecond system clock rate (linux divider=10). If this cannot be done, then Jamulus will need its own hardware.

[WolfganP]

I don't know if you're referring to a 10msec system timer or something else in the code, but Jamulus already uses a 1 (or 2) msec timer for the audio processing threads (including opus de/re-compression and channels' mixing).

See

jamulus/src/server.cpp

Lines 66 to 82 in 6d662e3

	void CHighPrecisionTimer::Start()
	{
	// reset position pointer and counter
	iCurPosInVector = 0;
	iIntervalCounter = 0;
	if ( bUseDoubleSystemFrameSize )
	{
	// start internal timer with 2 ms resolution for 128 samples frame size
	Timer.start ( 2 );
	}
	else
	{
	// start internal timer with 1 ms resolution for 64 samples frame size
	Timer.start ( 1 );
	}
	}

[ghost]

Thanks for showing the code. I see that It has been there for some time. Some parts of Jamulus, if changed or not carefully considered, are sensitive to the KISS principle.
Thats all.