Change FBMjpegScalingFactor make device slow

[coinhu1995]

I change FBMjpegScalingFactor to 60 then device seem to be slower than normal

Environment:

• Iphone XS Max
• OS: 13.5.1
• Xcode 12.5.1

[KazuCocoa]

Possibly the cause is inside XCTest API:

WebDriverAgent/WebDriverAgentLib/Utilities/FBMjpegServer.m

Lines 102 to 106 in a182489

	[proxy _XCT_requestScreenshotOfScreenWithID:[[XCUIScreen mainScreen] displayID]
	withRect:CGRectNull
	uti:(__bridge id)kUTTypeJPEG
	compressionQuality:screenshotCompressionQuality
	withReply:^(NSData *data, NSError *error) {

The end to generate the screenshot is by _XCT_requestScreenshotOfScreenWithID which is provided by XCTest.
Then, nothing can do in Appium layer. We only can tweak the scale factor.

[nanoscopic]

Generally Mjpeg coming out of WDA is quite slow because it fetches the screenshot at full resolution size. The resizing code isn't carefully written to use hardware acceleration to do the resizing, and so it takes a bit longer than it should.

There are a few ways to get better video from an iOS device:

1. Use Daniel Paulus quicktime_video_hack repo or similar things. A few people have developed code that uses the same technique to use the usb video stream of h264 that can be enabled that optimally sends out the screen content. I wrote one of those things myself as well... For the most part this method works well.

2. Write your own video broadcast extension and use that. This is the path I chose to go down and embedded into ControlFloor. This results in a much faster frame rate than can possibly be done with Xctest. I'd highly recommend this. This is an officially supported Apple path. ( whereas XCT... is not; is private API )

3. Use something that illegally breaks the crazy encryption crap surrounding AirPlay. There were/are some options to do this. I think Apple pressured most of the people working on this to stop. I'd recommend against it as Apple seems to be clearly against it, and generally one should not do things Apple dislikes.

Since this project is just an Xctest, its options are limited. The only thing that could be done to improve the speed is to optimize how the resizing is done a bit better. I'd also recommend not using Mjpeg, and not using Http. My fork of WDA uses NNG instead to reduce overhead slightly. That is, though, a minor optimization and barely makes a difference.

[krishtoautomate]

is it possible to buffer vide stream in h264 format to increase performance. mjpeg seems to be bit heavy. apple supports H.264/AAC MOV encoding without effecting performance.

using Replaykit or AVFoundation is not stable as current wda video buffer, which is loads faster comparatively.
Replaykit : need separate xcuitest to launch app and trigger start broadcast.
AVFoundation : it is similar to quicktime player, need reset of USB which fails 30-40% of times and current running tests in devices will be disturbed.