RDCMan protocol error 3334 - largeaddressaware

[kpr0th]

Ran into a problem where RDCMan was failing to start a connection through a BeyondInsight Password Safe proxy from a virtual machine accessed through a BeyondTrust "Bomgar" jump client. It was reporting unknown protocol error: 3334. Searched online and found that some other similar tools (such as mRemoteNG) had a similar issue, and the solution suggested was to use Visual Studio's EDITBIN to enable /LARGEADDRESSAWARE mode on the RDCMan.exe binary. Lo and behold, I tested and that did solve the problem. Though it also invalidated the digital signature on the executable...

Will Microsoft please re-release RDCMan.exe with the largeaddressaware mode active (or release a 64-bit version instead?), so we don't have to use this manual edit for this corner case?

[foxmsft]

This is a great finding, I wasn't aware of it, thank you!

[kpr0th]

Adding to the mix: these were virtual machines that were misbehaving and showing the protocol error 3334. The largeaddressaware flag did not fix all of the instances that we had in our environment. I tracked another root cause down, which is that "Virtualization Based Security" was not enabled on some of the endpoints in vCenter, but we had group policy in place that was trying to enable the VBS-related features in the guest O/S. After enabling that option at the VM level in vCenter, the problem was resolved for the endpoints that were still having issues.