Testing server problems

[Varunram]

We should arrive at a better testing server solution that we can rely on and which doesn't fail randomly. The pattern (wrt failures) sorted according to categories is as follows :

1. The default setup works fine and the other two newer solutions fail
2. The newer solutions work and the older one (default) fails (Added more graceful handling to some errors on startup #406)
3. One of the newer test build runs fine, default runs fine (Added support for p2p function calls #407, Ability to start a LNDC proxy disconnected #404)
4. All three fail (Restructure the way stuff is handled in lnp2p #403)

We should consider moving to Travis in this situation since even Travis seems more reliable than the setup that we have right now. Else, we should arrive at a single test, disable all the other ones and make sure that doesn't fail.

[delbonis]

For #406 I'm not sure what the problem was, but it seems like the issue was similar to the problems we were having before where for some reason commands just "didn't run". For some reason it couldn't find the lit executable?

The problem with #407 seems like that's actually a race condition or (very rare) nondeterminism with lit or runtest.py with open tx inclusion.

Error: Channel height doesn't match new block.

For #404 I'm not sure.

For #403 it's because there was a bug introduced where lit mixes up PKHes:

RPC call failed: Remote PKH doesn't match. Quitting!