Error: Rpc(Transport(Error in the WebSocket handshake: i/o error: unexpected end of file #22
Comments
|
Reproduced it with |
|
Yes, it'a a jsonrpsee error for sure, the You can replace Would be good get some logs from polkadot node, it looks weird with EOF error in the handshake FWIW |
Do we have to modify the source for that? It uses a docker image and such... |
I think niklas meant using tracing-subscriber on our there's no docker involved in that |
Yes, it's one line in the utils code On the polkadot node side you could do:
However, I think Wireshark is the way to go as this setup seems quite complicated and probably performs huge number of calls. |
This comment was marked as outdated.
This comment was marked as outdated.
|
Those nodes are not even running jsonrpsee, it's running v0.9.21 i.e, the old jsornpc library. I guess if you guys could provide some instructions how to run this locally then I could have a look |
damn, sorry about that! here's another set.
there's two ways to reproduce: with zombienet's native mode or kubernetes mode. doing then you do |
|
another relevant point is that if this happens both with 0.9.21 and 0.9.22 maybe it's not really |
Yeah it at least makes me calmer :) The logs are pretty much useless on the node side so you can turn them off, it could be some nginx/proxy setting or something that rejects the connection too. It never reaches the jsonrpc code servers as I can see both in I pretty much know nothing about the zombienet or kubernetes :) On the subxt side you guys could try to open the websocket connection like this in a loop with some sleeps to try it a couple of times before killing the node but again if you find some logs from subxt/jsonrpsee on the client side that would be helpful |
|
The RPC trace log is blank for the node that does not get the connection. Therefore it could also be the kubernetes/docker stuff which does not correctly forward the port. PS: I tried again with a python script to connect to the IPs printed in console. |
Hi @ggwpez, I will double check this. You can't connect to the rpc endpoint? |
|
on the rust code, the error is originating on the |
|
Yes, that's when connecting the RPC endpoint under the hood in subxt. so if you could try something like this instead to try connect a couple of times if the RPC endpoint is busy or something: let mut i = 0;
let client = loop {
let err = match WsClientBuilder::default() {
.connection_timeout(std::time::Duration::from_secs(60)).
.build(&node)
.await {
Ok(c) => break c,
Err(e) => e
};
i += 1;
// what you think is a reasonable limit
if i > 10 {
return Err(err.into());
}
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
}
let api = ClientBuilder::new()
.set_client(client)
.build()
.await?
.to_runtime_api::<runtime::RuntimeApi<DefaultConfig, PolkadotExtrinsicParams<DefaultConfig>>>(
);But as it doesn't work for Oliver in python either there is mostly something flaky with the RPC endpoint configuration. |
Yea I added a retry loop. First time it gets the EOF error and then just "connection refused". Nothing in the node log. |
I modified the sender like this: and then called zombienet with: got these huge logs: cc @niklasad1 |
|
How are you fetching the WS ports from the "polkadot nodes" to connect to? From the logs I can see that old nodes are responsive except It would be good to know if the actual port is open, like doing |
|
Hi @niklasad1, Also, I just try to connect using polkadotjs and works
Could be the case that the Thanks! |
|
@pepoviola can I somehow print all port forwardings with the |
Yes, @ggwpez you can run this command in another terminal And you get something like this And then you can see the open port for each I can write an small bash script if you needed. |
|
I'm investigating from a different perspective. Then I manually fill the the error never happens on this approach, which indicates that there's something about the way zombienet is executing the |
|
@pepoviola sorry it seems this includes the latest |
|
Okay thanks for testing @bernardoaraujor ! |
|
although #28 was merged, the problem remains |
Hi @bernardoaraujor, I just run the test and works as expected. But I think, and I talk about this with @ggwpez that moving the Thanks! |
I got it to work a few times too, but it seemed to be just luck. If I tried repeatedly, it actually fails more frequently than not.
awesome! |
|
after many repeated tests on so I incrementally increased every version from I'm tempted to close this issue as the problem is gone, but it would be interesting to have some theories on what could have been the root cause. @niklasad1 does this info on the versions bring any insights? were there any meaningful changes on |
The major thing is that |
|
reopening as my investigation in October was a false alarm. issue persists, it's just not reproducible in a deterministic way. |
|
@pepoviola about this:
Is this already possible on zombienet? I think this is the only viable solution. |
while testing on
kubernetesmode, I am facing the following errors:A few patterns that caught my attention:
nativemode.utilsstarts reading funded accounts from:tests/stps/funded-accounts.json, although there's no RPC call in this process... it's just loading a json file locally.utils::senderstarts signing the transactions to be dispatched, which is way before any actual RPC call happens.utils::senderper node. As I increase the number of nodes in the setup, it's always the lastutils::senderthat dies (e.g.: 5 nodes, no txs are dispatched for node 4)@niklasad1 does this look like anything related to
jsonrpseeto you?all the points listed above tell me it's not, but maybe your experienced eyes can rule this out from a different perspecitve.
cc @ggwpez @pepoviola
The text was updated successfully, but these errors were encountered: