Connection counts are 2x for the same settings since we stopped counting ones in grace period

[olizilla]

We're running 1 gateway node with the changes from #50

With the connmgr settings

"GracePeriod": "2m",
"HighWater": 5000,
"LowWater": 4000,

the older nodes consistently have ~4.5k peers. The new node slowly accrued more peers than it could trim. When it got to 15k, we rolled out a change to reduce LowWater to 3000, but the node is still averaging over 10K peers.

https://protocollabs.grafana.net/d/c8a7ZhKZz/gateway-slis?orgId=1&refresh=2m&from=1573027473386&to=1573031073386&var-location=All&var-interval=1m&var-instance=All

Would reducing the GracePeriod be a better choice for keeping the peer count closer to 5k in this situatuion?

We'll need to add a note to the release notes to let folks know they should expect 2x peer counts for long running nodes with the next release.

[Stebalien]

So, I'd expect us to have almost exactly HighWater more peers. Previously, on nodes receiving and making many connections, all the peers we kept around were in the grace period. Now, we keep HighWater peers in addition to those in the grace period.

We could change this to LowWater more peers.

Would reducing the GracePeriod be a better choice for keeping the peer count closer to 5k in this situatuion?

Yes, and that should also flush out most transient peers performing DHT lookups. I'd try halving it and then going from there.

We'll need to add a note to the release notes to let folks know they should expect 2x peer counts for long running nodes with the next release.

For nodes creating a bunch of connections, yes. Nodes that don't form 1000s of connections a minute aren't going to have this problem.

[Stebalien]

(and thanks for the report, this kind of information is really helpful)