metrics logging for pubsub_notifications is excessive

[bji]

Problem

I am running the votalizer against my secondary node; and I find that log lines like this:

[2023-10-09T22:41:14.674170206Z INFO solana_metrics::metrics] datapoint: pubsub_notifications created_to_queue_time_us=7i

are logged hundreds of millions of times per day. In a daily validator log of 406775704 log lines, 392969431 (or 96.6% of log lines) are this single log line.

Surely this excessive level of logging of this one metric is not needed? It takes up a huge amount of disk space and I don't know what effect it has on the validator performance itself to be logging this line so frequently.

Proposed Solution

Eliminate this log line altogether; or log it only once every N occurrences (maybe 10,000 occurrences?) if it's useful to periodically know that whatever it is logging, is happening.

[steviez]

I can take this one; I previously condensed some logging to make things less noisy in #31691, but I must have overlooked this datapoint. Looks like it can be folded into the existing SentNotificationStats struct

[bji]

Awesome, thanks for triaging this issue and making a plan, much appreciated!