handle incorrect lastupdatedAt

[adityathebe]

2024-12-09 16:11:12 +0000 UTC 2024-12-09 16:13:42 +0000 UTC
21:58:41.77 WRN (watch.homelab) receive lag 2m30s. receivedAt: 2024-12-09 16:13:42 +0000 UTC lastUpdatedTime: 2024-12-09 16:11:12 +0000 UTC [Deployment/default/ittools]
2024-12-09 16:13:41 +0000 UTC 2024-12-09 16:13:42 +0000 UTC
2024-12-09 16:13:41 +0000 UTC 2024-12-09 16:13:42 +0000 UTC
2024-12-09 16:13:41 +0000 UTC 2024-12-09 16:13:42 +0000 UTC

I scaled down Deployment/default/ittools to 0 replicas.
2.5 minutes later, I scaled it up to 1 replica.

config-db received 4 updates from the shared informer.
The first object gives us an incorrect lastUpdatedAtTime (2m30s ago) (I checked the object and there's no correct time we can extract from it).
If I had scaled it up after 24hrs, the lastupdatetime would have been 24hrs.
However the subsequent updates for the object had correct timestamps.

Due to this, the receive lag is highly skewed.

[adityathebe]

I think - instead of instrumenting the objects as soon as they arrive, we should be buffering.
In the example above, we should be recording 1 lag time for the 4 updates we received.

or we record the receive lag right before we consume it - i.e. after dequeuing.
Because we already dedup the 4 objects into 1 object before we consume.