feat: jitter on cron jobs

[adityathebe]

No description provided.

[moshloop]

I think the jitter should be once off at the start, i.e. delay the scheduling of the job by random(min(duration till next runtime, 1h))

[adityathebe]

@moshloop I'm leaving off delaying the initial scheduling even though we discussed we would add a 100% jitter to it.

The problem is that if we delay the scheduling by running entryID, err := cronRunner.AddJob(schedule, delayedJob) in a goroutine, the AddToScheduler func returns with no error but the job actually hasn't been scheduled. i.e. job.entryID = nil.

If the job needs to be rescheduled (scraper CRD updated) when it's still waiting for first schedule, then it's not possible to unschedule is as we won't have the entryID.

I think let's try with per iteration jitter for now and see how it goes?

[moshloop]

I think this will have fixed 15m jitters, so if interval * delayPercent then interval = 15m