feat: do not store notification events to job history. Keep track of any filter errors in the notification table itself.

[adityathebe]

Resolves: #542
Merge after: flanksource/duty#287

[adityathebe]

@moshloop On a second look, I think what we currently have in events/notifications.go file should stay there and shouldn't be move the notification package.

I see the notification package as our mail package which is just the client to send notification. It can be called from any other places to send a notification. So, I think it shouldn't hold in any logic for event queue handling like

• creating a notification.send event from check.failed event, or
• deciding the title and body for incident related events.

We could move to a new notification/events package instead but I still think what we currently have is fine.

[moshloop]

@adityathebe - notifications.go is still too large

• These all should be moved to notification package and the prefix removed: NotificationEventPayload, NotificationTemplate, labelsTemplate, defaultTitleAndBody, sendNotification
• addNotificationEvent should be trimmed down to only forwarding the relevant details to notification package - e.g. GetNotifications should only be called from within the notification package,
• getEnvForEvent should remain in events.go

[adityathebe]

@moshloop I've made the changes.