Cron job script feedback

[cduhn17]

💡 Summary

When a cron job to run all pe-report scans starts there is not positive feedback as to when the scan started, stopped or incremental progress and completion of scans. Also there we would want to know when the scans are completed so that we know when the optimal time is to run report generation.

Motivation and context

We have ELK to monitor logs and we can use the logs to build dashboards to show all of the pe-reports metrics

Implementation notes

• add start and stop indicators in logs of when the scans start/stop
• make a org count query and then count total complete from logs to establish percentage complete
  The following is the log nomenclature to track:
  IntelX: Running IntelX on <org_id>

DnsTwist: Running dnstwist on <org_id>

HIBP: Running HIBP on <org_id>

Shodan: Running IPs for <org_id>

DNSMonitor: Running DNSMonitor on <org_id>

Cybersixgill Mentions: Fetching mention data for <org_id>

Cybersixgill Alerts: Fetching alert data for <org_id>

Cybersixgill credentials: Fetching credential data for <org_id>

Acceptance criteria

How do we know when this work is done?

• When start time is recorded
• When end of scan time is recorded
• When percent completed is established