Set up Newrelic and investigate disk usage issues

[vinayvenu]

Since release 4.0, we've been having consistently high disk usage on the prod db. ETL was shut down on Sep 15 and brought back up on Sep 18. During this time, EBS byte balance stayed high. This points to the ETL job being the cause of high disk usage. This story is to

Acceptance criteria

1. Set up NewRelic on avni-etl so that we can identify these issues
2. Identify and resolve bottlenecks causing high disk usage

[petmongrels]

new relic config added to keeweb.
this needs to be present in start.sh -Dnewrelic.environment=production -javaagent:/opt/newrelic/newrelic.jar (this doesn't seem be changed / managed by deploy scripts right now)

[petmongrels]

sync telemetry insert seems to be taking a disproportionate amount of time compared to value added by it.

[vinayvenu]

Missing index?

…

On Mon, Sep 18, 2023, 19:49 Vivek Singh ***@***.***> wrote: sync telemetry insert seems to be taking a disproportionate amount of time compared to value added by it. — Reply to this email directly, view it on GitHub <#71 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACUBHRDNWNONYGYE72KMPTX3BKABANCNFSM6AAAAAA44JECHI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[petmongrels]

Created the index.
There are no other queries of big concern as of now on ETL. There are few slow ones but they are less frequently run, so not very time consuming. The RDS graph looks better (not sure it is only because of this or if we made other changes as well).