backend: Pull ledger data from Hubble

[acharb]

Right now ledger data is aggregated from Horizon. Instead we can pull it from Hubble. Benefits:

• Dashboard makes a lot of calls to Horizon, which creates some intermittent rate limit errors (eg. of an update with a lot of Horizon calls, and right now there's three of those)
• if the ledgers cache in redis ever gets lost/corrupted, the catchup using Horizon makes a lot of calls to Horizon that creates rate limit errors (took 8 pod restarts to finish today's catchup)
• dashboard v2 has some additional ledger queries (1HR ledgers, 24HR ledgers) that will be easier to pull from hubble using sql than aggregating like it is doing now

Downsides to consider:

• hubble has a cost/query (rough estimate is about 600 mb/query 1GB/query)
• hubble is updated every 30 minutes

[acharb]

rough estimate of 1 GB/query at $5/TB = .0005 $/query
making a call every 30 mins (hubble updates every 30 mins) = about $.25/day. So pretty negligible

[Selhar]

Hey @acharb, just to clarify, all calls on v2 backend using the stellar sdk + horizon endpoint should now make bigquery calls, correct? Are there any exceptions or edge cases i should be aware? Thanks in advance!

[acharb]

hey, so after looking at figma designs, we also need this data for testnet, which unfortunately our Big Query datasets don't have. I didn't think about testnet when I created this ticket :/ .

So I think instead we should keep using stellar sdk + horizon to get the 1HR, 24HR, and 30D ledger data. The 30D ledger data is already being pulled in for V1, so we'd be adding the 1HR and 24HR data. Then switching between mainnet and testnet will use the same logic.

Lmk if you have any questions, I'll close this ticket, these two tickets should encompass the work that's needed: #213 #214