v2 api for atxs

[dshulyak]

api: spacemeshos/api#281

• watch support. subscribe to atxs emitted after validating and saving database
• encode the rest of the fields
• add support for limit and offset and make use of it in List api. public services should not expose Stream, as it is much harder to track resources for streams
• extract into separate service that has access only for to db pointer, in order to enable readonly replicas
• always order by epoch asc, id
• matcher for watcher based on request parameters
• refactor services with generics
• extract query builder into separate module
• test coverage

---

streaming full atxs

stream all atxs before and including publish epoch 2:

grpcurl -plaintext -d '{"end_epoch": 2}' 127.0.0.1:9093 spacemesh.v2.ActivationStreamService.Stream

stream all atxs that belong to the specific smesher

grpcurl -plaintext -d '{"node_id": "MaAcOOLRj0pffyWO8gfuuWiBszQG2ueq/SNbbMnHHoU="}' 127.0.0.1:9093 spacemesh.v2.ActivationStreamService

the same as above, but also continue waiting for atxs from same smesher

grpcurl -plaintext -d '{"node_id": "MaAcOOLRj0pffyWO8gfuuWiBszQG2ueq/SNbbMnHHoU=", "watch": true}' 127.0.0.1:9093 spacemesh.v2.ActivationStreamService.Stream

streaming headers

all queries above will return full atx data, which in total is about ~1KB. it is useful if someone knows what they are looking for, but generally less amount of data is sufficient. for this purpose StreamHeaders exists and will return much less data:

grpcurl -plaintext -d '{"node_id": "MaAcOOLRj0pffyWO8gfuuWiBszQG2ueq/SNbbMnHHoU="}' 127.0.0.1:9093 spacemesh.v2.ActivationStreamService.StreamHeaders

{
  "v1": {
    "id": "sJAmeZqJr3m40nhQ4y5pMh+uW0rK6uyQGhEN0K8JRe8=",
    "nodeId": "MaAcOOLRj0pffyWO8gfuuWiBszQG2ueq/SNbbMnHHoU=",
    "publishEpoch": 3,
    "coinbase": "sm1qqqqqqq986rj6x5lfzctc6h8wf6uddqpdxr6n2c36nvmx",
    "units": 4,
    "baseHeight": 18880,
    "ticks": 9412
  }
}

listing headers and atxs

streaming if more convenient to use (less memory footprint, can read whole collection from same stream), however
it is harder to enforce fair distribution of resources for multiple concurrent users. for public api it is better to use req/resp pattern
with strict limits and no ability to continuously watch collections.

list full atxs

grpcurl -plaintext -d '{"start_epoch": 2, "limit": 100, "offset": 100}' 127.0.0.1:9092 spacemesh.v2.ActivationService.List

list headers

grpcurl -plaintext -d '{"start_epoch": 2, "limit": 10000, "offset": 10000}' 127.0.0.1:9092 spacemesh.v2.ActivationService.ListHeaders

note that limit and offset are effective only for specific query, so that if you started iteration with start_epoch: 2, you can't just bump start_epoch to 3 and continue using same offset and limit if you care about getting correct results.

scalable read only replicas

non-streaming api by design references only database instance, therefore we can scale it to match demand and isolate from the node that listens to consensus:

• binary that enables only isolated read only apis
• replicated enabled with https://litestream.io/

versioning

data in spacemesh is mostly static and not expected to change constantly. for the case when it does change, object can be bumped to the next version. all responses return protobuf oneof version type.

other tasks

read only apis:

• rewards
• accounts
• layers (applied block, opinion)

---

• ballots / proposals
• hare certificates
• blocks
• transaction results (copy from original and add lister)
• malicious identities (remove references to lru cache and add lister)

---

other interesting api is for mempool txs, inclusion but it will be considered separately

[codecov]

Codecov Report

Merging #5233 (af95f33) into develop (a66b274) will decrease coverage by 0.8%.
The diff coverage is 5.1%.

@@            Coverage Diff            @@
##           develop   #5233     +/-   ##
=========================================
- Coverage     78.1%   77.4%   -0.8%     
=========================================
  Files          266     267      +1     
  Lines        31943   32251    +308     
=========================================
+ Hits         24953   24964     +11     
- Misses        5481    5777    +296     
- Partials      1509    1510      +1     

Files	Coverage Δ
api/grpcserver/config.go	100.0% <100.0%> (ø)
node/node.go	67.2% <100.0%> (+<0.1%)	⬆️
sql/atxs/atxs.go	76.5% <0.0%> (-8.8%)	⬇️
api/grpcserver/v2/activation.go	3.8% <3.8%> (ø)

... and 4 files with indirect coverage changes

[pigmej]

I think we also need some way to display the count of ATX per epoch. That could be used to show sync atx progress.

It should be per epoch.