cleanup repo structure

.github/workflows/docker.yml

@@ -2,6 +2,9 @@ name: Build & Push Docker Image
 
 on:
   push:
+    paths:
+      - "api-service/**"
+      - "proto/**"
     branches:
       - "main"
     tags:
@@ -58,4 +61,5 @@ jobs:
           GOPRIVATE: github.com/pinax-network,github.com/eosnationftw,github.com/pomelo-io
           SSH_AUTH_SOCK: /tmp/ssh_agent.sock
         run: |
+          cd api-service
           task build:docker

.github/workflows/go.yml

@@ -46,4 +46,5 @@ jobs:
           GOPRIVATE: github.com/pinax-network,github.com/eosnationftw,github.com/pomelo-io
           SSH_AUTH_SOCK: /tmp/ssh_agent.sock
         run: |
+          cd api-service
           task build

README.md

@@ -1,36 +1,6 @@
-# Ethereum Consensus Layer Blob Service
-
-### From Source
-
-- Build substreams and start KV sink
-```bash
-$ cd substreams
-$ task protogen # if needed
-$ task sink
-```
-
-- Start Blobs service
-```bash
-$ task protogen
-$ task generate:go
-$ task start:service
-```
-
-### Or, with Docker:
-
-- Start KV sink
-```bash
-$ cd substreams
-$ task start:docker`
-```
-
-- Start Blob service
-```bash
-$ task start:docker`
-```
-
-
-### Query
-```bash
-$ curl -v http://localhost:8080/eth/v1/beacon/blob_sidecars/7677000
-```
+# Ethereum Blobs Monorepo
+This repository contains several components:
+- `proto` - Blobs protobuf definitions
+- `substreams` - Substreams modules to stream blobs data from substreams-enabled Consensus Layer Firehose
+- `subgraph` - Substreams-based Subgraph that uses substreams module
+- `api-service` - REST API service to access to blobs via substreams sink that can be used as a drop-in replacement for Consensus Layer clients' blob_sidecar API

.gitignore → api-service/.gitignore

@@ -1,5 +1,6 @@
 config*.yaml
 !config.example.yaml
+!config.docker.yaml
 blob-service
 __debug_bin*
 

api-service/README.md

@@ -0,0 +1,24 @@
+# Blobs REST API Service
+This service provides REST API access to Ethereum blobs via KV sink backend.
+To run it you need to have KV sink backend running and consuming [blobs substreams](../substreams) locally or on the server
+
+### Quick Start
+
+- From source:
+```bash
+> task protogen         # generate Go protobuf bindings
+> task generate:go      # generate swagger docs
+> task start:service    # build and start service
+```
+
+- Or, with Docker:
+```bash
+> task start:docker
+```
+
+
+### Query
+```bash
+> curl -v http://localhost:8080/eth/v1/beacon/blob_sidecars/7677000
+> curl -v http://localhost:8080/health
+```

Taskfile.yaml → api-service/Taskfile.yaml

@@ -102,15 +102,15 @@ tasks:
     cmds:
       - docker run
         -p 8080:8080
-        -v $(pwd)/docker-config.yaml:/app/config.yaml
+        -v $(pwd)/config.docker.yaml:/app/config.yaml
         ghcr.io/pinax-network/blob-service:v0.6.2
         -config /app/config.yaml
 
   protogen:
     desc: "Builds types from protobuf files"
     cmds:
       - mkdir -p pb
-      - protoc --go_out=./pb --go_opt=paths=source_relative --proto_path=substreams/proto pinax/ethereum/blobs/v1/blobs.proto
+      - protoc --go_out=./pb --go_opt=paths=source_relative --proto_path=../proto pinax/ethereum/blobs/v1/blobs.proto
       - ls pb/pinax/ethereum/blobs/v1/*.pb.go | xargs -n1 -IX bash -c 'sed s/,omitempty// X > X.tmp && mv X{.tmp,}'
 
   test:format:

test.http → api-service/test.http

@@ -2,12 +2,11 @@ GET http://localhost:8080/eth/v1/beacon/blob_sidecars/7677000
 
 ###
 
-GET http://localhost:8080/eth/v1/beacon/blob_sidecars/7658912
+GET http://localhost:8080/eth/v1/beacon/blob_sidecars/head
 
 ###
 
-GET http://localhost:8080/eth/v1/beacon/blob_sidecars/head
-
+GET http://localhost:8080/health
 
 
 ###
@@ -16,4 +15,4 @@ GET http://goerli-arch41.mar.eosn.io:5052/eth/v1/beacon/blob_sidecars/7677000
 
 ###
 
-GET http://goerli-arch41.mar.eosn.io:5052/eth/v1/beacon/blob_sidecars/7658912
+GET http://goerli-arch41.mar.eosn.io:5052/eth/v1/beacon/blob_sidecars/head

subgraph/README.md

@@ -0,0 +1,20 @@
+# Ethereum Blobs Subgraph
+
+### Running locally
+
+- Set up `$SUBSTREAMS_ENDPOINT` and `$SUBSTREAMS_API_TOKEN` env variables
+
+- Start Graph node, IPFS node and Postgres database in Docker:
+```bash
+> cd ./graph-node
+> ./up.sh -c        # use -c flag to cleanup database and start from scratch
+```
+
+- Build, create and deploy the subgraph
+```bash
+> yarn build
+> yarn create-local
+> yarn deploy-local
+```
+
+- Query: http://localhost:8000/subgraphs/name/blobs/graphql

substreams/README.md

@@ -1,39 +1,92 @@
-# Ethereum Beacon chain Blobs Substream
+# Ethereum Blobs Substream
+This substreams package offers two sink map modules:
+- `kv_out` - to sink into KV store
+- `graph_out` - to sink into [substreams-based Subgraph](../subgraph)
 
-### [Latest Releases](https://github.com/pinax-network/substreams/releases)
+### [Latest Releases](https://github.com/pinax-network/blob-service/releases)
 
-### Quickstart
+### Quick Start
 
 ```bash
-$ make gui
+> make protogen   # if needed
+> make gui
 ```
 
 or
 
 ```bash
-$ substreams gui -e goerli-scdm37b.mar.eosn.io:10016 https://github.com/pinax-network/substreams/releases/download/blobs-v0.2.0/eth-blobs-v0.2.0.spkg map_blobs -s -100 --plaintext
+> substreams gui -e goerli-cl.substreams.pinax.network:443 map_blobs -s -100
 ```
 
-### Graph
+### Sink
+
+To start sinking the data using KV sink into local KV store use
+```bash
+> task sink
+```
+
+
+- Or, with Docker:
+```bash
+> task start:docker
+```
+
+
+
+### Package Info
 
 ```mermaid
 graph TD;
   map_blobs[map: map_blobs];
   sf.beacon.type.v1.Block[source: sf.beacon.type.v1.Block] --> map_blobs;
-
+  kv_out[map: kv_out];
+  map_blobs --> kv_out;
+  graph_out[map: graph_out];
+  map_blobs --> graph_out;
 ```
 
-### Modules
-
 ```yaml
 Package name: eth_blobs
-Version: v0.2.0
+Version: v0.6.2
+Doc: This substreams package lets you stream Consensus Layer EIP-4844 blobs with attached meta data.
+
+    Among the supported chains are:
+    - mainnet-cl: eth-cl.substreams.pinax.network:443
+    - goerli-cl: goerli-cl.substreams.pinax.network:443
+    - sepolia-cl: sepolia-cl.substreams.pinax.network:443
+    - holesky-cl: holesky-cl.substreams.pinax.network:443
+    - gnosis-cl: gnosis-cl.substreams.pinax.network:443
+    - chiado-cl: chiado-cl.substreams.pinax.network:443
+
+Image: [embedded image: 49353 bytes]
 Modules:
 ----
 Name: map_blobs
 Initial block: 0
 Kind: map
 Input: source: sf.beacon.type.v1.Block
-Output Type: proto:eth.blobs.v1.Blobs
-Hash: ca9133543d1628df5681107b2ea68c4ade81cd8f
+Output Type: proto:pinax.ethereum.blobs.v1.Slot
+Hash: a110dc3d832afd7e85a0ff4a2d935b72b215d893
+
+Name: kv_out
+Initial block: 0
+Kind: map
+Input: map: map_blobs
+Output Type: proto:sf.substreams.sink.kv.v1.KVOperations
+Hash: 800435f53200217af6c523f95b90e10db3ac9517
+
+Name: graph_out
+Initial block: 0
+Kind: map
+Input: map: map_blobs
+Output Type: proto:sf.substreams.entity.v1.EntityChanges
+Hash: 46ef992415d463ec45bcbf6fe34b716b451b05ae
+
+Network: goerli-cl
+
+Sink config:
+----
+type: sf.substreams.sink.kv.v1.GenericService
+configs:
+- sink_config: <nil>
 ```

substreams/substreams.yaml

@@ -23,7 +23,7 @@ protobuf:
   files:
     - pinax/ethereum/blobs/v1/blobs.proto
   importPaths:
-    - ./proto
+    - ../proto
 
 binaries:
   default: