Skip to content

Commit

Permalink
RocksDB big refactor - use single DB with Column Families
Browse files Browse the repository at this point in the history
previously, we used 7 synchronized RocksDB Databases, but with the lack
of transaction support, we started facing synchronization problems

in this PR we are switching away from multiple DBs and using a single DB
with multiple column families (CFs), beyond solving the transaction
problem, we also see a performance improvement for batch writes

a Column Family (CF) can be thought as a table in PostgreSQL, where you
can store data under different schemas, and have the same key pointing
to two (or more) different pieces of data

adding a prefix to the database path still works, but now it's for the
single DB folder instead of the 7 folders like before

---

squashed messages:
- unify 7 DBs into one
- optimize `save_block`
- add flag to disable backups
- stop bundling "current_block" and other data into the DBs
- remove iterator support for special keys and indexes (bundled info)
- make iterator exit on deserialization errors
- change size of table caches
- e2e: verify block number after resetting chain (to check `reset_at`)
- simplify db creation for tests
- refac stuff: docs, logs, rename, move code, optimize, fixes, etc
- simplify `importer-offline` main
  • Loading branch information
marcospb19-cw committed Jun 17, 2024
1 parent 0420ec8 commit 165ff10
Show file tree
Hide file tree
Showing 20 changed files with 1,223 additions and 1,422 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ data/**/*.db
data/**/*.json
data/**/*.txt
data/**/*.rocksdb
data/rocksdb/

# E2E (Stratus)
e2e/artifacts
Expand Down
Loading

0 comments on commit 165ff10

Please sign in to comment.