-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize importer-offline
(batch writing + more parallelism)
#1168
Conversation
PR Reviewer Guide 🔍
|
Failed to generate code suggestions for PR |
183ff0b
to
89f4da0
Compare
09a0b3f
to
b7cf84f
Compare
importer-offline
(batch writing + more parallelism)
b7cf84f
to
9f3eabc
Compare
/// | ||
/// REMOVE THIS: Decrease by one to give room for the pending block. | ||
/// TODO: why `max/2-1` doesn't work? | ||
const STORAGE_WRITER_BATCHES_SIZE: usize = INMEMORY_TEMPORARY_STORAGE_MAX_BLOCKS / 3; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Auto-review requested changes: update these comments and quickly check if it's obvious why it behaves this way.
This commit divides importer-offline's task block-importer into block-executor and block-saver, to separate EVM and storage writing, now both can run in parallel Besides that, this enabled our RocksDB storage to perform block writes in batches, as a consequence, this leaves execution time as the new bottleneck for importer-offline, which hints that we don't need to optimize the storage further to improve this binary.
9f3eabc
to
621c7a2
Compare
Recent smaller changes achieved enough speed improvements in importer-offline, it's unclear now if we need to further improve it or not, closing for now. |
This PR divides
importer-offline
's taskblock-importer
intoblock-executor
andblock-saver
, to separate EVM and storage writing, now both can run in parallelBesides that, this enabled our RocksDB storage to perform block writes in batches, as a consequence, this leaves execution time as the new bottleneck for
importer-offline
, which hints that we don't need to optimize the storage further to improve this binary.