Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: reorganize storage modules #1880

Merged
merged 3 commits into from
Nov 25, 2024
Merged

Conversation

marcospb19-cw
Copy link
Contributor

@marcospb19-cw marcospb19-cw commented Nov 24, 2024

User description

leading to the creation of a trait for StratusStorage with multiple implementations, I reorganized how the structure of src/eth/storage modules


PR Type

Enhancement


Description

  • Reorganized and enhanced the storage modules, introducing new implementations for RocksDB, in-memory, and Redis storages.
  • Implemented versioning for Column Family values in RocksDB storage.
  • Replaced StoragePointInTime with a new PointInTime enum, updating all related components.
  • Refactored the ExternalRpc trait and its implementations, including the PostgreSQL-based external RPC.
  • Updated the StratusStorage implementation to use the new storage traits and PointInTime enum.
  • Implemented new type definitions for RocksDB storage, including conversions between domain types and RocksDB types.
  • Updated the Executor and EvmInput to use the new PointInTime enum.
  • Refactored and improved the configuration options for RocksDB.
  • Updated related binaries and tests to use the new storage implementations and traits.

Changes walkthrough 📝

Relevant files
Enhancement
14 files
rocks_cf.rs
Implement RocksDB Column Family handling                                 

src/eth/storage/permanent/rocks/rocks_cf.rs

  • Implemented RocksCfRef struct for handling RocksDB Column Families
  • Added methods for CRUD operations on Column Families
  • Implemented serialization and deserialization for keys and values
  • Added metrics export functionality
  • +423/-1 
    cf_versions.rs
    Implement Column Family versioning                                             

    src/eth/storage/permanent/rocks/cf_versions.rs

  • Implemented versioning for Column Family values
  • Added macros for implementing single-version CF values
  • Implemented conversion traits for CF values
  • Added tests for bincode serialization/deserialization
  • +289/-1 
    inmemory.rs
    Implement in-memory temporary storage                                       

    src/eth/storage/temporary/inmemory.rs

  • Implemented InMemoryTemporaryStorage for temporary storage
  • Added methods for managing pending blocks and transactions
  • Implemented conflict checking for transactions
  • +333/-1 
    inmemory.rs
    Implement in-memory permanent storage                                       

    src/eth/storage/permanent/inmemory.rs

  • Implemented InMemoryPermanentStorage for permanent storage
  • Added methods for managing accounts, slots, and blocks
  • Implemented point-in-time querying
  • +65/-7   
    postgres.rs
    Refactor PostgresExternalRpc implementation                           

    src/eth/external_rpc/postgres.rs

  • Renamed PostgresExternalRpcStorage to PostgresExternalRpc
  • Updated method signatures to match new trait definition
  • Implemented methods for reading and writing external data
  • +15/-19 
    stratus_storage.rs
    Update StratusStorage implementation                                         

    src/eth/storage/stratus_storage.rs

  • Updated StratusStorage to use new storage traits
  • Replaced StoragePointInTime with PointInTime
  • Added new_test method for creating test instances
  • +21/-10 
    evm_input.rs
    Update EvmInput to use PointInTime                                             

    src/eth/executor/evm_input.rs

  • Replaced StoragePointInTime with PointInTime
  • Updated method signatures to use new PointInTime enum
  • +9/-14   
    block.rs
    Implement BlockRocksdb for RocksDB storage                             

    src/eth/storage/permanent/rocks/types/block.rs

  • Implemented BlockRocksdb struct for RocksDB storage
  • Added conversion methods between Block and BlockRocksdb
  • +76/-1   
    rocks_state.rs
    Update RocksStorageState to use PointInTime                           

    src/eth/storage/permanent/rocks/rocks_state.rs

  • Updated method signatures to use PointInTime instead of
    StoragePointInTime
  • Implemented methods for reading accounts and slots with PointInTime
  • +8/-8     
    mod.rs
    Add RocksDB type definitions and tests                                     

    src/eth/storage/permanent/rocks/types/mod.rs

  • Added new type definitions for RocksDB storage
  • Implemented tests for bincode serialization/deserialization
  • +86/-1   
    executor.rs
    Update Executor to use PointInTime                                             

    src/eth/executor/executor.rs

  • Replaced StoragePointInTime with PointInTime
  • Updated method signatures to use new PointInTime enum
  • +6/-6     
    transaction_input.rs
    Implement TransactionInputRocksdb for RocksDB storage       

    src/eth/storage/permanent/rocks/types/transaction_input.rs

  • Implemented TransactionInputRocksdb struct for RocksDB storage
  • Added conversion methods between TransactionInput and
    TransactionInputRocksdb
  • +74/-1   
    mod.rs
    Refactor ExternalRpc trait and implementations                     

    src/eth/external_rpc/mod.rs

  • Renamed ExternalRpcStorage trait to ExternalRpc
  • Updated related type and function names
  • Moved PostgresExternalRpc implementation to this module
  • +15/-12 
    rpc_downloader.rs
    Update rpc_downloader to use new ExternalRpc trait             

    src/bin/rpc_downloader.rs

  • Updated import paths for ExternalRpc trait
  • Replaced ExternalRpcStorage with ExternalRpc in function signatures
  • +4/-9     
    Configuration changes
    1 files
    rocks_config.rs
    Implement RocksDB configuration options                                   

    src/eth/storage/permanent/rocks/rocks_config.rs

  • Implemented RocksDB configuration options
  • Added different configuration profiles (LargeSSTFiles, FastWriteSST,
    Default)
  • Implemented cache settings
  • +169/-1 
    Additional files (token-limit)
    22 files
    point_in_time.rs
    ...                                                                                                           

    src/eth/primitives/point_in_time.rs

    ...

    +3/-3     
    address.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/types/address.rs

    ...

    +25/-1   
    index.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/types/index.rs

    ...

    +25/-1   
    unix_time.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/types/unix_time.rs

    ...

    +22/-1   
    gas.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/types/gas.rs

    ...

    +23/-1   
    size.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/types/size.rs

    ...

    +23/-1   
    miner_nonce.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/types/miner_nonce.rs

    ...

    +18/-1   
    chain_id.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/types/chain_id.rs

    ...

    +19/-1   
    difficulty.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/types/difficulty.rs

    ...

    +20/-1   
    hash.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/types/hash.rs

    ...

    +19/-1   
    mod.rs
    ...                                                                                                           

    src/eth/primitives/mod.rs

    ...

    +2/-0     
    mod.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/mod.rs

    ...

    +5/-2     
    rocks_db.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/rocks_db.rs

    ...

    +2/-2     
    mod.rs
    ...                                                                                                           

    src/eth/mod.rs

    ...

    +1/-0     
    rocks_batch_writer.rs
    ...                                                                                                           

    src/eth/storage/permanent/rocks/rocks_batch_writer.rs

    ...

    +2/-1     
    select_external_blocks_and_receipts_in_range.sql
    ...                                                                                                           

    src/eth/external_rpc/sql/select_external_blocks_and_receipts_in_range.sql

    ...

    +7/-1     
    select_external_blocks_in_range.sql
    ...                                                                                                           

    src/eth/external_rpc/sql/select_external_blocks_in_range.sql

    ...

    +6/-1     
    insert_external_block_and_receipts.sql
    ...                                                                                                           

    src/eth/external_rpc/sql/insert_external_block_and_receipts.sql

    ...

    +3/-1     
    insert_external_balance.sql
    ...                                                                                                           

    src/eth/external_rpc/sql/insert_external_balance.sql

    ...

    +4/-1     
    select_max_external_block_in_range.sql
    ...                                                                                                           

    src/eth/external_rpc/sql/select_max_external_block_in_range.sql

    ...

    +4/-1     
    select_external_balances.sql
    ...                                                                                                           

    src/eth/external_rpc/sql/select_external_balances.sql

    ...

    +5/-1     
    select_max_imported_block.sql
    ...                                                                                                           

    src/eth/external_rpc/sql/select_max_imported_block.sql

    ...

    +3/-1     

    💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

    Copy link

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    ⏱️ Estimated effort to review: 5 🔵🔵🔵🔵🔵
    🧪 No relevant tests
    🔒 No security concerns identified
    ⚡ Recommended focus areas for review

    Performance Concern
    The clear method uses a delete range followed by individual deletes, which may be inefficient for large datasets. Consider using a more efficient bulk delete operation if available.

    Logging Improvement
    The export_metrics function uses unwrap_or_default() for several metrics. Consider logging errors or warnings when these operations fail to retrieve the expected values.

    Potential Memory Leak
    The InMemoryTemporaryStorage keeps a history of states in a NonEmpty vector. There should be a mechanism to limit the size of this history to prevent unbounded growth.

    Configuration Complexity
    The DbConfig enum and its implementation contain complex configuration options. Consider breaking this down into smaller, more manageable components or providing higher-level abstractions for common use cases.

    Error Handling
    The read_max_block_number_in_range method uses unwrap() which can cause panics. Consider using proper error handling and propagation instead.

    Copy link

    PR Code Suggestions ✨

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Score
    Possible issue
    Handle potential absence of sender account when processing failed external transactions

    Ensure that the sender variable is properly initialized before using it in the
    from_failed_external_transaction method. Consider moving the sender initialization
    outside the match statement to ensure it's always defined.

    src/eth/executor/executor.rs [331-335]

     false => {
    -    let sender = self.storage.read_account(receipt.from.into(), PointInTime::Pending)?;
    +    let sender = self.storage.read_account(receipt.from.into(), PointInTime::Pending)?
    +        .ok_or_else(|| anyhow::anyhow!("Sender account not found"))?;
         let execution = EvmExecution::from_failed_external_transaction(sender, &receipt, block_timestamp)?;
         let evm_result = EvmExecutionResult {
             execution,
    Suggestion importance[1-10]: 8

    Why: This suggestion addresses a potential issue where the sender account might not exist, which could lead to runtime errors. The improved code handles this case explicitly, enhancing the robustness of the transaction processing.

    8
    Improve error handling for potential data inconsistencies in the database

    Implement error handling for the case where rocks_address or rocks_index don't match
    the expected values, which could indicate data corruption or inconsistency.

    src/eth/storage/permanent/rocks/rocks_state.rs [328-340]

     if let Some(((rocks_address, rocks_index, _), value)) = self
         .account_slots_history
         .iter_from(iterator_start, rocksdb::Direction::Reverse)?
         .next()
     {
         if rocks_address == address.into() && rocks_index == index.into() {
             Ok(Some(Slot {
                 index: index.into(),
                 value: value.into_inner().into(),
             }))
         } else {
    -        Ok(None)
    +        Err(anyhow::anyhow!("Data inconsistency: Unexpected address or index in account_slots_history"))
         }
     } else {
         Ok(None)
     }
    Suggestion importance[1-10]: 8

    Why: This suggestion addresses a potential security and data integrity issue. By implementing proper error handling for unexpected data in the database, it improves the robustness of the code and helps detect data corruption or inconsistencies early. This is crucial for maintaining the reliability of the storage system.

    8
    Improve error handling for retrieving historical values in storage operations

    Implement proper error handling for the case when no historical values are found for
    a given block number in the get_at_block method. This could prevent potential panics
    or unexpected behavior.

    src/eth/storage/permanent/inmemory.rs [334-336]

     pub fn get_at_block(&self, block_number: BlockNumber) -> Option<T> {
    -    self.0.iter().take_while(|x| x.block_number <= block_number).map(|x| &x.value).last().cloned()
    +    self.0.iter()
    +        .take_while(|x| x.block_number <= block_number)
    +        .last()
    +        .map(|x| x.value.clone())
     }
    Suggestion importance[1-10]: 6

    Why: The suggestion improves the clarity and efficiency of the code by simplifying the chain of operations and making the intent clearer. While it doesn't fundamentally change the behavior, it does make the code more readable and potentially easier to maintain.

    6
    General
    Refactor the point-in-time logic to reduce duplication and improve maintainability

    Replace the manual match statements with a more generic approach using an enum
    method, reducing code duplication and improving maintainability.

    src/eth/storage/permanent/rocks/rocks_state.rs [312-340]

    -match point_in_time {
    -    PointInTime::Mined | PointInTime::Pending => {
    -        let query_params = (address.into(), index.into());
    -
    -        let Some(account_slot_value) = self.account_slots.get(&query_params)? else {
    -            return Ok(None);
    -        };
    -
    -        Ok(Some(Slot {
    -            index: index.into(),
    -            value: account_slot_value.into_inner().into(),
    -        }))
    -    }
    -    PointInTime::MinedPast(number) => {
    -        let iterator_start = (address.into(), (index).into(), number.into());
    -
    -        if let Some(((rocks_address, rocks_index, _), value)) = self
    -            .account_slots_history
    -            .iter_from(iterator_start, rocksdb::Direction::Reverse)?
    -            .next()
    -        {
    -            // ...
    -        } else {
    -            Ok(None)
    +impl PointInTime {
    +    fn read_slot<F, G>(&self, current: F, historical: G) -> Result<Option<Slot>>
    +    where
    +        F: FnOnce() -> Result<Option<Slot>>,
    +        G: FnOnce(BlockNumber) -> Result<Option<Slot>>,
    +    {
    +        match self {
    +            PointInTime::Mined | PointInTime::Pending => current(),
    +            PointInTime::MinedPast(number) => historical(*number),
             }
         }
     }
     
    +point_in_time.read_slot(
    +    || {
    +        let query_params = (address.into(), index.into());
    +        self.account_slots.get(&query_params)?.map(|account_slot_value| {
    +            Slot {
    +                index: index.into(),
    +                value: account_slot_value.into_inner().into(),
    +            }
    +        }).transpose()
    +    },
    +    |number| {
    +        let iterator_start = (address.into(), index.into(), number.into());
    +        self.account_slots_history
    +            .iter_from(iterator_start, rocksdb::Direction::Reverse)?
    +            .next()
    +            .map(|((rocks_address, rocks_index, _), value)| {
    +                // ... (existing logic)
    +            })
    +            .transpose()
    +    }
    +)
    +
    Suggestion importance[1-10]: 7

    Why: This suggestion offers a more generic and maintainable approach to handling different point-in-time scenarios. It reduces code duplication and improves readability by encapsulating the logic in an enum method. This change could lead to easier maintenance and extension of the codebase in the future.

    7

    @marcospb19-cw marcospb19-cw merged commit 0e6f444 into main Nov 25, 2024
    34 checks passed
    @marcospb19-cw marcospb19-cw deleted the reorganize-storage-modules branch November 25, 2024 12:07
    @gabriel-aranha-cw
    Copy link
    Contributor

    Final benchmark:
    RPS Stats: Max: 1151.00, Min: 487.00, Avg: 900.81, StdDev: 56.64
    TPS Stats: Max: 1055.00, Min: 756.00, Avg: 874.12, StdDev: 65.56
    Plot: View Plot

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    None yet
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    2 participants