Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export stats #2326

Merged
merged 9 commits into from
Apr 11, 2018
Merged

Export stats #2326

merged 9 commits into from
Apr 11, 2018

Conversation

theoreticalbts
Copy link
Contributor

This code implements two plugins:

  • block_data_export_plugin allows export of per-block data; it effectively implements Routing per-block data #2244.
  • stats_export_plugin uses the interface provided by block_data_export_plugin to dump GPO and per-user bytes per block. The intent is that this data will be consumed by a Python script for further processing.

The data path implemented by block_data_export_plugin will be extremely useful going forward, especially when combined with a UNIX FIFO connected to some external script.

I also implemented a simple thread pool to allow all available processor cores to be utilized for the JSON conversion of exported data objects. Unfortunately Boost's thread pools aren't suitable, because (1) they're marked experimental and may be subject to change in future Boost versions, and (2) there's no way to set the thread stack size, which we need because FC JSON processing uses stack very heavily.

When processing the main blockchain, approximately 40 GB of stats are produced. To minimize disk usage, you can do streaming compression of the output JSON by simply using a UNIX FIFO connected to pbzip2 or similar.

stats_export_plugin_impl( stats_export_plugin& _plugin ) :
_db( appbase::app().get_plugin< chain_plugin >().db() ),
_self( _plugin ),
_export_plugin( appbase::app().get_plugin< block_data_export_plugin >() )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because you are using the block_data_export_plugin as a dependency, you should specify it in APPBASE_PLUGIN_REQUIRES

@mvandeberg
Copy link
Contributor

mvandeberg commented Apr 11, 2018

I only reviewed commits e548eae and 9354983 because the others are included in #2325

@mvandeberg mvandeberg merged commit eef5327 into develop Apr 11, 2018
@mvandeberg mvandeberg deleted the 2018-04-11-stats-export branch October 23, 2018 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants