Skip to content

Commit

Permalink
Use aggregated reading for 1D selections. (#374)
Browse files Browse the repository at this point in the history
* Use aggregated reading for 1D selections.

Aggregating scattered reads was implemented in

  * 8366465 adds bulkRead,
  * a7d1453 for 2D selections.

However, it was only used for edge indexes. We measured the performance
on an HDD and SSD for random access patterns with different streak
lengths. We found that for all cases, including streak length of 1,
using bulkRead was preferrable over both `0.1.24` (using
`HighFive::ElementSet`) and `0.1.29` (using `HighFive::HyperSlab`).

We investigated if aggregating multiple pages was beneficial and found
that it was not.

* Use `1* min_gap_size`.
  • Loading branch information
1uc authored Oct 17, 2024
1 parent 588434f commit f9f161f
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 19 deletions.
7 changes: 1 addition & 6 deletions src/hdf5_reader.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,7 @@ class Hdf5PluginRead1DDefault: virtual public Hdf5PluginRead1DInterface<T>
public:
std::vector<T> readSelection(const HighFive::DataSet& dset,
const Selection& selection) const override {
if (selection.ranges().empty()) {
return {};
}

return dset.select(detail::_makeHyperslab(selection.ranges()))
.template read<std::vector<T>>();
return detail::readCanonicalSelection<T>(dset, selection);
}
};

Expand Down
27 changes: 14 additions & 13 deletions src/read_canonical_selection.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,26 @@ namespace bbp {
namespace sonata {
namespace detail {

template <class Range>
HighFive::HyperSlab make_hyperslab(const std::vector<Range>& ranges) {
HighFive::HyperSlab slab;
for (const auto& range : ranges) {
size_t i_begin = std::get<0>(range);
size_t i_end = std::get<1>(range);
slab |= HighFive::RegularHyperSlab({i_begin}, {i_end - i_begin});
}

return slab;
}

template <class T>
std::vector<T> readCanonicalSelection(const HighFive::DataSet& dset, const Selection& selection) {
if (selection.empty()) {
return {};
}

return dset.select(make_hyperslab(selection.ranges())).template read<std::vector<T>>();
constexpr size_t min_gap_size = SONATA_PAGESIZE / sizeof(T);
constexpr size_t max_aggregated_block_size = 1 * min_gap_size;

auto readBlock = [&](auto& buffer, const auto& range) {
size_t i_begin = std::get<0>(range);
size_t i_end = std::get<1>(range);
dset.select({i_begin}, {i_end - i_begin}).read(buffer);
};

return bulk_read::bulkRead<T>([&readBlock](auto& buffer,
const auto& range) { readBlock(buffer, range); },
selection.ranges(),
min_gap_size,
max_aggregated_block_size);
}

template <class T>
Expand Down

0 comments on commit f9f161f

Please sign in to comment.