Skip to content

Commit

Permalink
Update version map unsync tolerance to 200ms
Browse files Browse the repository at this point in the history
Before this change version_map's reload_interval was equal to the
unsync_tolerance. This means that we would ~always miss the cache.

The check for outdated cache is as follows:
- When reading into cache set `last_reload_time = now -
  unsync_tolerance`
- When checking if cache is outdated we check whether `now -
  last_reload_time > reload_interval

This change updates the unsync tolerance to 200ms. This will mean the
cache is valid for 1.8s

Tests uncovered that we were not flushing the version_map when clearing
the storage. This change fixes this as well.
  • Loading branch information
IvoDD committed Mar 27, 2024
1 parent fb17771 commit c0e5f35
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 2 deletions.
4 changes: 3 additions & 1 deletion cpp/arcticdb/util/constants.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@ constexpr auto string_nan = std::numeric_limits<position_t>::max() - 1;

constexpr auto NaT = std::numeric_limits<timestamp>::min();

static constexpr decltype(timestamp(0) - timestamp(0)) ONE_SECOND = 1'000'000'000;
static constexpr decltype(timestamp(0) - timestamp(0)) ONE_MILLISECOND = 1'000'000;

static constexpr decltype(timestamp(0) - timestamp(0)) ONE_SECOND = 1'000 * ONE_MILLISECOND;

static constexpr decltype(timestamp(0) - timestamp(0)) ONE_MINUTE = 60 * ONE_SECOND;

Expand Down
2 changes: 1 addition & 1 deletion cpp/arcticdb/version/version_map.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ class VersionMapImpl {
*/
using MapType = std::map<StreamId, std::shared_ptr<VersionMapEntry>>;

static constexpr uint64_t DEFAULT_CLOCK_UNSYNC_TOLERANCE = ONE_SECOND * 2;
static constexpr uint64_t DEFAULT_CLOCK_UNSYNC_TOLERANCE = ONE_MILLISECOND * 200;
static constexpr uint64_t DEFAULT_RELOAD_INTERVAL = ONE_SECOND * 2;
MapType map_;
bool validate_ = false;
Expand Down
1 change: 1 addition & 0 deletions cpp/arcticdb/version/version_store_api.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1125,6 +1125,7 @@ std::vector<SliceAndKey> PythonVersionStore::list_incompletes(const StreamId& st
}

void PythonVersionStore::clear(const bool continue_on_error) {
version_map()->flush();
if (store()->fast_delete()) {
// Most storage backends have a fast deletion method for a db/collection equivalent, eg. drop() for mongo and
// lmdb and iterating each key is always going to be suboptimal.
Expand Down

0 comments on commit c0e5f35

Please sign in to comment.