Evict wave usage metrics after 120 days

[munishchouhan]

This PR will do the following:

1. add functionality to scan and delete keys with specific dates.
2. add corn to run every day and delete keys with current date - 120

[pditommaso]

I think this can be implemented more easily by specifying the value ttl in the redis map

wave/src/main/groovy/io/seqera/wave/service/counter/AbstractCounterStore.groovy

Line 42 in b2790ee

provider.inc(getPrefix(), key, value)

See this pattern

wave/src/main/groovy/io/seqera/wave/service/cache/AbstractCacheStore.groovy

Lines 70 to 82 in b2790ee

	@Override
	void put(String key, V value, Duration ttl) {
	delegate.put(key0(key), serialize(value), ttl)
	}
	@Override
	boolean putIfAbsent(String key, V value, Duration ttl) {
	delegate.putIfAbsent(key0(key), serialize(value), ttl)
	}
	boolean putIfAbsent(String key, V value) {
	delegate.putIfAbsent(key0(key), serialize(value), getDuration())
	}

[munishchouhan]

The problem is we are using hash in this case and TTL is normally applied on the key and not on the fields, I will dig further to see if there is any way to achieve this for a filed in a key

[munishchouhan]

here is a thread saying its not possible
https://stackoverflow.com/questions/16545321/how-to-expire-the-hset-child-key-in-redis

[munishchouhan]

There is this PR that implemented expiration on hashes
redis/redis#13172
its available on version 7.2

[pditommaso]

What's preventing expiring the key ?

wave/src/main/groovy/io/seqera/wave/service/metric/impl/MetricsServiceImpl.groovy

Lines 106 to 107 in b2790ee

	def org = getOrg(email)
	def key = getKey(prefix, LocalDate.now().format(DATE_FORMATTER), null)

[munishchouhan]

This is the key in our case

wave/src/main/groovy/io/seqera/wave/service/metric/MetricsCounterStore.groovy

Lines 39 to 41 in 909bda1

	protected String getPrefix() {
	return 'metrics/v1'
	}

This is field:

wave/src/main/groovy/io/seqera/wave/service/metric/impl/MetricsServiceImpl.groovy

Lines 106 to 107 in b2790ee

	def org = getOrg(email)
	def key = getKey(prefix, LocalDate.now().format(DATE_FORMATTER), null)

see here:

wave/src/main/groovy/io/seqera/wave/service/counter/AbstractCounterStore.groovy

Line 42 in 909bda1

provider.inc(getPrefix(), key, value)

[pditommaso]

I think solution 1 here, would make the trick

[munishchouhan]

I think solution 1 here, would make the trick

I caanot access this

[pditommaso]

TLDR;

Set the TTL on the entire hash map key:

HINCRBY myhash field 1
EXPIRE myhash 60  # Set TTL of 60 seconds on the entire hash map

Essentially an inc followed by an expire operation

[pditommaso]

Actually this should be done only for *day* metrics, therefore the ones having /d/<date> in the key

[munishchouhan]

There is this method for expiry. but it still only required key and not field

[pditommaso]

What's wrong with that?

[munishchouhan]

What's wrong with that?

ok you are correct, if i use expiry it does work
I will develop it using this

[munishchouhan]

@pditommaso I have added two test failing and success to show why expire with key wont work, because we need to evict the field in a key and not the whole key, i am using 1 second as ttl

failing test:

wave/src/test/groovy/io/seqera/wave/service/counter/impl/RedisCounterProviderTest.groovy

Line 87 in abd522a

def 'failing: should expire the hash'(){

success test: where everything will be null

wave/src/test/groovy/io/seqera/wave/service/counter/impl/RedisCounterProviderTest.groovy

Line 100 in abd522a

def 'successful: should expire the hash'(){

[munishchouhan]

the problem is we are using redis hash, where we have to fields, key and field
expirying key does not help our use case

[pditommaso]

Considering this is not critical. I'd say to park until Redis 7.2 is available, so that we can use expired on the field

[munishchouhan]

Hash expiration functionality will be available in jedis 5.2.0
https://github.com/redis/jedis/releases/tag/v5.2.0-beta4
and required redis 7.4

[munishchouhan]

Hash expiration functionality has been added, but the jedis dependency is still in beta, once the final version is released. I will make this pr ready for review

[munishchouhan]

Redis v 5.2.0 has been released
https://github.com/redis/jedis/releases/tag/v5.2.0

Now this PR needs Redis 7.4 or later