Research groupcache as alternative cache client to memcached

[bwplotka]

I think something which would easy up Thanos deployment with cache is https://github.com/golang/groupcache This is project done my Memcached author and it's replicated in mem cache. The only tradeoff is gossip between replicas which is not that basd to setup.

To research and benchmark (:

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[kakkoyun]

I have started experimenting with groupcache.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

Needed!

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[GiedriusS]

I'm working on this.

[GiedriusS]

My semi-working branch: https://github.com/GiedriusS/thanos/tree/experiment/groupcache_caching_bucket.

Some thoughts after working on this:

There is no way to remove a group after it has been registered with groupcache without changing groupcache's code itself. This means that if we were to use it at the Thanos Query level where we'd have to dynamically register/unregister groups (lists of StoreAPI nodes that they can see), groupcache's would need to be improved.

Another huge limitation is that groupcache doesn't serialize multiple requests into one HTTP request. When opening a bigger dashboard, lots of requests are made and it can easily lead to:

2021/06/23 07:15:12 http: Accept error: accept tcp [::]:9091: accept4: too many open files; retrying in 10ms
2021/06/23 07:15:12 http: Accept error: accept tcp [::]:9091: accept4: too many open files; retrying in 20ms

This is also quite ineffective. Again, groupcache's code would need to be improved to be able to ask for multiple keys at once. This is probably the biggest thing hindering further progress.

What's more, Thanos code assumes that all caches follow the same load/store model whereas groupcache loads everything during the load() phase. This is a bit awkward for us and we can work around this easily in almost all cases without changing our own caching interface besides the iter:... keys where the value is encoded before being stored in the cache. This means that our groupcache layer would need to know about the codec as well and in this way adding a dependency on the upper layer, making things a bit less clear. Hence, some smarter abstractions need to be made.

Finally, vanilla groupcache doesn't have support for TTLs but that can be worked around by using some kind of fork such as https://github.com/mailgun/groupcache.

[GiedriusS]

Yep, and for now, I would only focus on the caching bucket. Then we could move on to the other types of cache that we have in Thanos. This is because the latter is more involved since you'll have to inevitably change the internal cache interface a bit.

[kernelpanic77]

Hello @GiedriusS ! I am relatively new to the Thanos project, but would love to get involved and potentially contribute. Any pointers or resources for me to get started with Thanos and understand this issue better ?

Thanks,
Ishan

[akshatdalton]

Hola! Thanos team. I am a newbie here, but I have other Open Source experience. I would love to contribute here along with learning about new technologies. It feels great that my contribution will be useful to many. I love to take on challenges. If this is an active issue I can work on, please do let me know. Or, if there is any other challenge that I need to complete first, I am ready for that. Looking forward to contributing here actively.

Thank You
Akshat
Open Source Enthusiast

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[yeya24]

Still valid

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[GiedriusS]

Already in Thanos. Follow up: #5037.