Mark posting group lazy if it has a lot of keys #7961
Conversation
Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: Ben Ye <[email protected]>
yeya24
changed the title
Signed-off-by: Ben Ye <[email protected]>
cmd/thanos/store.go
Outdated
| @@ -204,6 +205,9 @@ func (sc *storeConfig) registerFlag(cmd extkingpin.FlagClause) { | |||
| cmd.Flag("store.enable-lazy-expanded-postings", "If true, Store Gateway will estimate postings size and try to lazily expand postings if it downloads less data than expanding all postings."). | |||
| Default("false").BoolVar(&sc.lazyExpandedPostingsEnabled) | |||
|
|
|||
| cmd.Flag("store.posting-group-max-keys", "Mark posting group as lazy if it fetches more keys than the configured number. Only valid if lazy expanded posting is enabled. 0 disables the limit."). | |||
There was a problem hiding this comment.
How could the user know what the appropriate number here should be? Could we calculate & use some pre-defined threshold?
There was a problem hiding this comment.
That's a good question... For us, we actually added some additional logs in our fork to log cardinalities for those queries that triggered lazy posting optimization. So that we have insights about the size. But logging those information in Thanos might be too expensive.
Do you think it would help if we introduce a native histogram to track posting group total posting sizes and number of keys?
I thought about making it a percentage value based on your block size. For example, 1% of your number of series, or 10% of your total label pairs but I couldn't find a good way to define that. Percentage will make the threshold very small for small blocks even though they don't cause any impact.
There was a problem hiding this comment.
@GiedriusS One idea we had is to mark lazy if number of keys is much larger than the max possible series for the query (or smallest cardinality posting group).
number of keys / max_series_to_match > R
For a query, if the matcher with the smallest posting could match 1000 series. Then it can have at most 1000 different values for a single label. If the current posting group matches 100K keys it means we match at most 1K values and throw away 99K keys. Do you think this is a good metric? We can start with threshold like 100 and tune it accordingly
Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: Ben Ye <[email protected]>
| @@ -250,6 +250,14 @@ Flags: | |||
| The maximum series allowed for a single Series | |||
| request. The Series call fails if this limit is | |||
| exceeded. 0 means no limit. | |||
| --store.posting-group-max-key-series-ratio=100 | |||
There was a problem hiding this comment.
Maybe you could also add that one should look at thanos_bucket_store_lazy_expanded_posting_groups_total if they want to understand whether this value needs to be changed?
There was a problem hiding this comment.
Addressed in latest commit!
Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: Ben Ye <[email protected]>
|
PTAL @GiedriusS, thanks for the review! |
* mark posting group lazy if it has a lot of add keys Signed-off-by: Ben Ye <[email protected]> * update docs Signed-off-by: Ben Ye <[email protected]> * rename labels Signed-off-by: Ben Ye <[email protected]> * changelog Signed-off-by: Ben Ye <[email protected]> * change to use max key series ratio Signed-off-by: Ben Ye <[email protected]> * update docs Signed-off-by: Ben Ye <[email protected]> * mention metrics Signed-off-by: Ben Ye <[email protected]> * update docs Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: Ben Ye [email protected]
Fixes #7956
Changes
store.posting-group-max-keysto mark a posting group as lazy if it tries to fetch too many keys. 0 disables the limit. This is only effective if lazy expanded posting is enabled.thanos_bucket_store_lazy_expanded_posting_groups_totalto track total number of posting groups that are marked to lazy for different reasonsVerification