From 2b68bee5f424a4545762fde8f3dd8ceff7651d59 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Wojciech=20Tyczy=C5=84ski?= Date: Fri, 15 Jul 2022 08:10:48 +0200 Subject: [PATCH] Add SIG Scalability FAQ about object sizes --- sig-scalability/configs-and-limits/faq.md | 36 +++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/sig-scalability/configs-and-limits/faq.md b/sig-scalability/configs-and-limits/faq.md index e88f253a3e3..6b5c4ac0b54 100644 --- a/sig-scalability/configs-and-limits/faq.md +++ b/sig-scalability/configs-and-limits/faq.md @@ -27,6 +27,42 @@ that is set up the same way may have troubles handling small tens of QPS if the requests are fairly expensive. +### What is the ideal size for API objects we should target? + +Technically, the only hard limit that we have is the one of 1.5MB for +the size of individual objects. That said, approaching that limit is +definitely not recommended unless absolutely necessary. + +In typical usecases, huge majority of objects doesn't exceed ~20KB of +size and this is the usecase that is best tested and many optimizations +assume (which are done based on existing tests) silently assume that. + +If we look into individual objects larger than 20kB, significant majority +of cases that we've seen were representing a single pattern of grouping +multiple "subobjects" into a single object. The best example of that +from the core Kubernetes is `Endpoints` API, which is effectively grouping +all endpoints backing a given Kubernetes Service into a single object. +Those kind of APIs proved to be problematic for multiple different reasons, +including: +- the objects become large and even small change (of a single subobject) + becomes expensive from the system perspective +- they become a contention point if different agents are updating different + subobjects +- they become wasteful as we are able to get/watch only full objects and + many agents may not need information about all subobjects +As a result, this pattern should really be avoided. +In case of `Endpoints` API, we introduce the `EndpointSlice` API and if +singular objects are problematic for your usecase, this is the pattern +you should explore. + +So from scalability/performance perspective, the rule of thumb can be +summarized as: +- try to keep your object size below ~20kB +- if really needed, you can get to 100kB if it's not changing frequently +- if you can't keep your object size below 100kB, reach out to SIG + Scalability and discuss the usecase to see how we can make it performant + + ### How do you setup clusters for scalability testing? We are testing Kubernetes on two levels of scale: 100 nodes and 5000 nodes.