From f98756d18cb5d38a8800a6ec3ebeab221cb9f215 Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Tue, 17 Oct 2023 15:39:01 +0200 Subject: [PATCH 01/10] document resource parsing with examples --- .../pages/usage-guide/resources.adoc | 155 +++++++++++++++++- 1 file changed, 152 insertions(+), 3 deletions(-) diff --git a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc index e1ecc929..2c0ba8a7 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc @@ -2,7 +2,7 @@ include::home:concepts:stackable_resource_requests.adoc[] -If no resources are configured explicitly, the operator uses the following defaults for `SparkApplication`s: +If no resources are configured explicitly, the operator uses the following defaults for `SparkApplication` resources: [source,yaml] ---- @@ -29,7 +29,7 @@ executor: min: '250m' max: "1" memory: - limit: '4Gi' + limit: '1Gi' ---- For `SparkHistoryServer`s the following defaults are used: @@ -50,4 +50,153 @@ For more details regarding Kubernetes CPU limits see: https://kubernetes.io/docs Spark allocates a default amount of non-heap memory based on the type of job (JVM or non-JVM). This is taken into account when defining memory settings based exclusively on the resource limits, so that the "declared" value is the actual total value (i.e. including memory overhead). This may result in minor deviations from the stated resource value due to rounding differences. -NOTE: It is possible to define Spark resources either directly by setting configuration properties listed under `sparkConf`, or by using resource limits. If both are used, then `sparkConf` properties take precedence. It is recommended for the sake of clarity to use *_either_* one *_or_* the other. +NOTE: It is possible to define Spark resources either directly by setting configuration properties listed under `sparkConf`, or by using resource limits. If both are used, then `sparkConf` properties take precedence. It is recommended for the sake of clarity to use *_either_* one *_or_* the other. See below for examples. + +== Resource examples + +To illustrate resource configuration consider these the use-cases where resources are defined using CRD fields (which are then parsed internally to be passed to Spark as spark.conf settings). + +=== CPU + +CPU/Cores will be rounded up and only applied as a limit (instead of separately as request and limit). The rounding to the next integer value will result in the following: + + +|=== +|CRD |Spark conf + +|1800m +|2 + +|100m +|1 + +|1.5 +|2 + +|2 +|2 +|=== + +As each `SparkApplication` will be used to derive templates for the driver and executor pods (pods which are eventually terminated once the job is complete) it is not so crucial to set a resource request separately: the resource limit will be applied to `spark.conf` settings such that the maximum value is applied for the duration of the application. This means in effect that `config.resources.cpu.min` can be ignored by the user, although the field has not been removed to avoid any possible future breaking changes. + +Additionally, Spark allows for CPU limits to be set for the driver and executor using standard Spark settings (`spark.{driver|executor}.cores}`) as well as Kubernetes-specific ones (`spark.kubernetes.{driver,executor}.limit.cores`). This has been simplified so that the CPU is applied to both of these pairs of settings without making a distinction. + +=== Memory + +Memory values are not rounded but for `spark.{driver|executor}.memory}` are passed to Spark in such as a way that the overheads added by Spark are already implicitly declared. For example, if Spark adds 40% to the declared memory (as it does for Pyspark applications), then this is deducted internally from the declared value _before_ being written to `spark.conf`, so that once the overhead is applied, the effective value is the one defined by the user. An alternative is to do define the spark.conf settings explicitly and then let Spark apply the overheads to those values. Spark requires at least 384MB of memory for the driver and executors, so this will be taken into consideration calculating the final allocations. + +=== Example + +A SparkApplication defines the following resources: + +[source,yaml] +---- + ... + job: + config: + resources: + cpu: + min: 250m # <1> + max: 500m # <2> + memory: + limit: 512Mi # <3> + driver: + config: + resources: + cpu: + min: 200m # <4> + max: 1200m # <5> + memory: + limit: 1024Mi # <6> + executor: + config: + resources: + cpu: + min: 250m # <7> + max: 1000m # <8> + memory: + limit: 1024Mi # <9> + ... +---- + +This will result in the following Pod definitions: + +For the job: + +[source,yaml] +---- +spec: + containers: + - name: spark-submit + resources: + limits: + cpu: 500m # <2> + memory: 512Mi # <3> + requests: + cpu: 250m # <1> + memory: 512Mi # <3> +---- + +For the driver: + +[source,yaml] +---- +spec: + containers: + - name: spark + resources: + limits: + cpu: "2" # <5> + memory: 1Gi # <6> + requests: + cpu: "2" # <5> + memory: 1Gi # <6> +---- + +For each executor: + +[source,yaml] +---- +spec: + containers: + - name: spark + limits: + cpu: "1" # <8> + memory: 1Gi # <9> + requests: + cpu: "1" # <8> + memory: 1Gi # <9> +---- + +<1> CPU request (unchanged as this is the Job pod) +<2> CPU limit (unchanged as this is the Job pod) +<3> Memory is assigned to both request and limit values +<4> CPU request is effectively ignored +<5> CPU limit (also applied to request), rounded up from `1200m` to `2` +<6> Memory after reduction and re-addition of Spark overhead (so the declared value matches what is provisioned) +<7> CPU request is effectively ignored +<8> CPU limit (also applied to request), unchanged after rounding: `1000m` to `1` +<9> Memory after reduction and re-addition of Spark overhead (so the declared value matches what is provisioned) + +The spark.conf values corresponding to the above can be inspected in the job Pod definition: + +[source] +---- + ... + --conf "spark.driver.cores=1" + --conf "spark.driver.memory=640m" + --conf "spark.executor.cores=1" + --conf "spark.executor.memory=640m" + --conf "spark.kubernetes.driver.limit.cores=1" + --conf "spark.kubernetes.driver.limit.memory=1024m" + --conf "spark.kubernetes.driver.request.cores=1" + --conf "spark.kubernetes.driver.request.memory=1024m" + --conf "spark.kubernetes.executor.limit.cores=1" + --conf "spark.kubernetes.executor.limit.memory=1024m" + --conf "spark.kubernetes.executor.request.cores=1" + --conf "spark.kubernetes.executor.request.memory=1024m" + --conf "spark.kubernetes.memoryOverheadFactor=0.0" + ... +---- + +These correspond to the resources listed above, with the exception of `spark.{driver|executor}.memory` where indeed the Spark internal overhead of 384MB has been deducted from 1024MB. \ No newline at end of file From 60514aab92eb0b896bc87b4f031530a9a997c25f Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Tue, 17 Oct 2023 15:43:21 +0200 Subject: [PATCH 02/10] updated changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 80944ad8..ea6f4e19 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ All notable changes to this project will be documented in this file. - [BREAKING] use product image selection instead of version ([#275]). - [BREAKING] refactored application roles to use `CommonConfiguration` structures from the operator framework ([#277]). - Let secret-operator handle certificate conversion ([#286]). +- Extended resource-usage documentation ([#297]). ### Fixed @@ -32,6 +33,7 @@ All notable changes to this project will be documented in this file. [#281]: https://github.com/stackabletech/spark-k8s-operator/pull/281 [#286]: https://github.com/stackabletech/spark-k8s-operator/pull/286 [#288]: https://github.com/stackabletech/spark-k8s-operator/pull/288 +[#297]: https://github.com/stackabletech/spark-k8s-operator/pull/297 ## [23.7.0] - 2023-07-14 From e8babfeb7c014b4d6bb900b45a646089e738e302 Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Tue, 17 Oct 2023 16:09:35 +0200 Subject: [PATCH 03/10] minor changes --- docs/modules/spark-k8s/pages/usage-guide/resources.adoc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc index 2c0ba8a7..dff6f948 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc @@ -79,11 +79,13 @@ CPU/Cores will be rounded up and only applied as a limit (instead of separately As each `SparkApplication` will be used to derive templates for the driver and executor pods (pods which are eventually terminated once the job is complete) it is not so crucial to set a resource request separately: the resource limit will be applied to `spark.conf` settings such that the maximum value is applied for the duration of the application. This means in effect that `config.resources.cpu.min` can be ignored by the user, although the field has not been removed to avoid any possible future breaking changes. -Additionally, Spark allows for CPU limits to be set for the driver and executor using standard Spark settings (`spark.{driver|executor}.cores}`) as well as Kubernetes-specific ones (`spark.kubernetes.{driver,executor}.limit.cores`). This has been simplified so that the CPU is applied to both of these pairs of settings without making a distinction. +Additionally, Spark allows CPU limits to be set for the driver and executor using standard Spark settings (`spark.{driver|executor}.cores}`) as well as Kubernetes-specific ones (`spark.kubernetes.{driver,executor}.limit.cores`). The CRD values for max. CPU is applied to both of these pairs of settings without making a distinction. === Memory -Memory values are not rounded but for `spark.{driver|executor}.memory}` are passed to Spark in such as a way that the overheads added by Spark are already implicitly declared. For example, if Spark adds 40% to the declared memory (as it does for Pyspark applications), then this is deducted internally from the declared value _before_ being written to `spark.conf`, so that once the overhead is applied, the effective value is the one defined by the user. An alternative is to do define the spark.conf settings explicitly and then let Spark apply the overheads to those values. Spark requires at least 384MB of memory for the driver and executors, so this will be taken into consideration calculating the final allocations. +Memory values are not rounded as is the case with CPU. Values for `spark.{driver|executor}.memory}` are passed to Spark in such as a way that the overheads added by Spark are already implicitly declared: this overhead will be applied using a factor of 0.1 (JVM jobs) or 0.4 (non-JVM jobs), being not less than 384MB, the minimum overhead applied by Spark. Once the overhead is applied, the effective value is the one defined by the user. This keeps the values transparent: what is defined in the CRD is what is actually provisioned. Unadjusted memory settings are applied for `spark.kubernetes.{driver|executor}.{request|limit}.memory}`. + +An alternative is to do define the spark.conf settings explicitly and then let Spark apply the overheads to those values. === Example From be550cb76d4465c55cc492e374f05e33a5e5540c Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Tue, 17 Oct 2023 16:16:50 +0200 Subject: [PATCH 04/10] more minor changes --- docs/modules/spark-k8s/pages/usage-guide/resources.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc index dff6f948..0e98f3e2 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc @@ -180,7 +180,7 @@ spec: <8> CPU limit (also applied to request), unchanged after rounding: `1000m` to `1` <9> Memory after reduction and re-addition of Spark overhead (so the declared value matches what is provisioned) -The spark.conf values corresponding to the above can be inspected in the job Pod definition: +The spark.conf values derived from the above can be inspected in the job Pod definition: [source] ---- @@ -201,4 +201,4 @@ The spark.conf values corresponding to the above can be inspected in the job Pod ... ---- -These correspond to the resources listed above, with the exception of `spark.{driver|executor}.memory` where indeed the Spark internal overhead of 384MB has been deducted from 1024MB. \ No newline at end of file +These correspond to the resources listed above for the job/driver/executor Pods, with the exception of `spark.{driver|executor}.memory` where indeed the Spark internal overhead of 384MB has been deducted from 1024MB. \ No newline at end of file From 262edbf6c89b161553ecd4c27262a1095a97d56d Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Tue, 17 Oct 2023 17:55:42 +0200 Subject: [PATCH 05/10] removed unused spark conf argument --- .../pages/usage-guide/resources.adoc | 8 +-- rust/crd/src/lib.rs | 51 ------------------- 2 files changed, 2 insertions(+), 57 deletions(-) diff --git a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc index 0e98f3e2..c2d3cfb7 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc @@ -54,7 +54,7 @@ NOTE: It is possible to define Spark resources either directly by setting config == Resource examples -To illustrate resource configuration consider these the use-cases where resources are defined using CRD fields (which are then parsed internally to be passed to Spark as spark.conf settings). +To illustrate resource configuration consider the use-case where resources are defined using CRD fields (which are then parsed internally to be passed to Spark as spark.conf settings). === CPU @@ -83,7 +83,7 @@ Additionally, Spark allows CPU limits to be set for the driver and executor usin === Memory -Memory values are not rounded as is the case with CPU. Values for `spark.{driver|executor}.memory}` are passed to Spark in such as a way that the overheads added by Spark are already implicitly declared: this overhead will be applied using a factor of 0.1 (JVM jobs) or 0.4 (non-JVM jobs), being not less than 384MB, the minimum overhead applied by Spark. Once the overhead is applied, the effective value is the one defined by the user. This keeps the values transparent: what is defined in the CRD is what is actually provisioned. Unadjusted memory settings are applied for `spark.kubernetes.{driver|executor}.{request|limit}.memory}`. +Memory values are not rounded as is the case with CPU. Values for `spark.{driver|executor}.memory}` - this is the amount of memory to use for the driver process (i.e. where SparkContext is initialized) and executor processes respectively - are passed to Spark in such as a way that the overheads added by Spark are already implicitly declared: this overhead will be applied using a factor of 0.1 (JVM jobs) or 0.4 (non-JVM jobs), being not less than 384MB, the minimum overhead applied by Spark. Once the overhead is applied, the effective value is the one defined by the user. This keeps the values transparent: what is defined in the CRD is what is actually provisioned for the process. An alternative is to do define the spark.conf settings explicitly and then let Spark apply the overheads to those values. @@ -190,13 +190,9 @@ The spark.conf values derived from the above can be inspected in the job Pod def --conf "spark.executor.cores=1" --conf "spark.executor.memory=640m" --conf "spark.kubernetes.driver.limit.cores=1" - --conf "spark.kubernetes.driver.limit.memory=1024m" --conf "spark.kubernetes.driver.request.cores=1" - --conf "spark.kubernetes.driver.request.memory=1024m" --conf "spark.kubernetes.executor.limit.cores=1" - --conf "spark.kubernetes.executor.limit.memory=1024m" --conf "spark.kubernetes.executor.request.cores=1" - --conf "spark.kubernetes.executor.request.memory=1024m" --conf "spark.kubernetes.memoryOverheadFactor=0.0" ... ---- diff --git a/rust/crd/src/lib.rs b/rust/crd/src/lib.rs index fd638bb9..bec01421 100644 --- a/rust/crd/src/lib.rs +++ b/rust/crd/src/lib.rs @@ -838,22 +838,6 @@ fn resources_to_driver_props( { let memory = subtract_spark_memory_overhead(for_java, limit)?; props.insert("spark.driver.memory".to_string(), memory); - - let limit_mb = format!( - "{}m", - MemoryQuantity::try_from(limit) - .context(FailedToConvertJavaHeapSnafu { - unit: BinaryMultiple::Mebi.to_java_memory_unit(), - })? - .scale_to(BinaryMultiple::Mebi) - .floor() - .value as u32 - ); - props.insert( - "spark.kubernetes.driver.request.memory".to_string(), - limit_mb.clone(), - ); - props.insert("spark.kubernetes.driver.limit.memory".to_string(), limit_mb); } Ok(()) @@ -890,25 +874,6 @@ fn resources_to_executor_props( { let memory = subtract_spark_memory_overhead(for_java, limit)?; props.insert("spark.executor.memory".to_string(), memory); - - let limit_mb = format!( - "{}m", - MemoryQuantity::try_from(limit) - .context(FailedToConvertJavaHeapSnafu { - unit: BinaryMultiple::Mebi.to_java_memory_unit(), - })? - .scale_to(BinaryMultiple::Mebi) - .floor() - .value as u32 - ); - props.insert( - "spark.kubernetes.executor.request.memory".to_string(), - limit_mb.clone(), - ); - props.insert( - "spark.kubernetes.executor.limit.memory".to_string(), - limit_mb, - ); } Ok(()) @@ -1076,18 +1041,10 @@ mod tests { "spark.kubernetes.driver.limit.cores".to_string(), "1".to_string(), ), - ( - "spark.kubernetes.driver.limit.memory".to_string(), - "128m".to_string(), - ), ( "spark.kubernetes.driver.request.cores".to_string(), "1".to_string(), ), - ( - "spark.kubernetes.driver.request.memory".to_string(), - "128m".to_string(), - ), ] .into_iter() .collect(); @@ -1124,18 +1081,10 @@ mod tests { let expected: BTreeMap = vec![ ("spark.executor.cores".to_string(), "2".to_string()), ("spark.executor.memory".to_string(), "128m".to_string()), // 128 and not 512 because memory overhead is subtracted - ( - "spark.kubernetes.executor.limit.memory".to_string(), - "512m".to_string(), - ), ( "spark.kubernetes.executor.request.cores".to_string(), "2".to_string(), ), - ( - "spark.kubernetes.executor.request.memory".to_string(), - "512m".to_string(), - ), ( "spark.kubernetes.executor.limit.cores".to_string(), "2".to_string(), From db892890eef7c0aa36eecaef95079f5822e7380e Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Wed, 18 Oct 2023 13:22:43 +0200 Subject: [PATCH 06/10] write cpu/min to cpu/request and exclude redundant conf setting --- rust/crd/src/lib.rs | 33 +++++++++++-------- .../kuttl/resources/10-assert.yaml.j2 | 2 +- .../resources/12-deploy-spark-app.yaml.j2 | 2 -- 3 files changed, 21 insertions(+), 16 deletions(-) diff --git a/rust/crd/src/lib.rs b/rust/crd/src/lib.rs index bec01421..f6521662 100644 --- a/rust/crd/src/lib.rs +++ b/rust/crd/src/lib.rs @@ -815,18 +815,21 @@ fn resources_to_driver_props( props: &mut BTreeMap, ) -> Result<(), Error> { if let Resources { - cpu: CpuLimits { max: Some(max), .. }, + cpu: CpuLimits { + min: Some(min), + max: Some(max), + }, .. } = &driver_config.resources { - let cores = cores_from_quantity(max.0.clone())?; + let min_cores = cores_from_quantity(min.0.clone())?; + let max_cores = cores_from_quantity(max.0.clone())?; // will have default value from resources to apply if nothing set specifically - props.insert("spark.driver.cores".to_string(), cores.clone()); props.insert( "spark.kubernetes.driver.request.cores".to_string(), - cores.clone(), + min_cores, ); - props.insert("spark.kubernetes.driver.limit.cores".to_string(), cores); + props.insert("spark.kubernetes.driver.limit.cores".to_string(), max_cores); } if let Resources { @@ -851,18 +854,24 @@ fn resources_to_executor_props( props: &mut BTreeMap, ) -> Result<(), Error> { if let Resources { - cpu: CpuLimits { max: Some(max), .. }, + cpu: CpuLimits { + min: Some(min), + max: Some(max), + }, .. } = &executor_config.resources { - let cores = cores_from_quantity(max.0.clone())?; + let min_cores = cores_from_quantity(min.0.clone())?; + let max_cores = cores_from_quantity(max.0.clone())?; // will have default value from resources to apply if nothing set specifically - props.insert("spark.executor.cores".to_string(), cores.clone()); props.insert( "spark.kubernetes.executor.request.cores".to_string(), - cores.clone(), + min_cores, + ); + props.insert( + "spark.kubernetes.executor.limit.cores".to_string(), + max_cores, ); - props.insert("spark.kubernetes.executor.limit.cores".to_string(), cores); } if let Resources { @@ -1035,7 +1044,6 @@ mod tests { resources_to_driver_props(true, &driver_config, &mut props).expect("blubb"); let expected: BTreeMap = vec![ - ("spark.driver.cores".to_string(), "1".to_string()), ("spark.driver.memory".to_string(), "128m".to_string()), ( "spark.kubernetes.driver.limit.cores".to_string(), @@ -1079,11 +1087,10 @@ mod tests { resources_to_executor_props(true, &executor_config, &mut props).expect("blubb"); let expected: BTreeMap = vec![ - ("spark.executor.cores".to_string(), "2".to_string()), ("spark.executor.memory".to_string(), "128m".to_string()), // 128 and not 512 because memory overhead is subtracted ( "spark.kubernetes.executor.request.cores".to_string(), - "2".to_string(), + "1".to_string(), ), ( "spark.kubernetes.executor.limit.cores".to_string(), diff --git a/tests/templates/kuttl/resources/10-assert.yaml.j2 b/tests/templates/kuttl/resources/10-assert.yaml.j2 index 99d08dbc..9954126d 100644 --- a/tests/templates/kuttl/resources/10-assert.yaml.j2 +++ b/tests/templates/kuttl/resources/10-assert.yaml.j2 @@ -39,7 +39,7 @@ spec: cpu: "2" memory: 1Gi requests: - cpu: "2" + cpu: "1" memory: 1Gi --- apiVersion: v1 diff --git a/tests/templates/kuttl/resources/12-deploy-spark-app.yaml.j2 b/tests/templates/kuttl/resources/12-deploy-spark-app.yaml.j2 index dc48fe9e..e033f926 100644 --- a/tests/templates/kuttl/resources/12-deploy-spark-app.yaml.j2 +++ b/tests/templates/kuttl/resources/12-deploy-spark-app.yaml.j2 @@ -25,12 +25,10 @@ spec: spark.kubernetes.executor.podNamePrefix: "resources-sparkconf" spark.kubernetes.driver.request.cores: "1" spark.kubernetes.driver.limit.cores: "1" - spark.driver.cores: "1" spark.driver.memory: "1g" spark.driver.memoryOverheadFactor: "0.4" spark.kubernetes.executor.request.cores: "1" spark.kubernetes.executor.limit.cores: "2" - spark.executor.cores: "2" spark.executor.memory: "2g" spark.executor.memoryOverheadFactor: "0.4" spark.executor.instances: "1" From b251b252611b7409158ecb823342aa7502d5c8b7 Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Wed, 18 Oct 2023 13:32:48 +0200 Subject: [PATCH 07/10] updated docs to reflect last changes --- .../spark-k8s/pages/usage-guide/resources.adoc | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc index c2d3cfb7..4329c461 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc @@ -151,7 +151,7 @@ spec: cpu: "2" # <5> memory: 1Gi # <6> requests: - cpu: "2" # <5> + cpu: "1" # <4> memory: 1Gi # <6> ---- @@ -163,7 +163,7 @@ spec: containers: - name: spark limits: - cpu: "1" # <8> + cpu: "1" # <7> memory: 1Gi # <9> requests: cpu: "1" # <8> @@ -173,11 +173,11 @@ spec: <1> CPU request (unchanged as this is the Job pod) <2> CPU limit (unchanged as this is the Job pod) <3> Memory is assigned to both request and limit values -<4> CPU request is effectively ignored -<5> CPU limit (also applied to request), rounded up from `1200m` to `2` +<4> CPU request, rounded up from `200m` to `1` +<5> CPU limit, rounded up from `1200m` to `2` <6> Memory after reduction and re-addition of Spark overhead (so the declared value matches what is provisioned) -<7> CPU request is effectively ignored -<8> CPU limit (also applied to request), unchanged after rounding: `1000m` to `1` +<7> CPU request, rounded up from `250m` to `1` +<8> CPU limit, unchanged after rounding: `1000m` to `1` <9> Memory after reduction and re-addition of Spark overhead (so the declared value matches what is provisioned) The spark.conf values derived from the above can be inspected in the job Pod definition: @@ -190,7 +190,7 @@ The spark.conf values derived from the above can be inspected in the job Pod def --conf "spark.executor.cores=1" --conf "spark.executor.memory=640m" --conf "spark.kubernetes.driver.limit.cores=1" - --conf "spark.kubernetes.driver.request.cores=1" + --conf "spark.kubernetes.driver.request.cores=2" --conf "spark.kubernetes.executor.limit.cores=1" --conf "spark.kubernetes.executor.request.cores=1" --conf "spark.kubernetes.memoryOverheadFactor=0.0" From bb124d889fba8de65b80d3d2bd18b02c98dea3eb Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Thu, 19 Oct 2023 11:20:26 +0200 Subject: [PATCH 08/10] Update docs/modules/spark-k8s/pages/usage-guide/resources.adoc Co-authored-by: Sebastian Bernauer --- docs/modules/spark-k8s/pages/usage-guide/resources.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc index 4329c461..7053031a 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc @@ -58,7 +58,7 @@ To illustrate resource configuration consider the use-case where resources are d === CPU -CPU/Cores will be rounded up and only applied as a limit (instead of separately as request and limit). The rounding to the next integer value will result in the following: +CPU request and limit will be rounded up to the next integer value will result in the following: |=== From a5c8e9b8706fe351eec903296b1fdd28e2d4d049 Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Thu, 19 Oct 2023 11:21:27 +0200 Subject: [PATCH 09/10] typo --- docs/modules/spark-k8s/pages/usage-guide/resources.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc index 7053031a..245684bc 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc @@ -58,7 +58,7 @@ To illustrate resource configuration consider the use-case where resources are d === CPU -CPU request and limit will be rounded up to the next integer value will result in the following: +CPU request and limit will be rounded up to the next integer value, resulting in the following: |=== From 1ddccd8870d9320b4d1fc11add5acdc3fb9378be Mon Sep 17 00:00:00 2001 From: Andrew Kenworthy Date: Thu, 19 Oct 2023 11:30:55 +0200 Subject: [PATCH 10/10] corrected text re. cores settings --- docs/modules/spark-k8s/pages/usage-guide/resources.adoc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc index 245684bc..0e218d25 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/resources.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/resources.adoc @@ -77,9 +77,7 @@ CPU request and limit will be rounded up to the next integer value, resulting in |2 |=== -As each `SparkApplication` will be used to derive templates for the driver and executor pods (pods which are eventually terminated once the job is complete) it is not so crucial to set a resource request separately: the resource limit will be applied to `spark.conf` settings such that the maximum value is applied for the duration of the application. This means in effect that `config.resources.cpu.min` can be ignored by the user, although the field has not been removed to avoid any possible future breaking changes. - -Additionally, Spark allows CPU limits to be set for the driver and executor using standard Spark settings (`spark.{driver|executor}.cores}`) as well as Kubernetes-specific ones (`spark.kubernetes.{driver,executor}.limit.cores`). The CRD values for max. CPU is applied to both of these pairs of settings without making a distinction. +Spark allows CPU limits to be set for the driver and executor using standard Spark settings (`spark.{driver|executor}.cores}`) as well as Kubernetes-specific ones (`spark.kubernetes.{driver,executor}.{request|limit}.cores`). Since `spark.kubernetes.{driver,executor}.request.cores` takes precedence over `spark.{driver|executor}.cores}`, `spark.{driver|executor}.cores}` is not specified by the operator when building the spark-submit configuration. === Memory