Skip to content

Commit

Permalink
Merge pull request #6140 from alphajc/master
Browse files Browse the repository at this point in the history
feat(tencentcloud): remove dependency on tke interface
  • Loading branch information
k8s-ci-robot authored Oct 10, 2023
2 parents d7d33c7 + 5d0fac9 commit 07df121
Show file tree
Hide file tree
Showing 75 changed files with 30,073 additions and 24,249 deletions.
68 changes: 40 additions & 28 deletions cluster-autoscaler/cloudprovider/tencentcloud/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,15 @@ The following policy provides the minimum privileges necessary for Cluster Autos
{
"effect": "allow",
"action": [
"tke:DeleteClusterInstances",
"tke:DescribeClusterAsGroups",
"as:ModifyAutoScalingGroup",
"as:RemoveInstances",
"as:StopAutoScalingInstances",
"as:DescribeAutoScalingGroups",
"as:DescribeAutoScalingInstances",
"as:DescribeLaunchConfigurations",
"as:DescribeAutoScalingActivities"
"as:DescribeAutoScalingActivities",
"cvm:DescribeZones",
"cvm:DescribeInstanceTypeConfigs",
"vpc:DescribeSubnets"
],
"resource": [
"*"
Expand Down Expand Up @@ -72,8 +72,6 @@ env:
key: tencentcloud_secret_key
- name: REGION
value: YOUR_TENCENCLOUD_REGION
- name: REGION_NAME
value: YOUR_TENCENCLOUD_REGION_NAME
- name: CLUSTER_ID
value: YOUR_TKE_CLUSTER_ID
```
Expand Down Expand Up @@ -124,10 +122,6 @@ spec:
key: tencentcloud_secret_key
- name: REGION
value: YOUR_TENCENCLOUD_REGION
- name: REGION_NAME
value: YOUR_TENCENCLOUD_REGION_NAME
- name: CLUSTER_ID
value: YOUR_TKE_CLUSTER_ID
image: ccr.ccs.tencentyun.com/tkeimages/cluster-autoscaler:v1.18.4-49692187a
imagePullPolicy: Always
name: cluster-autoscaler
Expand All @@ -143,25 +137,9 @@ spec:
name: tz-config
hostAliases:
- hostnames:
- cbs.api.qcloud.com
- cvm.api.qcloud.com
- lb.api.qcloud.com
- tag.api.qcloud.com
- snapshot.api.qcloud.com
- monitor.api.qcloud.com
- scaling.api.qcloud.com
- ccs.api.qcloud.com
ip: 169.254.0.28
- hostnames:
- tke.internal.tencentcloudapi.com
- clb.internal.tencentcloudapi.com
- cvm.internal.tencentcloudapi.com
- tag.internal.tencentcloudapi.com
- as.tencentcloudapi.com
- cbs.tencentcloudapi.com
- cvm.tencentcloudapi.com
- vpc.tencentcloudapi.com
- tke.tencentcloudapi.com
ip: 169.254.0.95
restartPolicy: Always
serviceAccount: kube-admin
Expand All @@ -176,6 +154,40 @@ spec:
name: tz-config
```
### Auto-Discovery Setup
### Scaling up from 0 nodes
When scaling up from 0 nodes, the Cluster Autoscaler reads ASG tags to derive information about the specifications of the nodes
i.e labels and taints in that ASG. Note that it does not actually apply these labels or taints - this is done by an AWS generated
user data script. It gives the Cluster Autoscaler information about whether pending pods will be able to be scheduled should a new node
be spun up for a particular ASG with the asumption the ASG tags accurately reflect the labels/taint actually applied.
The following is only required if scaling up from 0 nodes. The Cluster Autoscaler will require the label tag
on the ASG should a deployment have a NodeSelector, else no scaling will occur as the Cluster Autoscaler does not realise
the ASG has that particular label. The tag is of the format
`k8s.io/cluster-autoscaler/node-template/label/<label-name>`: `<label-value>` or `tencentcloud:<label-name>`: `<label-value>` is
the name of the label and the value of each tag specifies the label value.

Example tags:

- `k8s.io/cluster-autoscaler/node-template/label/foo`: `bar`
- `tencentcloud:foo`:`bar`

The following is only required if scaling up from 0 nodes. The Cluster Autoscaler will require the taint tag
on the ASG, else tainted nodes may get spun up that cannot actually have the pending pods run on it. The tag is of the format
`k8s.io/cluster-autoscaler/node-template/taint/<taint-name>`:`<taint-value:taint-effect>` is
the name of the taint and the value of each tag specifies the taint value and effect with the format `<taint-value>:<taint-effect>`.

Example tags:

- `k8s.io/cluster-autoscaler/node-template/taint/dedicated`: `true:NoSchedule`

From version 1.14, Cluster Autoscaler can also determine the resources provided
by each Auto Scaling Group via tags. The tag is of the format
`k8s.io/cluster-autoscaler/node-template/resources/<resource-name>`.
`<resource-name>` is the name of the resource, such as `ephemeral-storage`. The
value of each tag specifies the amount of resource provided. The units are
identical to the units used in the `resources` field of a Pod specification.

Example tags:

Auto Discovery is not supported in TencentCloud currently.
- `k8s.io/cluster-autoscaler/node-template/resources/ephemeral-storage`: `100G`
74 changes: 74 additions & 0 deletions cluster-autoscaler/cloudprovider/tencentcloud/client/client.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
/*
Copyright 2021 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package client

import (
"context"
"encoding/json"
"time"

"github.com/pkg/errors"
"k8s.io/klog/v2"

"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/tencentcloud/metrics"
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/tencentcloud/tencentcloud-sdk-go/common"
tchttp "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/tencentcloud/tencentcloud-sdk-go/common/http"
"k8s.io/autoscaler/cluster-autoscaler/cloudprovider/tencentcloud/tencentcloud-sdk-go/common/profile"
)

// Client defined a tencentcloud client
type Client interface {
Send(ctx context.Context, req tchttp.Request, res tchttp.Response) error
}

// NewClient new a Client
func NewClient(credential common.CredentialIface, region string, cpf *profile.ClientProfile) Client {
c := &client{}
c.Init(region).
WithCredential(credential).
WithProfile(cpf)
return c
}

type client struct {
common.Client
}

// Send a http request
func (c *client) Send(ctx context.Context, req tchttp.Request, resp tchttp.Response) error {
if req == nil || resp == nil {
return errors.New("req & resp are not allowed to be empty")
}
if c.GetCredential() == nil {
return errors.New("Send require credential")
}

start := time.Now()
err := c.Client.Send(req, resp)

// 上报指标
duration := time.Since(start)
metrics.RegisterCloudAPIInvoked(req.GetService(), req.GetAction(), err)

// 打印日志
responseBytes, _ := json.Marshal(resp)
requestBytes, _ := json.Marshal(req)
klog.V(4).Infof("\"invoke cloud api\" Host=%s Service=%s Action=%s Duration=%s Request=%s Response=%s",
req.GetDomain(), req.GetService(), req.GetAction(), duration.String(), string(requestBytes), string(responseBytes))

return err
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
Copyright 2016 The Kubernetes Authors.
Copyright 2021 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -17,7 +17,7 @@ limitations under the License.
package metrics

import (
tencent_errors "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/tencentcloud/tencentcloud-sdk-go/tencentcloud/common/errors"
tencent_errors "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/tencentcloud/tencentcloud-sdk-go/common/errors"
k8smetrics "k8s.io/component-base/metrics"
"k8s.io/component-base/metrics/legacyregistry"
)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
/*
Copyright 2021 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package doc
Loading

0 comments on commit 07df121

Please sign in to comment.