Run kubent as cronjob

[shomeprasanjit]

@stepanstipl : i am trying to run kubent within a pod and have created a docker container for the same. I am getting the below error while trying to do so.

root@dnsutils-ivt:/# kubent -k /.kube/config -o json
8:39PM INF >>> Kube No Trouble `kubent` <<<
8:39PM INF version 0.5.1 (git sha a762ff3c6b5622650b86dc982652843cc2bd123c)
8:39PM INF Initializing collectors and retrieving data
8:39PM INF Target K8s version is 1.20.7-eks-d88609
8:39PM INF Retrieved 2 resources from collector name=Cluster
8:39PM ERR Failed to retrieve data from collector error="secrets is forbidden: User \"system:node:ip-10-150-9-122.us-west-2.compute.internal\" cannot list resource \"secrets\" in API group \"\" at the cluster scope: can only read namespaced object of this type" name="Helm v2"
8:39PM ERR Failed to retrieve data from collector error="list: failed to list: secrets is forbidden: User \"system:node:ip-10-150-9-122.us-west-2.compute.internal\" cannot list resource \"secrets\" in API group \"\" at the cluster scope: can only read namespaced object of this type" name="Helm v3"
8:39PM INF Loaded ruleset name=custom.rego.tmpl
8:39PM INF Loaded ruleset name=deprecated-1-16.rego
8:39PM INF Loaded ruleset name=deprecated-1-22.rego
8:39PM INF Loaded ruleset name=deprecated-1-25.rego
[]
root@dnsutils-ivt:/#

checked system:node clusterrole which already has the necessary permission for accessing secrets.

- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  verbs:
  - get
  - list
  - watch

Also, provide cluster-admin role to a service account IVT which is used for spinning up the cronjob or deployment.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ivt-kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: someserviceaccount
  namespace: XXXXX

The requirement is to identify:

1. what is barring kubent to run as a pod from inside the cluster.
2. What should be the right RBAC permission needed instead of providing it with cluster-admin prividege.

[stepanstipl]

Hi @shomeprasanjit thanks for raising this 👍 .

So you won't be able to use the node role, as K8s has a special Node Authorization mechanism, which allows nodes to only access ConfigMaps/Secrets for the relevant Pods, i.e. the Pods bound to that node.

I think you'll need to create a new ClusterRole and ClusterRoleBinding for the relevant ServiceAccount. Permissions to list and read all resources should be sufficient, i.e.:

kind: ClusterRole
metadata:
  name: reader
rules:
# note that  "" indicates only the core API group -> use "*" for all
- apiGroups: ["*"]
  verbs: ["get", "watch", "list"]

I think the watch verb shouldn't be needed either, but I'll need to confirm that. I've created #241 to add an example K8s deployment manifest.

🎄

[shomeprasanjit]

@stepanstipl : still getting the same error. the additional permission you mentioned didn't help.

9:40AM ERR Failed to retrieve data from collector error="Get "https://8FC721A9E1108C283C54D098D686A42B.yl4.us-west-2.eks.amazonaws.com/api/v1/secrets?labelSelector=OWNER%3DTILLER\": getting credentials: exec: exec: "aws": executable file not found in $PATH" name="Helm v2"

9:40AM ERR Failed to retrieve data from collector error="list: failed to list: Get "https://8FC721A9E1108C283C54D098D686A42B.yl4.us-west-2.eks.amazonaws.com/api/v1/secrets?labelSelector=owner%3Dhelm\": getting credentials: exec: exec: "aws": executable file not found in $PATH" name="Helm v3"

[stepanstipl]

Hi @shomeprasanjit - the error seems to be different. The first one you shared was about missing permissions:

8:39PM ERR Failed to retrieve data from collector error="secrets is forbidden: User \"system:node:ip-10-150-9-122.us-west-2.compute.internal\" cannot list resource \"secrets\" in API group \"\" at the cluster scope: can only read namespaced object of this type" name="Helm v2"

The second error:

9:40AM ERR Failed to retrieve data from collector error="Get "https://8FC721A9E1108C283C54D098D686A42B.yl4.us-west-2.eks.amazonaws.com/api/v1/secrets?labelSelector=OWNER%3DTILLER\": getting credentials: exec: exec: "aws": executable file not found in $PATH" name="Helm v2"

seems to be referring to the missing aws executable. I'm wondering however how is it happening, as this would point to your kubeconfig being configured with user.exec.command: aws. Can you please check the content of your kubeconfig file?

There should be no need for one inside the container, as since #237 kubent should correctly recognize that it's being executed from within the cluster and pickup the relevant service account.

[shomeprasanjit]

Hello @stepanstipl, I didn't install aws cli which caused the issue.. sorry for the confusion. but I am still seeing the same problem where it is erroring out with invalid configuration: no configuration has been provided error message when run within a pod.

root@dnsutils-ivt:/aws# kubent version
4:47AM INF >>> Kube No Trouble `kubent` <<<
4:47AM INF version 0.5.1 (git sha a762ff3c6b5622650b86dc982652843cc2bd123c)
4:47AM INF Initializing collectors and retrieving data
4:47AM ERR Failed to initialize collector: <nil> error="invalid configuration: no configuration has been provided"
4:47AM ERR Failed to initialize collector: <nil> error="invalid configuration: no configuration has been provided"
4:47AM ERR Failed to initialize collector: <nil> error="invalid configuration: no configuration has been provided"
4:47AM INF Loaded ruleset name=custom.rego.tmpl
4:47AM INF Loaded ruleset name=deprecated-1-16.rego
4:47AM INF Loaded ruleset name=deprecated-1-22.rego
4:47AM INF Loaded ruleset name=deprecated-1-25.rego
root@dnsutils-ivt:/aws#

Here is the full manifest for your reference.

root@dnsutils-ivt:/aws# kubent version
4:47AM INF >>> Kube No Trouble `kubent` <<<
4:47AM INF version 0.5.1 (git sha a762ff3c6b5622650b86dc982652843cc2bd123c)
4:47AM INF Initializing collectors and retrieving data
4:47AM ERR Failed to initialize collector: <nil> error="invalid configuration: no configuration has been provided"
4:47AM ERR Failed to initialize collector: <nil> error="invalid configuration: no configuration has been provided"
4:47AM ERR Failed to initialize collector: <nil> error="invalid configuration: no configuration has been provided"
4:47AM INF Loaded ruleset name=custom.rego.tmpl
4:47AM INF Loaded ruleset name=deprecated-1-16.rego
4:47AM INF Loaded ruleset name=deprecated-1-22.rego
4:47AM INF Loaded ruleset name=deprecated-1-25.rego
root@dnsutils-ivt:/aws#

Here is my kubeconfig

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: XXXXXX
    server: URL 
  name: klsdjfklsdf
contexts:
- context:
    cluster: sjdklfjlskd
    user: lsdkjflsd
  name: lsdkjfsd
current-context: ksjdfklsdjf
kind: Config
preferences: {}
users:
- name: sdfsdfsdfsdf
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-west-2
      - eks
      - get-token
      - --cluster-name
      - clustername-abc
      command: aws   

can you please provide me a correct manifest to implement it from within Kubernetes environment.

[stepanstipl]

@shomeprasanjit just to confirm, you're trying to connect to the very same cluster where your pod is running, correct?

If so, there's no need for the kubeconfig at all, simply remove it from your pod (also you should not need the awscli - that was because of the exec part in your kubeconfig).

kubent should correctly recognize it's running within the cluster, and use the K8S service account associated with t a pod. (What's happening in your case - because you have kubeconfig, it will take precedence)

[milanholubstratox]

Hi,

I've prepared #302 which comes with functional cronjob, pushgateway and grafana dashboard - more details in my fork https://github.com/milanholubstratox/kube-no-trouble/blob/CNINF-502-kubernetes-cronjob-deployment/docs/kubernetes-deploy-cronjob.md
As part of the task I've slightly modified Dockerfile and published docker image on https://hub.docker.com/r/oxuscloud/kubent - feel free to use it.

Please check it out!

@stepanstipl - could you review my PR?

[github-actions]

Message to comment on stale issues. If none provided, will not mark issues stale

[github-actions]

This Issue was closed because it has been stalled for 30 days with no activity.

[stepanstipl]

This should be covered in #241