-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refine CVE check in check script for k8s version policy #779
base: main
Are you sure you want to change the base?
Conversation
initial codes pushed to the git, the rest will be upcoming after tests |
Signed-off-by: Piotr <[email protected]>
Signed-off-by: Piotr <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few remarks regarding the standard. The code I can't review in a reasonable way because it seems to be work in progress (which is okay). Maybe you can mark this PR as draft?
3. CI Integration | ||
* Trivy | ||
- Providers should integrate Trivy into their CI pipeline to automatically scan Kubernetes cluster components, | ||
including kubelet, apiserver, and others. | ||
- The CI job MUST fail if critical vulnerabilities (CVSS >= 8) are detected in the cluster components. | ||
- JSON reports from Trivy scans should be reviewed, and Trivy's experimental status should be monitored for changes | ||
in output formats. | ||
* nvdlib (Fallback): | ||
- If Trivy fails or cannot meet requierements, nvdlib MUST be used as a fallback to query CVE data for Kubernetes | ||
versions, laveraging CPE-based searches to track vunerabilities for specific versions. | ||
- Providers using nvdlib MUST periodically query for critical cunerabilities affecting the Kubernetes version in production. | ||
|
||
4. TBD |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This standard has been stabilized already, for better or worse. New requirements can only be introduced in a new major version (then v3). However, I'm not sure that this was the original objective of this PR; here, we mainly wanted some tooling for the compliance check, and the providers are free to use whatever tools they want. (We can put these items into the implementation notes though, but only as non-authoritative recommendation!)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does that mean that I should restore original version of standard?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be good to make your research results available. We should just reframe them as guidelines for operators. We could write a blog post. I would then ask you to get feedback from Team Container. It would be good to talk to people who already use Trivy.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I have done right now is restore the original standard text and drop the changes.
According to the code, there were several changes made:
- Integrated Trivy for scanning Kubernetes pod images for security vulnerabilities.
- Fixed issue with ClusterInfo object being incorrectly passed where kubeconfig path was expected.
- Added logging improvements to provide clearer insights during version compliance checks.
- Refined the code structure to handle K8s image scanning and cluster versioning in an async manner.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I have done right now is restore the original standard text and drop the changes
This is not what I see.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would then ask you to get feedback from Team Container
Have you done that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I have done right now is restore the original standard text and drop the changes
@mbuechse I do apologize, I have reverted it now, it was lost somewhere on my git in the mess with the branches
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would then ask you to get feedback from Team Container
Have you done that?
I have not, I will bring that topic on the nearest container call(last week there was not a container call at all).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good!
Sorry, I went ahead and marked this as draft, and I changed the title as well to give context. |
- Integrated Trivy for scanning Kubernetes pod images for security vulnerabilities. - Fixed issue with ClusterInfo object being incorrectly passed where kubeconfig path was expected. - Addressed SSL certificate verification error when making external HTTP requests by adding proper handling. - Updated the compliance check logic to ensure correct validation of Kubernetes cluster versions and vulnerability checks. - Added logging improvements to provide clearer insights during version compliance checks. - Refined the code structure to handle K8s image scanning and cluster versioning in an async manner. Signed-off-by: Piotr <[email protected]>
Signed-off-by: Piotr <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See my remarks. Two general remarks:
- it would be good to have some explanation what is being done and how it relates to the standard -- please expand the docstrings and/or README.md
- do we now have redundant checks? the old CVE check doesn't seem to be gone?
3. CI Integration | ||
* Trivy | ||
- Providers should integrate Trivy into their CI pipeline to automatically scan Kubernetes cluster components, | ||
including kubelet, apiserver, and others. | ||
- The CI job MUST fail if critical vulnerabilities (CVSS >= 8) are detected in the cluster components. | ||
- JSON reports from Trivy scans should be reviewed, and Trivy's experimental status should be monitored for changes | ||
in output formats. | ||
* nvdlib (Fallback): | ||
- If Trivy fails or cannot meet requierements, nvdlib MUST be used as a fallback to query CVE data for Kubernetes | ||
versions, laveraging CPE-based searches to track vunerabilities for specific versions. | ||
- Providers using nvdlib MUST periodically query for critical cunerabilities affecting the Kubernetes version in production. | ||
|
||
4. TBD |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I have done right now is restore the original standard text and drop the changes
This is not what I see.
3. CI Integration | ||
* Trivy | ||
- Providers should integrate Trivy into their CI pipeline to automatically scan Kubernetes cluster components, | ||
including kubelet, apiserver, and others. | ||
- The CI job MUST fail if critical vulnerabilities (CVSS >= 8) are detected in the cluster components. | ||
- JSON reports from Trivy scans should be reviewed, and Trivy's experimental status should be monitored for changes | ||
in output formats. | ||
* nvdlib (Fallback): | ||
- If Trivy fails or cannot meet requierements, nvdlib MUST be used as a fallback to query CVE data for Kubernetes | ||
versions, laveraging CPE-based searches to track vunerabilities for specific versions. | ||
- Providers using nvdlib MUST periodically query for critical cunerabilities affecting the Kubernetes version in production. | ||
|
||
4. TBD |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would then ask you to get feedback from Team Container
Have you done that?
|
||
async def get_k8s_pod_images(kubeconfig, context=None) -> list[str]: | ||
"""Get the list of container images used by all the pods in the Kubernetes cluster.""" | ||
cluster_config = await kubernetes_asyncio.config.load_kube_config(kubeconfig, context) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This variable is not used (this fact gets reported by flake8 as well), and this doesn't seem right?!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
deleted
connector = aiohttp.TCPConnector(limit=5) | ||
async with aiohttp.ClientSession(connector=connector) as session: | ||
cve_affected_ranges = await collect_cve_versions(session) | ||
releases_data = fetch_k8s_releases_data() | ||
|
||
try: | ||
logger.info(f"Checking cluster specified by {kubeconfig_path}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs some more explanation, because almost the same message will be displayed in line 577 (only better, because there, it contains the context as well).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would that be satisfying?
f""" Initiating scan on the Kubernetes cluster specified by kubeconfig at '{kubeconfig_path}'
{' with context ' + config.context if config.context else ''}.
Fetching cluster information and verifying access.""")
scanner provides in the output additional info regarding vulnerability.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My point was that it looks like a duplicate of the other line, and it appears to me that the script now tries to achieve the same objective with two different means, one after the other.
Signed-off-by: Piotr <[email protected]>
…d, providing more specific description for initiating scan on the kubernetes, removing cluster_config variable Signed-off-by: Piotr <[email protected]>
Signed-off-by: Piotr <[email protected]>
Signed-off-by: Piotr <[email protected]>
Co-authored-by: Matthias Büchse <[email protected]> Signed-off-by: Piotr <[email protected]>
Signed-off-by: Piotr <[email protected]>
Hi @mbuechse and @piobig2871,
It looks like it loops over the path to the kubeconfig and uses every character as image to scan.
|
Signed-off-by: Piotr <[email protected]>
Signed-off-by: Piotr <[email protected]>
Hi @jschoone there was a problem with one argument, I have fixed the issue. |
Hi @piobig2871 looks good now! ./k8s_version_policy.py --kubeconfig /tmp/kubeconfig
....
version-policy-check: PASS I'd just recommend to check for ./k8s_version_policy.py -k /tmp/kubeconfig
WARNING: The EOL data in k8s-eol-data.yml isn't up-to-date.
INFO: Initiating scan on the Kubernetes cluster specified by kubeconfig at /tmp/kubeconfig
with context .
Fetching cluster information and verifying access.
INFO: Scanning image: registry.k8s.io/kube-apiserver:v1.29.10
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/metrics-server/metrics-server:v0.7.2
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/sig-storage/csi-attacher:v4.4.2
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/sig-storage/csi-snapshotter:v6.3.2
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/provider-os/cinder-csi-plugin:v1.29.1
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/coredns/coredns:v1.11.1
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/kube-controller-manager:v1.29.10
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/provider-os/openstack-cloud-controller-manager:v1.29.1
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/etcd:3.5.15-0
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: quay.io/cilium/cilium:v1.16.3@sha256:62d2a09bbef840a46099ac4c69421c90f84f28d018d479749049011329aa7f28
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.2
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/kube-scheduler:v1.29.10
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: quay.io/cilium/cilium-envoy:v1.29.9-1728346947-0d05e48bfbb8c4737ec40d5781d970a550ed2bbd@sha256:42614a44e508f70d03a04470df5f61e3cffd22462471a0be0544cf116f2c50ba
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.2
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/sig-storage/livenessprobe:v2.11.0
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: quay.io/cilium/operator-generic:v1.16.3@sha256:6e2925ef47a1c76e183c48f95d4ce0d34a1e5e848252f910476c3e11ce1ec94b
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/kube-proxy:v1.29.10
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Scanning image: registry.k8s.io/sig-storage/csi-resizer:v1.9.2
ERROR: Error running Trivy scan: [Errno 2] No such file or directory: 'trivy'
INFO: Checking cluster specified by default context in /tmp/kubeconfig.
INFO: The K8s cluster version 1.29.10 of cluster 'scs-cluster-129-admin@scs-cluster-129' is still in the recency time window.
version-policy-check: FAIL |
Co-authored-by: Kurt Garloff <[email protected]> Signed-off-by: Piotr <[email protected]>
No description provided.