Create a standard for the security of iaas service software

[josephineSei]

closes #749

[josephineSei]

We discussed today in the IaaS meeting, how to proceed.
The Conclusions were:

• expecting security patches to be deployed is reasonable for SCS-Compatible, but may be not easily possible to test
  -> this would need self-reports from CSPs in the reasonable timeframe, that they have updated their deployments
• SBOMs are part of SCS-Open and will give hints on both: overall software version and security patches
• behavior changes and/or must-have new API-endpoints should be done through standards with reasonable timeframe to become "mandatory"
  -> this means that a CSP will have to look through the standards and check, whether their used software version fulfill all standards
  -> the standards will have to be adjusted ob a regular basis
  -> we should think about also enabling another test for future requirements of standards (this means nearly all SHOULDS !)

This means I will rename the issue/PR and standard and will discard work on the tests, because the presence of security patches will not be easily testable.

[markus-hentsch]

Should we maybe consider cases here where vulnerabilities are not applicable to certain deployments? E.g. vulnerability concerns storage backend driver A whereas CSP is exclusively using storage backend driver B.
In such case, the update would not be a MUST for the CSP, in my opinion.

We should maybe put a preceeding step here to instruct the CSP to do an analysis first ASAP whether they are affected and if ...

• ... they are affected, they MUST apply the update and report having done so to SCS
• ... they are not affected, they MAY apply the update but MUST report a reasoning why they are not affected to SCS

[markus-hentsch]

Good first draft! I added some minor spelling/phrasing correction suggestions and some questions.
I think the topic is not easy and we will need to be careful about two matters in particular:

1. the timeframe/deadline imposed on CSPs to apply updates
2. the way CSPs report back to SCS/OSBA about their compliance concerning software versions

No. 1 needs to be a sweet spot between giving CSPs enough room to implement/test and being strict enough to ensure security in a meaningful way.

Regarding no. 2 we maybe could be a bit more precise or provide a template. Unless I have missed something this could currently range from hand-written letters to SBOM XML files that are gigabytes in size which no human can look through in a timely manner.

[anjastrunk]

I'm afraid, this will result in a lot of paper work and bureaucracy no one wants to do.

[josephineSei]

That really is a problem:
We want to create standards that should be followed. Therefore we need tests.
But future CVEs or OSSNs cannot be known now and may be not easily testable. Should we consider writing a test for each of these vulnerabilities? This might be several tests a year to write, if a test from the outside is possible at all. And those tests might be considered as attacks (I mean they somehow are ;) ) and might trigger responses on the side of CSPs.

So a CSP giving notice about the fixing of a vulnerability, which already implicates, that we need to trust the CSP, could be an option. Maybe - if we already need to trust the CSPs - we may reduce the paper work:
MArkus suggested that there should be some kind of form file, that could be provided. We could do something like:

How did you deal with the vulnerability on your deployment XYZ:
[ ] Deployment was NOT affected, because ....... (e.g. we use another backend)
[ ] Functionality was disabled in the deployment.
[ ] Deployment was updated with a fix for the vulnerability, that .... (e.g. was provided be upstream openstack)

And only require short descriptions like above.
This may even be easier to evaluate in the compliance checker?

[anjastrunk]

I see your point @josephineSei and agree with @markus-hentsch. A form file could be a good trade-off. In fact, the file could be publicly available and evaluated automatically, which reduces paper-work. Assuming, we do not have any evil CSP (which will be recognizes sooner or later anyway ;)), I am fine with this approach.

[anjastrunk]

I had a look to this standard, too. IMO the standard should dictate patch time explicitly, such as two weeks. Describing time passed between publishing a patch and applying it, should not be described softly, such as "very soon", "as soon as possible". The standards does not bring any value-added to customers, as everybody interprets these words differently.

Secondly, I'm not convinced of the process a CSP uses to proof a certain vulnerability is fixed. It looks like manual steps are required, which are error-prone and will delay process additionally. I prefer to develop an automatic process, e.g. using digital signatures.

[artificial-intelligence]

For now I just want to comment on something I don't see explained or discussed neither in the proposal nor in any of the current commentary:

This standard - in it's current form - talks broadly about "(the) software" comprising the IaaS layer, and I assume this mostly is meant to mean Openstack software.

However I would find it important that we clarify this, because a typical IaaS Deployment is comprised of the following - very much different - software layers, especially when it comes to security updates.

• Firmware needed for bare metal components, e.g. NIC and SSD firmware, CPU microcode
• basic operating system software needed to boot and manage hardware, e.g. grub bootloader, linux kernel, initrd etc.
• basic operating system - usually linux - packages to support a base layer of software and to start the next layer: systemd, glibc, a package manager (apt/dpkg), network configuration software (Netplan), a container runtime (docker, podman, k8s), others
• some kind of container, which often contains it's own layer of basic operating system packages again (see above point), minus the container runtime stuff itself
• the actual openstack services, usually python files (these are mostly covered by the openinfra security advisories/ OSSAs)
• direct openstack service dependencies also developed under the openinfra umbrella - also covered by the openinfra security process, e.g. oslo libraries
• (in)direct openstack service dependencies developed outside openinfra security process, e.g. docker-py, various low level python librariers, like python-ldap (for keystone)
• adjacent services deployed as containers alongside openstack infrastructure, e.g. logging and monitoring (fluentd, prometheus, ELK stack, keycloak) which are also not part of the openinfra security process

So I think it is important to at least distinguish which software from the above list the CSP should patch. If the answer to this is: "all of it" then we must be clear that only a tiny fraction of it is covered by openinfra security processes!

e.g. even direct dependencies of core openstack services are actually not monitored by openinfra for security issues if these are not developed under the openinfra umbrella!

of course, if issues are reported these are fixed, but e.g. the requirements files listing tested versions which are known to work with various openstack services are no indicator that these releases are secure!

So I think this is important to highlight to the user.
This is the reason upstream provided kolla docker images are only provided for testing purposes and not for production, because these are not regularly updated for security fixes.

So imho we should keep in mind that CSPs need to monitor way more than just OSSA announcements if they want to be secure.

Thanks

[josephineSei]

For now I just want to comment on something I don't see explained or discussed neither in the proposal nor in any of the current commentary:

This standard - in it's current form - talks broadly about "(the) software" comprising the IaaS layer, and I assume this mostly is meant to mean Openstack software.

Indeed this standard should be applied on the IaaS Level, meaning all services with user-accesible APIs on that layer (this includes OpenStack and S3 if used).

Nevertheless we definitely should talk about security on all layers from Metal to KaaS. We may discuss this in the IAM call next week. But I think putting all of it into a single standard may be too complex?

However I would find it important that we clarify this, because a typical IaaS Deployment is comprised of the following - very much different - software layers, especially when it comes to security updates.

* Firmware needed for bare metal components, e.g. NIC and SSD firmware, CPU microcode

* basic operating system software needed to boot and manage hardware, e.g. grub bootloader, linux kernel, initrd etc.

* basic operating system - usually linux - packages to support a base layer of software and to start the next layer: systemd, glibc, a package manager (apt/dpkg), network configuration software (Netplan), a container runtime (docker, podman, k8s), others

* some kind of container, which often contains it's own layer of basic operating system packages again (see above point), minus the container runtime stuff itself

* the actual openstack services, usually python files (these are _mostly_ covered by the openinfra security advisories/ OSSAs)

* direct openstack service dependencies also developed under the openinfra umbrella - also covered by the openinfra security process, e.g. oslo libraries

* (in)direct openstack service dependencies developed outside openinfra security process, e.g. docker-py, various low level python librariers, like python-ldap (for keystone)

* adjacent services deployed as containers alongside openstack infrastructure, e.g. logging and monitoring (fluentd, prometheus, ELK stack, keycloak) which are also not part of the openinfra security process

So I think it is important to at least distinguish which software from the above list the CSP should patch. If the answer to this is: "all of it" then we must be clear that only a tiny fraction of it is covered by openinfra security processes!

This is a good point that needs to be integrated into the standard regardless of how detailed the standard itself is.

e.g. even direct dependencies of core openstack services are actually not monitored by openinfra for security issues if these are not developed under the openinfra umbrella!

of course, if issues are reported these are fixed, but e.g. the requirements files listing tested versions which are known to work with various openstack services are no indicator that these releases are secure!

So I think this is important to highlight to the user. This is the reason upstream provided kolla docker images are only provided for testing purposes and not for production, because these are not regularly updated for security fixes.

So imho we should keep in mind that CSPs need to monitor way more than just OSSA announcements if they want to be secure.

Thanks

Thank you for your feedback. I will integrate the focus on dependencies that are not monitored by OpenStack. For the rest I think we should discuss, whether this standard should have a very broad scope or just focus on the IaaS Layer. When we decide for the latter, we will most likely need standards for the other layers too.

[josephineSei]

@anjastrunk and I discussed @artificial-intelligence comment.
We both think, that we should have a standard for security and updates on each Layer from Hardware to OS to IaaS and up to KaaS.
I wrote 3 other issues, to reflect this:
Hardware Layer: #790
OS Layer: #791
KaaS Layer: #792

@artificial-intelligence I hope this reflects your concerns.

This would also narrow done the scope of this PR to exactly the IaaS Layer and dependencies of it.
If you miss some software or want to explicitly call out some in the standard, please let me know.

[fkr]

@Adri2000 @neuroserve @matfechner please review this - I think it is important that the view from CSPs is incorporated.

[markus-hentsch]

I added some minor suggestions to the standard text.

[markus-hentsch]

I think this is in a very good shape for a draft now. Any open points such as the exact manner of providing proof of applied security fixes or applicable SBOM formats can be discussed with the community once this draft lands in main, in my opinion.