How to report security issues

[rkoshak]

I don't know if this is an appropriate place to discuss this but I can't think of anywhere better to bring it up, and that's part of the problem.

There was a discussion on the forum (https://community.openhab.org/t/how-to-report-security-related-issues-discussion/66668) about the lack of a security issue reporting mechanism in the OH projects. Right now we don't have a good process for users to report security issues that are discovered.

Publicly reporting them and discussing them on the forum or in github issues doesn't seem to be the best idea as they immediately become zero-days as soon as they are published. Many other projects have a way for users to disclose security related issues privately so the project can have a chance to fix or at least mitigate the problem before publicly announcing it is a problem.

There was also discussion about how we can get the project to get involved with CVEs (full disclosure, I work for the company that invented CVE though I am not involved with any of that stuff) but I think looking into that would be a secondary issue that any security team would get to deal with. ;-)

So, is this a topic appropriate for the AC or is there some other organization available that can discuss and spin up a security team? If not, where can I bring this up. An issue on the core repo perhaps?

It feels like it might be appropriate for the AC as it is kind of a project wide concern.

It is a great idea and I think it would go a long way to show the maturity of OH as a project.

[ghys]

I have to admit this has been one of my biggest gripes with openHAB in general.
To be perfectly honest my approach is to consider it to be insecure by default (until it gets some kind of authn/authz framework built-in - the REST API exposes a lot of functionality without any restriction whatsoever ) and it's not even HTTPS by default so you have to rely on additional, external hardening to get any security - like a reverse proxy or separate VLANs.
This is not something that the average user is able to do however, and even with authentication, non-encrypted HTTP traffic can also be sniffed on LANs so bearer tokens and clear-text password/basic auth is not an appropriate measure to secure sensible information.

So I would say, if you run openHAB currently you kind of accept that such sensible info like credentials to third-party services might easily be retrieved.
Now this doesn't mean we couldn't mitigate the risks by limiting the things unauthenticated users can do with the REST API. Like, for example, the openHAB Cloud addon relies on you retrieving the "secret" on your filesystem to authenticate to the cloud service - and to some extent, configuring things via textual configuration protects you in that if you don't have (SMB/SSH) access to the server's filesystem, you won't be able to alter the configuration.

So in a nutshell, again I don't consider openHAB to be locally or remotely secure by default unless the issues stated above (find a way to get HTTPS easily, like Home Assistant/hass.io does with DuckDNS and get a proper authn/authz framework, but the HTTPS part is paramount) are resolved, so I wouldn't even consider issuing CVEs for it at this time. openHAB Cloud/myopenhab.org provides some kind of security for the remote access, or you could provide your own with nginx or similar, and restrict usage on your LAN, even if it defeats openHAB's purpose...

[rkoshak]

While I don't disagree with anything you said re the security of OH in general, I still don't know what your answer to the overall question of whether there should be a security team with a way to report security related issues to privately should exist and what should that look like if the answer is yes.

Is your position that OH is so hopelessly insecure that there is no point in having a security team? If so I might counter with "then how will it ever get any better?"

I only brought up the CVE idea because it was mentioned in the thread I linked to. Whether/when we ever get to the point of requesting/issuing CVEs is, IMHO, an exercise left to the security team, should one be established.

Do you think the establishment of a security team falls under the scope of the AC? Is this something Kai can/should decide on unilaterally?

I whole heartedly agree with all the security issues you have pointed out and have a bunch of my own that could be added to the list. But I want to be clear that this isn't what I intended to be the scope of this discussion. My questions are:

• should there be a special security team?
• should there be a private forum (I mean this in the generic meaning of the term, it could be a mail list, slack, or whatever) where security issues like all the ones you've listed should be discussed?
• should there be a way for users to submit issues to this private forum?
• does any of this fall under the scope of the AC to decide?

[ghys]

Is your position that OH is so hopelessly insecure that there is no point in having a security team? If so I might counter with "then how will it ever get any better?"

I wouldn't go that far as "hopelessly" ;)
The main problem IMHO is that while it's great for UIs, it exposes too much with an unsecure API. I think a security team could be valuable in order to at least acknowledge the security issues with running openHAB, like "you can get this password with a simple API call" or "you can make openHAB run arbitrary code remotely".

However, since there aren't necessarily any easy solutions to these issues available currently, I wonder if keeping openHAB secure is really an openHAB issue as it stands now (!)... I mean, you can send commands to any of your items (and your underlying systems) with a simple curl command, without any kind of authentication. There are still technical measures we could think of to help somehowin some areas, like allowing to update passwords in configuration with an HTTP POST but disallowing getting them back in clear text with an HTTP GET - this could be the security team's goal to track those.

[rkoshak]

This is exactly the sort of thing I think a security team would be responsible for. Identifying, tracking, and proposing or implementing fixes for these sorts of things.

But of course, we would need people to actually "staff" such a team. I can certainly offer my support but in the near term it would primarily be in a red team capacity as I'm not in a position to code the fixes.

[kaikreuzer]

I think the most pragmatic way would be to define that the AC should deal with such requests. It makes sense if we offer a channel where people can privately express their concerns, if they think they have found a vulnerability. And those might not be just in the runtime itself, it could also happen for openhab-cloud or the apps, a skill or whatever.
The AC would be responsible for assessing the severity, delegating the issue to the right developers and tracking the status - no need to do any coding yourself, @rkoshak 😄.
The question is how the communication channel could look like - can people send a private message here on Github? (I see that I can change the visibility to private, but I am not sure, if others can do so as well).
I could also set up an e-mail address, but I would want to avoid this additional maintenance as the distribution list would have to be kept in sync.

[rkoshak]

I did some quick checking and it seems that GitHub used to have a private message feature but no longer does and the work around I'm finding to fake it are really not acceptable.

Let me spend a little time researching to see if we have tools available here on GitHub to facilitate this. If not I'll look into what other projects do and use to see what the simplest approach we can set up and use would be. I'll come back here in a few of days with the options I can find.

[rkoshak]

Here is the results of my searching.

NodeJS uses slack: nodejs/security-wg#140

A best practice is to publish our security reporting and response policy as a SECURITY.MD file in our github repos. Tensorflow has a particularly good one to use as an exemplar. https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md

A search of SECURITY.MD files shows the most common approach for users to submit a security issue is through an email. We would try to use something like [email protected] I think. A few, like tensorflow will publish a public key so the email can be encrypted.

As for where/how to hold the resultant discussion to assign a severity rating and such I've found the following.

• Lots of projects use Slack including NodeJS.

• Hackerone is a commercial service that provides tools and a space to manage the whole security reporting and response process. It's used by a number of large projects like Ruby, Rails, and Django. Once we add a SECURITY.MD file we will qualify for their free open source support. I did not look deeply into exactly what they provide, but found their service recommended in several places including here.

• Here on Github we can create Private posts which should only be see able and respond able by members of the team where it's posted. One challenge I can see with this though is if we need to pull in a maintainer or contributor to the discussion there is no way we can do that except by creating a new public posting, or changing the AC team membership

They all have their advantages and disadvantages with no clear winner in terms of the "correct" choice. I kind of like the flexibility of Slack to bring in other people into the discussion on an as needed basis. But I don't like needing to rely on a third party service. I fear that private discussions here on Github may not be flexible enough in the long run, though it might be a good way to get started.

I think the next steps are as follows:

1. Establish our security reporting policy and procedures. How do we go from report to bug fix? How do we assign severity? etc.

2. Establish an email that security problems can be reported to and make sure it's monitored.

3. Choose and work out how we will discuss the security issue in private using one of the tools above (or something I've not found or considered).

4. Publish the SECURITY.MD file with the policy documented and email included. We don't need to be as detailed and comprehensive as tensorflow's, but I'd like to see more than "report security issues to this email."

@ghys, I think some of what you posted already in this thread would be good to reword and include in the SECURITY.MD file to document the fundamental security posture of OH and how (for the time being) those should be mitigated by not exposing OH.

I have some ideas on all of these which I'll document over the weekend. But I'd like to see any thoughts or ideas or strong feelings about this sooner rather than later.

Unrelated to the above I did discover that Github provides a dependency vulnerability checker, driven by CVE. I only bring it up in case it isn't already being used by OH.

[wborn]

Now that openHAB has a .github project we can also add a SECURITY.md file to it. After adding the file it will show in all openHAB repositories in "Security > Policy".

If you search for SECURITY.md github most organizations use an e-mail address for reporting security related issues. Some even provide a PGP key.

The main benefit of using an e-mail address is of course that everyone already uses e-mail so there isn't any barrier (setting up accounts/software) for people to report these important issues.

[rkoshak]

I was waiting for the build issues to die down but it looks like that we are getting to the point where we can pick back up on this issue. The big open question I think is how do we set up the email address and who should have access to it? I like the idea of providing the option of using PGP but I don't think we need to require it.

That will cover the reporting piece of the puzzle. Next we need to decide how to discuss submitted issues in private and bring in the appropriate developers (e.g. if the issue is with a specific binding). I need to go back and review the research I did many months ago on this part of the topic. IIRC I think we can manage it through github, but many projects use some other mechanism like Slack or Discord.

[wborn]

Setting up a security policy also shouldn't be restricted to issues in the openHAB source code.

I think a lot more damage can be done due to security issues in the openHAB infrastructure. These would allow for unauthorized access of GitHub repositories, web servers, community forums, build servers, Bintray, Docker Hub etc. A lot of damage can be done when an attacker finds a way to inject malignant code into one of the distribution downloads.

[kaikreuzer]

I managed to set up [email protected], which automatically shows the messages in our forum at https://community.openhab.org/u/kai/messages/group/architecture-council. I think this is pretty convenient as we can all see, triage and discuss the issues there and all our comments will be encrypted (only the initial report is sent unencrypted via e-mail).
I hoped to be able to also setup [email protected] to be automatically forwarded to [email protected], but unfortunately, Discourse cannot route the message anymore as the addressee is being changed by the forwarding. Would you think [email protected] is good enough or shall I try to find some other way to make [email protected] work?

Wrt processes on how to deal with reports, I'd think that we can first see what comes in - I doubt that it'll be much and we can then learn from this and decide how to set up a suitable process in future.

[wborn]

That's certainly a convenient solution! 👍 Do you know if that solution also allows for contacting the e-mail sender and is the sender automatically part of all replies sent to the message? I like [email protected] even better, but wouldn't spend days on it to get it working. ;-)

[rkoshak]

I wholly agree. Infrastructure as well as architectural issues are all fair game for reporting. Just look at what was discovered with Webmin recently. https://arstechnica.com/information-technology/2019/08/the-year-long-rash-of-supply-chain-attacks-against-open-source-is-getting-worse/

I think having the emails automatically forwarded to a closed part of the forum is pretty genius. the community.openhab.org part of the email doesn't bother my much. Most people will just click on the link or copy it into their mail client so that's no big deal.

wborn does have a good point though. It is likely that we will need to get more information from the original reporter beyond the initial submission.

If we want to test this out, I have an issue I've been holding on to a bit that I can report. It has more to do with the default set of stuff distributed with with OH rather than a specific technical vulnerability. I'll type it up in an email and we can see how this all works. Just let me know when we are ready.

[ghys]

I'm all in for having security issues sent via email forwarded to Discourse as key people who are able to handle them are active on the forum and will automatically receive them without hassle.

However: as pointed out above, I think it's important to also have a way to open a communication channel with the reporter i.e. being able to reply, a simple "send-your-reports-here" no-reply email address is probably not enough.

Also, if it's not too much of a hassle, for "optics'" sake, I do feel [email protected] looks better than [email protected] as it means those issues are official project business and taken seriously, rather than "delegated" to the community (this is in no way meant to be derogatory of course). The fact that you tried to set it up in the first place tells you were in the same line of thought ;)

[kaikreuzer]

Sorry for dropping the ball here.
So the forum solution just won't work. We cannot use [email protected] and we do not have a "back" channel to the reporter (which I agree is very important).
So what is our option then? I could setup an IMAP (+SMTP) account and share the credentials for it with the AC (assuming that the AC is the group that deals with security reports, at least for the start). This would allow us to easily contact the reporter, but we would not have a good way to discuss the issue internally. We would probably manually create a (closed) forum discussion for each new report.
Any better ideas?

[wborn]

Another option could be to create a HTTPS secured form on a website where people enter these issues. Then there could be a smart backend that takes care of notifying us. Something like Google Forms springs to mind. Amazon probably also has such a tool.

We could also build our own form with backend logic on the website. Maybe @ghys has some experience with this?

Ideally it would be nice if there was just some open source off-the-shelf solution we could use for this. I tried to search for one but didn't find any yet. I'd rather not start a new OSS project for this. 😉

[ghys]

Netlify has a forms feature (https://www.netlify.com/docs/form-handling/), it looks easy enough to set up on the website, the forms would end in their admin app and we could theoretically also be notified via Zapier (on Slack or Discourse: https://zapier.com/app-directory/netlify/integrations/discourse). Note that a form doesn't solve the lack of an ability to reply problem.

[wborn]

They also have a GitHub integration so we could also make a private repository for this. Then the integration could create the issue in the private repo. There we can discuss it and when the form also requires an e-mail address on submission we'd have a way to contact the submitter. :-)

[kaikreuzer]

The form isn't really better than using an e-mail address - as we would still need an e-mail address for reaching out to the reporter. A form is imho only additional effort, which does not bring much value.
Also, we do not have private Github repos, so I'd prefer to keep the discussions within Discourse. If we'd manage to get the inbound mails working for [email protected], this would be smooth.

[wborn]

Everyone has unlimited private GitHub repositories nowadays for free so that shouldn't be a blocking issue. I'm fine with either solution as long as we have one.

[rkoshak]

I think it's worth not reinventing the wheel here unless we really have to. Lots of projects use Slack and larger open source projects and many commercial companies use Hackerone which will offer it's services to us for free now that we have a SECURITY.MD file. The advantages of either approaches are that we have more control over who can and cannot participate and we don't have to build anything ourselves.

[ghys]

I just noticed GitHub now has a security advisory feature in beta. You can see it in a repo's security tab if you have admin access. Discussions there happen in private until the advisory is published and it seems to be possible to add collaborators, potentially outside the organization (not sure about that point).
This could be a good way of discussing with the reporter after the initial contact by email (I'm assuming everyone has a GitHub account)?

[rkoshak]

That sounds to be exactly what we need and we don't have to reinvent the wheel or move to some external service (even one we host ourselves). Assuming that works then we just need to figure out the [email protected] issue and we have something we can start with.

I've a couple of things in mind that I can use as the beta tester for the process when everything is set up and we think we are ready.

[ghys]

it seems to be possible to add collaborators, potentially outside the organization (not sure about that point)

According to https://help.github.com/en/articles/adding-a-collaborator-to-a-maintainer-security-advisory:

The person you add as a collaborator must have read permissions to the repository. If the repository is owned by an organization, the person must also be an organization member or outside collaborator on the repository.

In this case, if I understand correctly we would have to temporarily add the reporter as an outside collaborator (https://help.github.com/en/articles/adding-outside-collaborators-to-repositories-in-your-organization), with the Read permission on the repo - which seems to be harmless, doesn't everyone have access to public repos?

[ghys]

My view of the process so far:

1. we set up a security policy file (SECURITY.md) in all repos on GitHub and/or the .github repo (for example, based on the template suggested by GitHub - https://github.com/openhab/openhab-addons/security/policy) and also on a new page on openhab.org (About > Security) - this security policy would simply tell reporters to contact [email protected] and expect an answer within X days, otherwise they may publish their findings; it could also detail the rest of the process below for completeness. The reporter shall also have a GitHub account, and include it in the email for it to be considered;
2. we designate a "security response team" (could be only 1 person at first with an eventual backup, depending on volume) who have access to this email address;
3. the security response team, upon reception of a report, assesses and decides privately if it's accepted or denied - it may also privately contact a maintainer or another for advisory;
4. if the report is accepted, the security response team determines in which repo a draft security advisory should be created on GitHub (for instance, if it's an addon, https://github.com/openhab/openhab-addons/security/advisories, if it's cross-repo, https://github.com/openhab/.github/security/advisories, it could also choose to create all security advisories in the .github repo), and eventually classify the vulnerability using for instance CVSS v3
5. the security response team drafts the advisory, adds the reporter's GitHub account to the advisory's collaborators and eventually others - that may involve adding them as external collaborators for the project, which shouldn't give them any particular privilege;
6. the security response team replies to the original email with its decision (report accepted or not), and if applicable, the next steps of the process, with the URL of the draft security advisory;
7. discussions happen on the vulnerability between the reporter, the maintainers of the repo, the security response team and other collaborators on the security advisory's page until an eventual resolution or mitigation has been found and it's ready to be published;
8. the advisory is published, and possibly linked on the website's security page, blog and/or social media; we could also request a CVE ID through GitHub (https://help.github.com/en/github/managing-security-vulnerabilities/publishing-a-security-advisory#requesting-a-cve-identification-number) or consider becoming a CNA - although it would probably be overkill at this stage.

Wdyt? I'm reviving this because I have serious concerns about a couple of add-ons and am also willing to test the process.

[rkoshak]

I think it sounds great. I've nothing to add. I think everything is covered.

[kaikreuzer]

Yes, sounds excellent and it is pretty nice to have this Github feature available by now!
So whom should I put on the list for receiving [email protected] mails? Is this something we should maybe assign to the board of the foundation as the "guardians" of openHAB?

[ghys]

I doubt there will be objections to that. Besides, if we are to publish them on the website afterwards, the foundation would have to be involved anyways.

Note that creating advisories requires admin permissions on the repository, if we opt to create them in the relevant repo instead of the central .github one whenever possible (which makes more sense imho, for example https://github.com/openhab/openhab-android to discuss a vulnerability in the Android app, with the maintainer team) this doesn't leave many options as to who already has these broad admin permissions across all repos ;)

[kaikreuzer]

Ok, I have created https://github.com/openhab/.github/blob/master/SECURITY.md for a start and created [email protected], which is forwarded to [email protected].
@ghys Feel free to create the first draft advisories for the add-ons you have in mind :-)