Security Issues of the Mithril Software Architecture #1586
Replies: 15 comments 7 replies
-
|
@onyxstakepool Thanks for raising this issue. We think the actual picture is less bleak than the one you are painting here, but it's clear we are lacking good resources on the Mithril network threat model and an updated description of the architecture. We are planning to address your concerns promptly, eg. in the next few days (Work on ssue #1350 has been stalled but need to be moved forward). In order to help us provide the best possible answer, could you please provide some details on the "myriad of security issues" you are seeing here (taking into consideration our short discussion on the |
Beta Was this translation helpful? Give feedback.
-
|
@abailly-iohk Thanks for your respone. After learning in the Discord Moria channel that it is not technically necessary to run the Mithril signer on a block producer, I am much less concerned now about the security implications. Since a simple relay server is also sufficient for the Mithril signer only the KES key and the node certificate are at risk. Even if these two pieces are leaked from the relay, an attacker would not be able to replicate the block producer since the vrf key is still missing. The KES key and the node certificate can be quickly rotated without any harm to the real block producer. With this understanding, I suggest that the Mithril documentation is changed to recommend the naive deployment also for mainnet. See: Mithril signer deployment model My original security concern was that an attacker would spend considerable efforts to compromise the Mithril repository and thus gain access to the block producers and wreck havoc on the whole Cardano network. |
Beta Was this translation helpful? Give feedback.
-
|
IIRC we updated the suggested deployment model from relay to BP because some SPOs participating in Mithril were concerned about the potential of KES keys leaking when deployed on a relay which is inherently less protected 🤔 Your concern about supply-chain-based attacks is definitely valid and something we are also concerned with and should mitigate, but to be fair this is also something the cardano-node itself is subject to. We will take that into account in the threat model. |
Beta Was this translation helpful? Give feedback.
-
|
@onyxstakepool Thanks for this issue. Indeed, the new deployment model has been jointly designed with pioneer SPOs input and feedback, and they raised concerns about having the KES keys exposed on a relay. Prior to the As @abailly-iohk already mentioned, we are currently working on a threat model for the Mithril network. This will help us get a better picture of the security related issues, and it will also be an opportunity for the community to contribute and give some feedback. In the mean time, feel free to share any specific attack scenario that you would already have in mind 👍 Issue #1488 has also been created to make the architecture diagrams easier to understand. |
Beta Was this translation helpful? Give feedback.
-
|
Indeed, it was decided that keeping keys on world-facing cardano-node relays was not very acceptable. However, I don't see how an outgoing connection from a mithril-signer to an aggregator is a huge risk, personally. In my mind, it's no different than you trusting the NTP client app to connect to an NTP server and not "get hacked" in the process. "creates a myriad of security issues" I would like to see some example attack scenarios if i'm to be convinced. The way I see it, an attack would have to involve the mithril signer software having a serious bug PLUS the aggregator getting hacked at same time, crafting a malicious response to the signer, that could potentially execute some code in some "buffer overflow, etc". I guess theoretically possible? I think the engineers would have a better idea how possible it is to execute something like this. RE supply chain attacks, no different than people using any of the add-on software that SPOs use. Like... Koios Tools, Scripts, cncli, etc etc. Not to mention, the operating system where you are running your software from, and the huge number of libraries, packages, etc, not having a supply chain attack. Don't get me wrong, I do support the idea of Mithril just being integrated into the core protocol, but maybe it's a bit too early for this? |
Beta Was this translation helpful? Give feedback.
-
|
@disassembler just pointed out security issues with squid in the Mithril channel. |
Beta Was this translation helpful? Give feedback.
-
|
@onyxstakepool For the sake of completeness, you should also have posted @disassembler's answer
and also @jpraynaud's answer later on:
Squid is a piece of software and like all pieces of software can have vulnerabilities. Deploying and using any software for business critical missions require constant monitoring and attention to security and performance issues which, fortunately for the case of squid, are public and promptly fixed. |
Beta Was this translation helpful? Give feedback.
-
|
BTW, seems like Traffic is also subject to security exploits: https://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-19990/Apache-Traffic-Server.html |
Beta Was this translation helpful? Give feedback.
-
|
@abailly-iohk Thanks for addressing all the concerns above. Let me explain here why I keep voicing concerns about how Mithril is set up. First, as soon as you open a door (port) to your server, you are in the business of defending this door. So, for critical infrastructure you just do not open these doors unless absolutely necessary. For the cardano-node I am quite confident that every bit that flows in and out of the server is meticulously processed and checked. For the Mithril signer, relay, and aggregator the whole security concept looks much more ad hoc.
So, I must put a lot of trust into all these additional software pieces. What happens if the aggregator gets compromised or gets seized by the authorities? Then the aggregator software can be manipulated and potentially corrupt the signers and block producers of the stake majority or corrupt the databases of relay servers or even leak keys and so forth. You might still think this is all moot. Here is a story: Now we have the same pattern with Mithril. An unchecked http channel deep into the core SPO infrastructure that sends data back and forth waiting to be exploited. This is why I am concerned. Also, in the future there might be more than one aggregator with more delegated trust. The risks are just increasing. |
Beta Was this translation helpful? Give feedback.
-
|
While the described security issues imply certain requirements as necessities rather than configurable options, they actually reflect choices available to the SPO during installation and configuration. However, this would necessitate the SPO customize and thoroughly test the setup process, deviating from the steps provided in quick start guides. I will address each bullet point individually to clarify.
The same user is actually not a hard requirement. This simply provides "ease of use/setup" by using the same user. What this really provides is the required file permissions for all Mithril components without step by step instructions on the alternative configurations. Without using the same user for Cardano Node and Mithril Signer processes requires the user of the Mithril Signer process to have:
These requirements can be achieved without using the same user account. One way to handle this is by using
The signer does not require unrestricted access, it simply needs to be able to read the entire database to create signatures. The only write level access required is actually for Mithril Client, which could be run under its own use if the ACLs are setup to ensure any data written to the DB automatically inherits full read/write permissions for the user of Cardano Node process.
Yes, but only the KES secret key. As mentioned earlier an ACL could be applied that limits this access by user (UID) to only the KES secret key, and prevents reading any other keys, both public or private.
When setup correctly the Squid (or any) relay blocks access to anything except the block producer and potentially a backup block producer (hereafter referred to as BPs). I dare say "setup correctly" means that not only the proxy rules are limiting access to the BPs, that a firewall should also limit any socket connection to the BPs. With this setup, regardless of what relay software is actually used, there is reduced concerns about security due to vulnerabilities in the proxy as only the SPO's own infrastructure can reach the relay process. This also mitigates the potential for DOS/DDOS attacks taking down the SPO's Mithril relay. This really goes for any relay software whether its Squid, Nginx or Apache Traffic, they should all be secured, and if more than one Cardano Node relay exists I would suggest having a Mithril Relay (of whatever relay software) running on each. However, this does require setting up a "sidecar" load balancer (i.e. load balancer running on same host with Mithril Signer) to handle the redirection from the signer to your pool of Mithril Relays. Yes this makes the architecture a bit more complex, but it provides the redundancy/failover the current architecture would require for multiple relays.
Yes. However, from what I recall this is an HTTP protocol which uses a client / server model.
I don't believe its PUT or PATCH requests. However, I didn't tcpdump the traffic before writing this response to confirm but as long as it is still a HTTP client/server communications the difference between each is mostly moot for this discussion. The standards of an HTTP Response are:
My understanding of the aggregator is admittedly somewhat limited, only having an aggregator when doing the initial devnet testing. However given the Signer:
There is limited chance the Aggregator is controlling the Signer. Here is the description of the Aggregator:
While the description mentions collecting there is no actual inbound port opened for the Mithril Signer, as mentioned above. So the Aggregator has no intrinsic method to send traffic to the Signer. The client / server model of the HTTP protocol generally limits any type of "instructions" from the server (Aggregator) being sent to a client (Signer). Unless the client has some unique, I'd even call it "very odd", logic where it parses the HTTP Response Body for commands. Thus, I would not consider the Aggregator to be "controlling the signer". In reality the Aggregator should be "accepting" individual signatures from Mithril Signers to aggregate into a multi-signature. Therefore the aggregator could either accept or reject a signature from a Signer, but should have no other role in instructing the Signer to take actions on its behalf. @jpraynaud @abailly-iohk Please feel free to correct anything I've included in this post that you think I may have gotten incorrect about the Mithril Signer, Aggregator or the Client. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot @TrevorBenson for the thorough analysis! I did not take the time to address the various points in details and I think you're really doing it in a better way than I could do |
Beta Was this translation helpful? Give feedback.
-
|
@onyxstakepool Thanks a lot for voicing clearly your concerns. It's a bit late for most of us right now, so I won't be able to respond in details but let's keep the conversation going and make sure we address those in the clearest possible way. |
Beta Was this translation helpful? Give feedback.
This comment has been hidden.
This comment has been hidden.
-
|
That makes sense @ch1bo but it seems to me we are here discussing general principles and architecture rather than specific vulnerabilities, so perhaps doing it in the open would be better. Perhaps would it be even better to turn this into a Discussion? |
Beta Was this translation helpful? Give feedback.
-
|
Very good idea. Discussions have much better threading capabilities :) |
Beta Was this translation helpful? Give feedback.
-
|
What about running the mithril signer on your backup BP if you have one? It already has the KES key and node cert and in the event of a problem with the signer disrupting the node, your normal BP stays unaffected. |
Beta Was this translation helpful? Give feedback.
-
|
Is it common to have a backup block producer? |
Beta Was this translation helpful? Give feedback.
-
|
For bigger pools that mint lots of blocks per epoch it's essential imho. For smaller pools the extra cost most likely isn't justifiable... |
Beta Was this translation helpful? Give feedback.
-
|
It sounds like a possible option if you already run a backup BP and don't want to run on your main BP 👍 |
Beta Was this translation helpful? Give feedback.
-
|
IIUC there are already SPOs doing that. |
Beta Was this translation helpful? Give feedback.
-
|
@TrevorBenson @brouwerQ @abailly-iohk @jpraynaud @ch1bo I can report now that the multi-pool signer setup outlined in detail in issue #1605 works as intended after 5 epochs of operation on mainnet. This is a completely isolated setup form the stake pool operation. So many of the security concerns discussed above do not apply any more. Please review! |
Beta Was this translation helpful? Give feedback.
-
|
We have released a Thank you to all the participants! |
Beta Was this translation helpful? Give feedback.
-
Why
The current Mithril network architecture is susceptible to various types of denial-of-sevice attacks of the block producer nodes that run the Mithril signer. The goal of the Mithril network project to interact with the stake weighted majority of block producers makes this an even more pressing issue. The close interaction of the Mithril signer with the cardano-node database and the access to secret keys and certificates as outlined in the Architecture overview creates a myriad of security issues.
The design decision that a single server, the Mithril aggregator, is interacting with the most important Cardano nodes on the network, the stake-weighted majority of the block producers, is a huge security risk. Even an inadvert bug in the Mithril software could disrupt the majority of the block producers and lead to a catastrophic network failiure of the Cardano network.
Having such a single point of failure risk on the Cardano network is not acceptable.
It also puts an enormous trust burdon on a piece of software that is maintained outside of the main cardano-node repository.
What
A reasonable approach to mitigate the main security issues above is to integrate the Mithril signer code into the cardano-node software and use the establisched networking infrastructure to relay the required Mithril information over the Cardano P2P network. The Mithril aggregator can then listen for specific Mithril messages on the Cardano P2P network without the need to penetrate the firewalls and infrastructure of the stake pool operators.
Beta Was this translation helpful? Give feedback.
All reactions