Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data exfiltration via DNS tunneling #4036

Open
TonyWildish-BH opened this issue Jul 17, 2024 · 26 comments
Open

Data exfiltration via DNS tunneling #4036

TonyWildish-BH opened this issue Jul 17, 2024 · 26 comments
Labels
bug Something isn't working has workaround a workaround is available for this issue

Comments

@TonyWildish-BH
Copy link

Describe the bug
We have a guy doing penetration testing for us on our TRE, and he’s discovered that the DNS configuration is vulnerable, allowing DNS tunneling. He’s successfully exfiltrated data with iodine.

We tried enabling Defender for DNS, which it turns out doesn’t exist anymore, so I enabled Microsoft Defender for Servers Plan 2 instead, which we thought might give us the same protection. It doesn’t, he was still able to exfiltrate data.

Is there any MS service we can enable that will prevent DNS tunneling from the TRE?

Steps to reproduce

Attacker-side:

  1. Register a domain and create a VM on the Internet
  2. Setup an NS DNS record to point towards the VM
  3. Deploy Iodine server (10.0.0.1)

On the Workspace VM (only tested Linux at the moment)

  1. Download iodine client via apt on Ubuntu
  2. Establish a DNS tunnel using Iodine to the server
  3. An SFTP session was then able to exfiltrate 100 MB in about 2 and a half hours.

Azure TRE release version (e.g. v0.14.0 or main):
main, as of about May.

Deployed Azure TRE components - click the (i) in the UI:
UI Version: 0.5.21
API Version: 0.18.5

@TonyWildish-BH TonyWildish-BH added the bug Something isn't working label Jul 17, 2024
@tim-allen-ck
Copy link
Collaborator

We have come across this before, one approach is to create an allow list of packages on Nexus, so that only approved tools can be installed.

We will do some investigation to see if this can be blocked at the network level.

@TonyWildish-BH
Copy link
Author

Blocking packages at Nexus won't solve this problem, it's easy enough to get the necessary client set up to allow DNS tunneling by other means.

Cutting off network access completely is one option, but that will make the TRE too hard to use for most users.

We need a solution that blocks this at the network level. I thought Defender for DNS would do this, but it seems not to exist anymore.

@tim-allen-ck
Copy link
Collaborator

@jonnyry
Copy link
Contributor

jonnyry commented Jul 18, 2024

We tried enabling Defender for DNS, which it turns out doesn’t exist anymore, so I enabled Microsoft Defender for Servers Plan 2 instead, which we thought might give us the same protection. It doesn’t, he was still able to exfiltrate data.

Was it perhaps sending alerts, rather than straight out blocking the connection?

There's some documentation here for it although its rather basic: https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-dns-introduction

@TonyWildish-BH
Copy link
Author

@jonnyry , Defender for DNS appears to be deprecated, but the documentation is stale, yes. With Defender for Servers Plan 2 we do get alerts, so we can react, but not prevent the exfiltration from happening in the first place.

@tim-allen-ck , thanks for the link, but that appears to be monitoring, not prevention?

@marrobi
Copy link
Member

marrobi commented Jul 18, 2024

The only way I am aware of to prevent DNS tunnelling, is to have a DNS server with an allow list of domains.

There is no Azure service that I am aware of that does this is out of the box.

One possible solution would be to use an Azure DNS Private Resolver, with a wildcard to route unknown domain names to a bad IP. Not sure if this would work, but worth looking at.

Another option would be a solution like Adguard Home (not sure if there are other corporate solutions) that have a list of rules to allow/deny that support regex or similar.

The workspace template VNet would need to be amended to use the custom DNS server.

Ultimately there is a risk/usability discussion which comes down to the organisation stance on this. If think about "Safe People", you could also say the Airlock is an exfiltration risk as the Airlock Manager could approve the export of PII (intentionally or accidentally).

I do think having an allow list on Nexus would be a better default for the OSS project, but it has been debated in the past and we have left it to implementers of the solution to decide.

@SvenAelterman
Copy link
Collaborator

Indeed, in this case, both Defender for Servers and Sentinel would detect, but not prevent. They're still critically important aspects of a comprehensive risk reduction strategy. Additionally, the response to those alerts can be automated. For example, the offending IP can be completely blocked by the firewall.

It should be noted (and perhaps that's what Tony hinted at here) that even allow listing packages isn't going to cure all. I suspect you'll need to allow Python on the systems so that data analysis can be performed. I am not a Python developer, but I suspect a malicious insider could write Python code to perform DNS tunneling.

Configuring your EDR solution on the endpoints might be the next best step. The EDR might be able to prevent tools or maliciously code like that from executing.

Solutions could also include placing a third-party firewall with advanced DNS security upstream of the TRE firewall.

Marcus has provided several other feasible options.

What's the recommendation of your pen tester for blocking DNS tunneling?

@jemrobinson
Copy link

Shameless plug for a project I work on: The Alan Turing Institute Data Safe Haven which is also an Azure-based open-source TRE. We solve this problem by using Adguard Home to provide a DNS server inside the TRE which only allows connections to a pre-approved allowlist of domains.

@TonyWildish-BH
Copy link
Author

thanks for the replies. I'm aware that there's a tradeoff, we need to decide our appetite for risk, that's always going to be the case, but that's not the only consideration.

We're deploying the TRE for use with very sensitive data, and we have a responsibility to show that we're taking reasonable steps to protect it. I'm not overly surprised that such a problem exists in the TRE, but I'm disappointed that such known limitations aren't documented anywhere, I would have expected to find this mentioned in the docs. That would have allowed us to look into a resolution earlier, or possibly into other solutions.

Are there any other major security holes that you know of that aren't documented?

@SvenAelterman, we don't have the full report from our pen tester yet, but it's clear we can't go live with this problem.

@jemrobinson, I'd love to have a chat about how you implemented your solution. I've sent you a contact request via LinkedIn, can we pick up the conversation from there.

@marrobi
Copy link
Member

marrobi commented Jul 18, 2024

One solution could be to create an adguard home shared service.

Configure Vnets to use Azure DNS private resolver, which forwards to Azure Firewall as the DNS proxy, that proxies to ad guard.

Then if we find an alternative solution, or customers have a comercial offering, it's a case of just amending the Firewall proxy.

@SvenAelterman thoughts?

@marrobi
Copy link
Member

marrobi commented Jul 18, 2024

Ok, think I've got an easier fix, maybe not as as pretty as adguard home, but if create a private dns resolver and ruleset, link it to the workspace VNet you can restrict what external domains can be resolved. It still resolves privatelink addresses before going to the list of rules - this is why need to use a private resolver.

image

image

So suggest create a resolver in core, needs a dedicated subnet for outbound traffic, and link to the workspace VNets.

@SvenAelterman
Copy link
Collaborator

@marrobi I agree, definitely a much more supported solution because you can send the diagnostic logs to the Log Analytics Workspace, it's Azure native, etc.

You probably still should have the spokes send their DNS queries via the Firewall proxy and the Firewall should use the DNS private resolver.

@marrobi
Copy link
Member

marrobi commented Jul 19, 2024

Will the private endpoints attacked to the workspace VNet work that way?

@TonyWildish-BH
Copy link
Author

@marrobi, thanks for the pointers. I'm doing a PoC for this, one thing I haven't figured out is how to configure the rules to use the Azure internal DNS servers. Any suggestions?

It all works fine pointing at 8.8.8.8, but I'd rather not use Google's DNS servers if I can use the internal Azure DNS servers instead.

@marrobi
Copy link
Member

marrobi commented Jul 31, 2024

Absolutely - https://learn.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16

My open questions are:

  1. Can we do this with a single Private DNS resolver in core, or does that break private DNs zones only linked to the workspace VNet, such as azure monitor.

  2. How do we maintain list of allowed FQDNs, for example we use service tags for many of the workspace services. We likely need a set of top level domains but also the ability to add new ones as required. This could be similar to how firewall rules are added by workspace services.

@TonyWildish-BH
Copy link
Author

thanks for the quick reply, @marrobi. I've seen that page, but I don't think it helps. I tried putting 168.63.129.16 instead of 8.8.8.8 in my rules, and it's forbidden, the portal won't let me do it. Have I missed something?

@marrobi
Copy link
Member

marrobi commented Jul 31, 2024

Hmm. Maybe the resolver doesn't have access to the vnet. I can give it a go tomorrow. So you have a corporate DNS server? Maybe makes sense to use that. We could make it an option in the config.

@SvenAelterman might have some knowledge that helps.

@marrobi
Copy link
Member

marrobi commented Jul 31, 2024

Might be need an outbound endpoint into a vnet. I'll try figure it out tomorrow.

@marrobi
Copy link
Member

marrobi commented Aug 1, 2024

Been thinking about this more, if we have a private resolver in each workspace network, using the Firewall as a DNS proxy (requires standard SKU), does the firewall automatically block all other DNS traffic unless add a network rule for the FQDN? This would be easier to maintain as FQDNs already exist on the firewall.

https://learn.microsoft.com/en-us/azure/firewall/fqdn-filtering-network-rules

We need to look into the two options.

@marrobi
Copy link
Member

marrobi commented Aug 8, 2024

168.63.129.16

So seems like this isn't allowed. If you enable the DNS proxy on the firewall, then point the DNS resolver rule to the Firewall IP, this works. As Sven says you can also enable DNS logging on the firewall to monitor the DNS requests.

Still looking to see if we can do this on the firewall itself so we get Service Tag support, rather than on the DNS resolver.

@marrobi
Copy link
Member

marrobi commented Aug 9, 2024

Ok, have a configuration that works.

  • Firewall deployed with standard sku
  • Enable DNS proxy on the firewall
  • Deploy DNS resolver with outbound endpoint in the core VNet (created a new subnet)
  • Create new ruleset with allowed DNS suffix going to firewall IP and a all for . going to 0.0.0.0 (bit of a bodge, but works)
  • Link ruleset to workspace VNet

This way can have a centrally managed list, suggest deploy as part of the firewall shared service as can be updated in a similar way with FQDNs required by services.

@marrobi marrobi added the has workaround a workaround is available for this issue label Aug 9, 2024
@TonyWildish-BH
Copy link
Author

thanks Markus, I'll give that a go.

@marrobi
Copy link
Member

marrobi commented Aug 29, 2024

Hi @TonyWildish-BH, did you successfully make the changes? Any plans to provide a PR back to the project? Thanks.

@TonyWildish-BH
Copy link
Author

Hi @marrobi. Almost completed, now in final testing, I'll provide a PR when I'm sure it's working properly.

One thing I realised is that the whitelist can (probably) be completely empty, i.e. we can sinkhole everything. Then the VM can only communicate with the Azure services and other services in the TRE, such as Nexus etc. That seems to work in the limited testing I've done so far.

The only thing that would make that not true is if certain applications rely on being able to at least look up the name of a server. They may be able to survive not being able to connect to it, but may spit the dummy if they can't even resolve the name. I don't know of any that do that, but it's possible.

The nice thing about sinkholing everything is that I don't have to rely on the Google DNS servers for the whitelist, I haven't found a way to use MS DNS servers, as noted above.

@marrobi
Copy link
Member

marrobi commented Aug 29, 2024

You can use the Firewall IP as the DNS server as DNS proxy is enabled, and that will in turn use Azure DNS.

Some DNS names will need to be able to resolve - for example to access Azure storage it will need to access public DNS which then returns a privately resolvable DNS name.

Maybe you can share a draft PR and we can test it out and offer feedback?

@marrobi
Copy link
Member

marrobi commented Nov 28, 2024

There is now a service to help with this in preview

https://learn.microsoft.com/en-us/azure/dns/dns-security-policy

https://learn.microsoft.com/en-us/azure/dns/dns-traffic-log-how-to

If someone is able to give this a go, we can look to get it integrated into the project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working has workaround a workaround is available for this issue
Projects
Development

No branches or pull requests

6 participants