Identifying unmaintained/unmanaged/abandoned assets using expired certificates and home page content analysis

[iamthefrogy]

I use httpx for mass website scanning and also for creating a Configuration Management Database (CMDB) for my personal use. One of the use cases from an Attack Surface Management (ASM) point of view is to identify abandoned, unmaintained, or unsupported websites within a large target. For example, a company like Tesla might have thousands of subdomains and over 500 apps on the Internet. One of the challenges for large organizations is maintaining an accurate CMDB, dealing with the dynamic nature of work procedures, and addressing visibility issues when teams deploy something on the internet without the infosec team's oversight.

Sometimes, teams deploy applications on the Internet and forget to decommission them properly after they are no longer in use. One of the biggest problems for large companies is finding abandoned or unsupported apps facing the Internet that are no longer in use, but require cleanup activities. If this is not done, these apps could be prone to subdomain takeover or could be using obsolete technology that is prone to exploitation.

I was wondering if httpx could provide a way to identify abandoned or unmaintained assets on the Internet out of all the apps it has found, using a single switch. The logic I was thinking of is as follows:

Mandatory:

• Certificates expired more than 6 months ago.
• Certificates expired more than 1 year ago.

Optional:

• Many broken links on the website.
• The home page is the default page of the web server.
• The home page has no links.
• The home page is blank.
• The home page has a default error (client or server-side error).
• The site relies on technology that is no longer supported or considered secure, such as Flash.

If at least 1 of the mandatory criteria and 2 of the optional criteria are matched, that asset is identified as a potential abandoned or unmaintained asset.

For the mandatory criteria, you would use the -tls-grab option, but I am not sure whether httpx can identify certificates that have expired more than 6 months or 1 year ago, or if users can search for all apps with expired certificates.

What do you think of the above logic and the idea of expanding httpx functionalities?

[GeorginaReeder]

Thanks so much for this @iamthefrogy - we'll definitely take a look into it, we appreciate you starting this issue!