Modernize Virustotal API scraping

[theteleforce]

Address #194 by modernizing the Virustotal scraper.

Some time in the past few years, Virustotal added a few verification methods to the subdomain API. This pull request changes the endpoint and adds headers so that the request succeeds.

This solves the same problem as #285, but does not require an API key and uses the existing parser logic.

This pull request has been tested on Python 3.9.2, Kali Linux rolling 2021/2.

[its0x08]

Did almost the same thing for VirusTotal without using API key!
👍🏻

[famdude]

It worked. Tnx!

[ko2sec]

works fine, thanks for the fix.

[OMGd0g]

I'm still learning to Linux here and I'm in douobt which file to change... When doing a search i find it at 2 locations
/usr/lib/python3/dist-packages/sublist3r.py
/usr/lib/python3/dist-packages/theHarvester/discovery/sublist3r.py
Should I edit both for this fix to work?

Thanks :)

[ko2sec]

I'm still learning to Linux here and I'm in douobt which file to change... When doing a search i find it at 2 locations

/usr/lib/python3/dist-packages/sublist3r.py

/usr/lib/python3/dist-packages/theHarvester/discovery/sublist3r.py

Should I edit both for this fix to work?

Thanks :)

theHarvester is another tool, you need to edit /usr/lib/python3/dist-packages/sublist3r.py

[wvdongen]

The fix works, merge it!

[Defirence]

Can we get this merged please? cc: @aboul3la

[Defirence]

@AetherBreeze self.headers["Accept-Ianguage"] should this not be corrected to self.headers["Accept-Language] ? My guess is this might cause an undefined variable error.

[xmegax]

it worked , thanks

[firingdove]

Address #194 by modernizing the Virustotal scraper.

Some time in the past few years, Virustotal added a few verification methods to the subdomain API. This pull request changes the endpoint and adds headers so that the request succeeds.

This solves the same problem as #285, but does not require an API key and uses the existing parser logic.

This pull request has been tested on Python 3.9.2, Kali Linux rolling 2021/2.

I am new to linux and using sublist3r. please it possible to provide a step by step instruction on how I can solve the problem. Because I tried using the API key and is not working. thanks

[firingdove]

it worked , thanks

Please How did you do it?

[xmegax]

it worked , thanks

Please How did you do it?

use this repo.
https://github.com/AetherBreeze/Sublist3r.git

[firingdove]

it worked , thanks

Please How did you do it?

use this repo. https://github.com/AetherBreeze/Sublist3r.git

thanks Man. it is working.

[Defirence]

Can we merge this change or is this project now considered being in "development hell"? If it works, WHY can't we commit this? Hello?

EDIT: Looking at @aboul3la 's front page it seems they're either a: working full-time b: dead (hopefully not) or c: this project has been abandoned.

I'm going with the latter being c: Project maintenance has been abandoned. This is a great tool, sad to see even PR's are not being merged. If anyone is looking for a working version that respects the new changes use @AetherBreeze 's forked repo here:

https://github.com/AetherBreeze/Sublist3r

[panchal2001]

working as good

[panchal2001]

Can we merge this change or is this project now considered being in "development hell"? If it works, WHY can't we commit this? Hello?

EDIT: Looking at @aboul3la 's front page it seems they're either a: working full-time b: dead (hopefully not) or c: this project has been abandoned.

I'm going with the latter being c: Project maintenance has been abandoned. This is a great tool, sad to see even PR's are not being merged. If anyone is looking for a working version that respects the new changes use @AetherBreeze 's forked repo here:

https://github.com/AetherBreeze/Sublist3r

Error: Virustotal probably now is blocking our requests
[!] Error: Google probably now is blocking our requests
[~] Finished now the Google Enumeration ...

[aleeshaford]

maybe it is possibility to decompile the inofficial VirusTotal Mobile app to have again a rate limit free scraping method without api key.

i haven't gotten that far yet, but it seems like the particular "partner" is calling urls to scan

[anurag-sachan]

works fine for me, thanks brother!

[Morgoth04]

Address #194 by modernizing the Virustotal scraper.
Some time in the past few years, Virustotal added a few verification methods to the subdomain API. This pull request changes the endpoint and adds headers so that the request succeeds.
This solves the same problem as #285, but does not require an API key and uses the existing parser logic.
This pull request has been tested on Python 3.9.2, Kali Linux rolling 2021/2.

I am new to linux and using sublist3r. please it possible to provide a step by step instruction on how I can solve the problem. Because I tried using the API key and is not working. thanks

Almost 4 years late, sorry:

1. The url line in the class corresponding to Virustotal (class Virustotal(enumratorBaseThreaded):) of the sublist3er.py program must be corrected (base_url = 'https://www.virustotal.com/api/v3/domains/{domain}/subdomains').
   2)2) Place the apikey that was obtained once registered as a virustotal user in the variable sel.apikey (self.apikey = os.getenv('VT_APIKEY', 'hereenterthealphanumeric code'))
   3)Save the file
2. Overwrite the sublist3r.py file in the Kali Linux path where the one you have was placed
3. Try with the command sudo ./sublist3r.py -d domainchosen.com