radb.net not returning CIDR data anymore

[HQJaTu]

This is related to #239.

I created a version which would support optional data sources for AS-information, but the change was not approved.

Example of a data query:

$ python3
Python 3.7.5 (default, Oct 17 2019, 12:21:00)
>>> from ipwhois import IPWhois, Net
>>> from ipwhois.asn import ASNOrigin
>>>
>>> ip = '216.108.228.108'
>>> asn = 26277
>>> net = Net(ip, allow_permutations=False)
>>> asn_query = ASNOrigin(net)
>>> asn_result = asn_query.lookup(asn='AS%d' % asn, asn_methods=['http'])
>>> asn_result
{'query': 'AS26277', 'nets': [], 'raw': None}

My version, querying with chosen data source of ipinfo.io. When calling python3, I force to use my own modified ipwhois-library. Also there is change in 1 line of code, the one where I choose asn_methods as my version prefers to know against which source the HTTP-query is done:

$ PYTHONPATH=../ipwhois/ python3
Python 3.7.5 (default, Oct 17 2019, 12:21:00)
>>> from ipwhois import IPWhois, Net
>>> from ipwhois.asn import ASNOrigin
>>>
>>> ip = '216.108.228.108'
>>> asn = 26277
>>> net = Net(ip, allow_permutations=False)
>>> asn_query = ASNOrigin(net)
>>> asn_result = asn_query.lookup(asn='AS%d' % asn, asn_methods=[ASNOrigin.ASN_SOURCE_HTTP_IPINFO])
>>> asn_result
{'query': 'AS26277', 'nets': [{'cidr': '216.108.224.0/24', 'description': 'Las Vegas NV Datacenter (PREMIANET)'}, {'cidr': '216.108.225.0/24', 'description': 'Las Vegas NV Datacenter (PREMIANET)'}, 

I still urge this multi-source option to be considered. When this worked last time against radb.net, it returned one (1) net for the above query. In above example, I'm omitting most of the reqonse, but reality is, for that query there are 64 CIDRs in response. Not 0. Not 1. Obviously, you can make 5 ASN-queries / day / IP-address on non-paid ipinfo.io API. This known limitation can be lifted by creating a paid account.

Please, reconsider my PR.

[secynic]

Please try the dev branch. Here is what I am seeing returned from RADB:
{'nets': [{'cidr': '72.18.192.0/20', 'description': 'Integra Telecom Route Object', 'maintainer': 'MAINT-AS7385', 'source': 'RADB', 'updated': '[email protected] 20080725'}, {'cidr': '216.108.224.0/20', 'description': 'Integra Telecom Route Object', 'maintainer': 'MAINT-AS7385', 'source': 'RADB', 'updated': '[email protected] 20080725'}, {'cidr': '64.235.32.0/19', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20040313'}, {'cidr': '64.235.32.0/20', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20041006'}, {'cidr': '64.235.48.0/21', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20041006'}, {'cidr': '64.235.56.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20041006'}, {'cidr': '64.235.61.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20041006'}, {'cidr': '64.235.62.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20041006'}, {'cidr': '64.235.63.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20041006'}, {'cidr': '64.235.58.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20041114'}, {'cidr': '72.18.195.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20050306'}, {'cidr': '72.18.200.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20050424'}, {'cidr': '72.18.201.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20050619'}, {'cidr': '72.18.206.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20050801'}, {'cidr': '72.18.207.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20050801'}, {'cidr': '72.18.204.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20050929'}, {'cidr': '72.18.205.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20050929'}, {'cidr': '72.18.202.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20051017'}, {'cidr': '72.18.203.0/24', 'description': 'Proxy-registered route object', 'maintainer': 'MAINT-AS3491', 'source': 'RADB', 'updated': '[email protected] 20051017'}, {'cidr': '64.235.32.0/19', 'description': 'REACH (Customer Route)', 'maintainer': 'MAINT-REACH-NOC', 'source': 'REACH\nchanged: [email protected] 20170213', 'updated': '[email protected] 20170213</code></pre>'}, {'cidr': '72.18.192.0/20', 'description': 'REACH (Customer Route)', 'maintainer': 'MAINT-REACH-NOC', 'source': 'REACH\nchanged: [email protected] 20170213', 'updated': '[email protected] 20170213</code></pre>'}, {'cidr': '216.108.224.0/20', 'description': 'REACH (Customer Route)', 'maintainer': 'MAINT-REACH-NOC', 'source': 'REACH\nchanged: [email protected] 20170213', 'updated': '[email protected] 20170213</code></pre>'}], 'query': 'AS26277', 'raw': None}

[RoboJaTu]

Dev-branch does work and returns valid data. The only minor thing is, that RADb returns only top-level network information. IPinfo.io will return all the /24 networks summing up to /19. What I typically do, is combine the networks to get what RADb will return directly. So, this is a non-issue to me.

Ultimately, I can use this version in my application. Thank you!

[abdullahdevrel]

I work for IPinfo, but I am an outsider to the project. Feel free to let me know if you have any questions about our data.

I understand that we do return ASN CIDR data only in our paid tier API services, but to address this, we do have one of our free databases to assist. The IP to ASN free database does provide range-level information. The IP to ASN database is created with a combination of WHOIS and peering records from BGP, so it is as accurate as it can get. The challenge, however, is that ranges are aggregated to reduce the file size, so you will not get parent CIDR data if the CIDR ranges are neighboring.

I would recommend playing with the database for a bit to see if it resolves the issues you are facing. Thanks!

[HQJaTu]

IPinfo.io has changed their policy. This change tightens web-UI usage, especially from library calls and removes ASN-queries from non-paid accounts.

However, at same time they do publish ASN database as a CSV-file. I have implemented support for this in my own fork: https://github.com/HQJaTu/ipwhois/blob/ipinfo.io/ipwhois/asn.py#L84

[abdullahdevrel]

I personally would not say "tighten," but we are looking for alternative solutions to some of the challenges we face. The idea is to provide a good set of free and high-quality data to the open-source community in the easiest way possible.

If you look at the licensing terms for the free database, it is CC-BY-SA 4.0, which permits commercial usage and distribution with attribution to IPinfo. It is the most lenient licensing term for a free database, regardless of the superior accuracy you are getting.

We are happy to support the open-source community and developers unconditionally, ensuring the data they receive is free and is unparallel in its quality. We do not actually have a monetization plan for open-source projects. Considering our free API has a rate limit and we charge for increasing rate limits, we launched a database that essentially now provides an infinite number of queries.

[secynic]

I'll take a look at the DB and see if it is something I can add support for without distributing it.

[abdullahdevrel]

I'll take a look at the DB and see if it is something I can add support for without distributing it.

Feel free to incorporate a repo specific access token that downloads the database on the user's machine periodically.