-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Cloudflare IP range importer #572
base: master
Are you sure you want to change the base?
Conversation
@mrchrisadams I think this PR is ready for review. I'm not sure how it can be tested or verified though. Will take your lead. |
hi @fershad ! If the importer works by fetching data, and then importing it, we could check by writing a test to run the importer and see if the provider, has the new IP ranges added. Are you cool with me adding a test along these lines to this PR? |
Thanks @mrchrisadams. Yes that kind of test does make sense. Please do add it, it would be a good reference to have for future importers as well. |
894c4f3
to
e7a5cd3
Compare
Old Energy EstimationEco-CI Output:
🌳 CO2 Data: |
Old Energy EstimationEco-CI Output:
🌳 CO2 Data: |
Hey @fershad - I've added:
|
Old Energy EstimationEco-CI Output:
🌳 CO2 Data: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi Fish! Thanks for making this! I can confirm this adds the IP ranges listed at the endpoints you listed.
I think this is good to merge in, but we'll likely need to make a change later to actually switch this on as I think it make sense to make sure we have a named technical contact to speak to first before running it on the regular.
@@ -14,3 +14,6 @@ source .venv/bin/activate | |||
dotenv run -- ./manage.py update_networks_in_db_amazon | |||
dotenv run -- ./manage.py update_networks_in_db_google | |||
dotenv run -- ./manage.py update_networks_in_db_microsoft | |||
|
|||
# TODO: activate once we have found our technical contact to talk to | |||
# dotenv run -- ./manage.py update_networks_in_db_cloudflare |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is the command we would run on weekly import like we do with AWS, GCP, and Microsoft
ipv6_data = ipv6_response.text | ||
|
||
# destructure the lines of each text file, to make one longer list | ||
return *ipv4_data.splitlines(), *ipv6_data.splitlines() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tweaked this to pull out the entries from file, so rather than returning two strings delimited by newlines like this:
ipv4_data, ipv6_data
We return a single list of strings, with each string representing an ipv4 or ipv6 address
*ipv4_data.splitlines(), *ipv6_data.splitlines()
from django.core.management.base import BaseCommand | ||
|
||
|
||
class Command(BaseCommand): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This command is what executed when we call update_networks_in_db_cloudflare
@@ -335,6 +335,9 @@ | |||
"GOOGLE_DATASET_ENDPOINT", default="https://www.gstatic.com/ipranges/cloud.json" | |||
) | |||
|
|||
CLOUDFLARE_PROVIDER_ID = env("CLOUDFLARE_PROVIDER_ID", default=None) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to set some default values here for the app to start up cleanly
Eco-CI Output:
🌳 CO2 Data: |
Fixes #569
This PR creates a new Importer for Cloudflare's public IP ranges. It fetches data from two text files which reflect the public IPv4 and IPv6 ranges that Cloudflare maintain.
TXT IPv4 list: https://www.cloudflare.com/ips-v4/#
TXT IPv6 list: https://www.cloudflare.com/ips-v6/#