This repository has been archived by the owner on Dec 1, 2021. It is now read-only.
Fixes for Issue #5 #6
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix for Issue #3 blacklist.sh
Updated the script to put the blacklist creation steps into a function so that an if, then, elif, then, else statement could be created. This new series of checks determines if the blacklist backup file exists on a reboot. If it does, it restores that backup list to speed up provisioning. If it does not then it creates a new list on reboot to ensure the device is protected rather than waiting until the next scheduled interval to process and create the list. The list will still be processed and updated at the scheduled interval as well. Additional logging is included to state whether the list is being restored from a backup or processed as new.
Added additional code to do a list comparison and write the results to the log file. This will show changes between each run of the script on what IPs were added or removed from the new list created by the script. This comparison will not run on a reboot of the USG to reduce provisioning time. Also changed "FireHOL" back to "Dynamic Threat List" to make sure @brontide instructions continue to work with this script.
Minor verbiage changes to logging messages.
More logging to track if no changes were made and also logging total changes made in comparison of previous list to new list during script execution.
Updated readme to include step from @bloqhed to create symbolic link so script will run on USG reboot as well as scheduled interval.
Another minor logging change to include total additions and removals from list. Also changing list name to "FireHOL" instead of "Dynamic Threat List" as FireHOL is where all my lists come from, and @brontide has archived his repo so the merge will not happen.
Updated "Dynamic Threat List" to "FireHOL" to be consistent with my script. Added in parentheses the firewall rule names shown in the 6.2.x and above controller firewall area for clarification.
Minor code efficiency update to consolidate if statements in list comparison code.
Clarification on config.gateway.json time intervals.
Renaming file from blacklist.sh to blocklist.sh. This update also includes all references inside the file to blacklist are now updated to blocklist. Additional logging directed to console by using echo so if script is run manually you have a better idea of what step it is on and what it is doing.
Remove file as it is now titled blocklist.sh
update references from blacklist to blocklist as the filename has changed.
update blacklist to blocklist and adjust interval to 12h from 24h
Added if [ -e $backupexists ]; then backupexists="TRUE"; else backupexists="FALSE"; fi. Adjusted if [ "$usgupt" != "min," ] && [ -e $backupexists ] test to if [ "$usgupt" != "min," ] && [ "$backupexists" == "TRUE" ]. Running the test the other way the [ -e ] portion was always defaulting to true even if the file did not exist which was causing the change comparison to run when it should not. The change comparison should only run if a previous backup file exists indicating a prior successful run of the script, and the USG uptime is over 59 minutes.
Disable previous to current list compare. Not really worth the extra processing time and CPU load.
Updated URLs for iplists as FireHOL github repository is no longer being updated. URLs now pull directly from firehol.org site which is still getting updated.
Removed FireHOL Level 3, too many false positives for me in the list.
Updated blocklist URLs
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I have updated the code to include logging and backing up the temporary list to the /config/scripts folder. I've also put in a few if statements. If the user creates a symbolic link as suggested by @bloqhed then the provisioning of the USG can take some time to complete as the entire script has to run before the USG is available again in the controller. I cannot find a way to get the crontab-spec @reboot to work. I've tried to trigger very simple scripts with this crontab-spec and they all fail to run, so bloqhed's idea of creating the symbolic link is how I'm getting the blacklist created at reboot. The first if statement will load the last known version of the blacklist backup file upon reboot to save time during provisioning. It accomplishes this by using the uptime command to determine if the uptime is equal to minutes. If the uptime is not equal to minutes then the script begins to process and create a new blacklist. You can see I've updated the URLs to use FireHOL. These people are great and the lists I'm using from them are already aggregates of the lists you were using plus many more. You can check them out here https://github.com/firehol/blocklist-ipsets, http://iplists.firehol.org/, https://firehol.org/. I've noticed an issue that if the for loop in the script fails to process and create the new blacklist for whatever reason it will still swap the temp and real lists thereby leaving you with an empty blacklist at the end of the process, which defeats the purpose. I've created a second if statement to determine if the temp blacklist does in fact contain items. If it does then that temp blacklist is backed up then swapped for the real list. This backup is then used at reboot of the USG, and also anytime the script fails to refresh/create the new blacklist during processing. Now you will always have a populated blacklist after it is created successfully the first time. I've also removed all sudo references from the script as when it is run by cron on the scheduled interval it is run by the root user so sudo is unecessary. If the end user wants to run the script manually they will need to enter sudo /config/scripts/blacklist.sh from the cli to make sure it executes properly. This will also ensure the blacklist is created immediately and that you have a valid blacklist backup file in case of future issues in script processing of a new blacklist. I've also updated the ipset -q add command in the for loop to ipset -! add which ensures only unique entries end up in the blacklist plus any errors are ignored rather than suppressed so the script will move on without exiting. Thank you so much for creating this script, it was exactly what I was looking on how to do for my USG!