Update existing lists #144
Conversation
|
This is quite a large diff. Happy to iterate if there are things you want to change. |
|
I'm not sure list patching can work reliably. A domain can be in another list instead of the previous list after blocklist update and would mess up the patching process. For example: We have two lists and the first one is full. If a new domain is added to the first list after the blocklist update, the last domain would end up being in the second list. We wouldn't be able to check its existence without going through all the current lists. Before blocklist update After blocklist update Things become even more complicated if the next update is from a different set of blockklists. Therefore, to make it work reliably, we can only go through all the current lists but that would defeat the purpose to save time and requests because there isn't an API to get all items of all the lists. |
|
This change does check all existing lists. It is a reliable operation and idempotent such that should an error occur, the same command can be used again to complete any missing changes. The benefits of this change are not only (slightly) faster operations, but avoiding any period where the lists or rules are unapplied which would leave the network and users vulnerable to accessing otherwise blocked hosts. The maximum requests in this flow is 2x number_of_lists (GET + PATCH to every list) in the worst case. The requests in the current flow is 2x number_of_lists (DELETE + POST to recreate every list) in the normal case. |
To avoid the need to delete all lists and recreate them, we can update existing lists only when their contents had changed. This processes the diffs between the desired list of domains and the existing lists. Removing entires that are no longer in the desired lists and appending any new entries. This prefers to minimize the number of PATCH calls by appending entries to the lists we're already patching for the removals. The priority for additions is: 1. Add to lists we're already patching for removals. 2. Add to existing lists with fewer than LIST_ITEM_SIZE entries. 3. Create a new list.
To avoid the need to delete all lists and recreate them, we can update existing lists only when their contents had changed.
This processes the diffs between the desired list of domains and the existing lists. Removing entires that are no longer in the desired list and appending any new entries. This prefers to minimize the number of PATCH calls by appending entries to the lists we're already patching for the removals.
The priority for additions is: