IP_ADDRESS_CACHE cache improvements #501
Conversation
rodpayne
commented
Mar 30, 2024
rodpayne
commented
- Only add entries to the cache if the lookup was successful. (The ones that fail may work next time.)
- Retain IP_ADDRESS_CACHE entries for up to four hours. (It takes almost a day to import a day's reports, so the lookups don't need to be extremely fresh.)
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #501 +/- ##
==========================================
+ Coverage 59.84% 59.93% +0.08%
==========================================
Files 12 12
Lines 1574 1575 +1
==========================================
+ Hits 942 944 +2
+ Misses 632 631 -1 ☔ View full report in Codecov by Sentry. |
| @@ -48,7 +48,7 @@ | |||
| MAGIC_XML = b"\x3c\x3f\x78\x6d\x6c\x20" | |||
| MAGIC_JSON = b"\7b" | |||
|
|
|||
| IP_ADDRESS_CACHE = ExpiringDict(max_len=10000, max_age_seconds=1800) | |||
| IP_ADDRESS_CACHE = ExpiringDict(max_len=10000, max_age_seconds=14400) | |||
There was a problem hiding this comment.
It may worth making this a config file setting, defaulting to the initial 1800, as different use-cases may warrant different values.
There was a problem hiding this comment.
Yes, making it a config file setting may be a good enhancement, but it may take me a while to figure out how to do it. I can take a look at it after the holiday.
I don't think that there are currently many people locked into the 1,800 setting since it was not working at all until last week.
There was a problem hiding this comment.
If someone would like to contribute a PR to add that option, be aware that although IP_ADDRESS_CACHE is only referenced twice:
https://github.com/domainaware/parsedmarc/blob/master/parsedmarc/__init__.py#L103
https://github.com/domainaware/parsedmarc/blob/master/parsedmarc/__init__.py#L897
configuration items related to report processing must go through several layers of function calls, such as when I just added the new function calls to address the concerns raised in #500.
- Actually save `source_type` and `source_name` to Elasticsearch and OpenSearch - Reverse-lookup cache improvements (PR #501 closes issue #498) - Update the included `dbip-country-lite.mmdb` to the 2024-03 version - Update `base_reverse_dns_map.csv` - Add new general config options (closes issue #500) - `always_use_local_files` - Disables the download of the reverse DNS map - `local_reverse_dns_map_path` - Overrides the default local file path to use for the reverse DNS map - `reverse_dns_map_url` - Overrides the default download URL for the reverse DNS map
