hashlookup-forensic-analyser version 1.3 - including Bloom filter improvements and bugs fixed

hashlookup-forensic-analyser version 1.3 - including Bloom filter improvements and bugs fixed. You can now specify the hash algorithm used for the Bloom filter sets.

Thanks to Jens Hubler for the contributions and discussions.

hashlookup-forensic-analyser analyses a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service.

v1.3 (2023-09-24)

Changes

• [doc] updated that all copyright owners are also the contributors. [Alexandre Dulaunoy]

• [cmd] update doc. [Alexandre Dulaunoy]

• [doc] updated. [Alexandre Dulaunoy]

• [Bloom filter] a new option added for non-hashlookup BF. [Alexandre Dulaunoy]

  A new option has been added --bloomfilters-lower-case to
  support now standard Bloom filter.

  Based on discussion from pull-request #15

Fix

• [live-linux] missing format [Alexandre Dulaunoy]

• [bloomfilter] improved PR #16. [Alexandre Dulaunoy]

Other

• Merge branch 'Hu6li-multiple_hash_support' into main. [Alexandre Dulaunoy]

• Support BF with lower and/or upper case hashes. [Jens Hubler]

hashlookup-forensic-analyser version 1.2 - bug fix release

hashlookup-forensic-analyser version 1.2 - bug fix release (2023-05-29)

hashlookup-forensic-analyser analyses a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service or offline with Bloom filters. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

Changes

• [workflow] remove a commercial scanner which is basically black-mailing you if you don't use the paid version. [Alexandre Dulaunoy]

• [doc] New year - updated copyrights. [Alexandre Dulaunoy]

• [doc] add reference to hashlookup.io. [Alexandre Dulaunoy]

Fix

• [hashlookup] set a default timeout for requests [Alexandre Dulaunoy]

Other

• Merge pull request #13 from Hu6li/fix-bloom-arg. [Alexandre Dulaunoy]

  Fixing typo

• Fixing typo. [Jens Hubler]

• Update README.md. [Alexandre Dulaunoy]

• Merge branch 'main' of github.com:hashlookup/hashlookup-forensic-analyser into main. [Alexandre Dulaunoy]

• Set theme jekyll-theme-cayman. [Alexandre Dulaunoy]

hashlookup-forensic-analyser version 1.1 released

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service or the Bloom filter from CIRCL hashlookup. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

This release includes the ability to load multiple Bloom filters at the same time. Thanks @wllm-rbnt for the contribution.

hashlookup-forensic-analyser version 1.0 released

A new version of the hashlookup-forensic-analyser has been released including a new --live-linux option to verify running process on a Linux machine if these are known or unknown binaries on hashlookup service.

This version 1.0 is considered as stable following the feedback received by the different users. Don't hesitate to provide feedback or ideas by opening issues.

New --live-linux option

python3 hashlookup-analyser.py --cache --print-unknown --live-linux. The option is actually reading the /proc directory to find all running processes and due to the fact the ./exe file in proc directory is the actual file, you can quickly find the known/unkown files from the running processes.

Complete changelog available below:

v1.0 (2022-05-07)

New

• [progress] disable progress option --disable-progress as progress is by default nowadays. [Alexandre Dulaunoy]

• [live_linux] new --live-linux option to display known and unknown processes running from a running linux instance. [Alexandre Dulaunoy]

  The process hashes are read from the /proc/PID/exe which is the actual
  file running before being loaded in memory.

Changes

• [lint] increase code complexity. [Alexandre Dulaunoy]

• [code] black -S [Alexandre Dulaunoy]

• [hash] skip bandit check on weak hash. [Alexandre Dulaunoy]

• [hashlib] the library is not used for security per se but for forensic and use SHA1 as the lookup service gives more potential results in SHA1 (due to old NSRL dataset) [Alexandre Dulaunoy]

• [file] black -S [Alexandre Dulaunoy]

• [doc] updated with new options. [Alexandre Dulaunoy]

Fix

• [action] remove mypy too many FP ref -> python/mypy#12246. [Alexandre Dulaunoy]

hashlookup-forensic-analyser version 0.9 released with MIME type statistics

hashlookup-forensic-analyser version 0.9 released with MIME type statistics

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service or the Bloom filter from CIRCL hashlookup.

This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

For the curious, 📖 A sample summary report is available.

New

• [report] add MIME type statistics. [Alexandre Dulaunoy]

• [feature] stats about mime_type added. [Alexandre Dulaunoy]

Changes

• [doc] fix the sample report link. [Alexandre Dulaunoy]

• [hashlookup-forensic-analyser] add mermaid pie chart for mime types. [Alexandre Dulaunoy]

• [doc] sample report updated. [Alexandre Dulaunoy]

• [doc] sample report updated. [Alexandre Dulaunoy]

• [requirements] filemagic added. [Alexandre Dulaunoy]

Fix

• [hashlookup-analyser] blake -S [Alexandre Dulaunoy]

• [hashlookup-analyser] add missing type annotation. [Alexandre Dulaunoy]

Other

• Set theme jekyll-theme-dinky. [Alexandre Dulaunoy]

hashlookup-forensic-analyser version 0.8 released including a report functionality

hashlookup-forensic-analyser version 0.8 released including a report functionality

A new --report option added to generate a report directory including a markdown summary and a JSON export of the results. A sample report in Markdown is available. The JSON includes all the found and unknown files discovered.

New

• [hashlookup] --report option added to generate a report directory including a markdown summary and a JSON export of the results. [Alexandre Dulaunoy]

Changes

• [doc] cleanup. [Alexandre Dulaunoy]

• [doc] add sample gist page of a summary report. [Alexandre Dulaunoy]

Fix

• [hashlookup] fix headline report typo. [Alexandre Dulaunoy]

• [hashlookup] shadowed variable removed. [Alexandre Dulaunoy]

• [hashlookup] shadowed loop variable removed. [Alexandre Dulaunoy]

Other

• Update README.md. [Alexandre Dulaunoy]

hashlookup-forensic-analyser version 0.7 released including various bugs fixed

hashlookup-forensic-analyser version 0.7 released including various bugs fixed

Changes

• [hashlookup] if a file cannot be read (e.g. Permission denied) for hashing. [Alexandre Dulaunoy]

  Those are skipped, accounted in the nonanalysed files.

• [ci] updated. [Alexandre Dulaunoy]

Fix

• [bug] better handling of filename and CSV export (fix issue with comma separated filenames) [Alexandre Dulaunoy]

hashlookup-forensic-analyser version 0.6 released including various bugs fixed and improved logging

hashlookup-forensic-analyser version 0.6 released including various bugs fixed and improved logging

usage: hashlookup-analyser.py [-h] [-v] [--extended-debug] [--progress] [-d DIR] [--print-all] [--print-unknown] [--include-stats] [--format FORMAT] [--cache] [--bloomfilter BLOOMFILTER]

Analyse a forensic target to find and report files found and not found in hashlookup CIRCL public service.

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Verbose output
  --extended-debug      Debug file processed along with the mode and type.
  --progress            Pring progress of the file lookup on stderr.
  -d DIR, --dir DIR     Directory to analyse
  --print-all           Print all files result including known and unknown
  --print-unknown       Print all files unknown to hashlookup service
  --include-stats       Include statistics in the CSV export
  --format FORMAT       Output format (default is CSV)
  --cache               Enable local cache of known and unknown hashes in /tmp/hashlookup-forensic-analyser
  --bloomfilter BLOOMFILTER
                        Specify filename of a bloomfilter in DCSO bloomfilter format

New

• [hashlookup] --progress option to display the number of files analysed, excluded, unknown and found in hashlookup. [Alexandre Dulaunoy]

  and a completely useless spinner added too.

  This fixes #7

Changes

• [hashlookup] enable the spinner by default and log on stderr. [Alexandre Dulaunoy]

• [doc] README updated with new options. [Alexandre Dulaunoy]

• [bin] requests missing. [Alexandre Dulaunoy]

• [import] order is important. [Alexandre Dulaunoy]

• [workflow] long lines again. [Alexandre Dulaunoy]

• [formatting] black -S --target-version py38 [Alexandre Dulaunoy]

• [hashlookup] --extended-debug option added to output the filetype. [Alexandre Dulaunoy]

  only useful for debugging

• [LICENSE] added. [Alexandre Dulaunoy]

Fix

• [bin] typo fixed. [Alexandre Dulaunoy]

• [hashlookup] exclude char and block device - catch incorrect fstat on size. [Alexandre Dulaunoy]

• [chg] excluded named pipe - FIFO. [Alexandre Dulaunoy]

hashlookup-forensic-analyser version 0.5 released including bloom filter support and caching

hashlookup-forensic-analyser version 0.5

Bloom filter support

If you don't want to share your lookups online and do faster lookup, hashlookup provides a bloom filter to download.

The file is around 700MB and can be stored locally in your home directory. hashlookup-analyser works in the same way, --bloomfilter option allows to specify the filename locatoon of the bloom filter.

python3 bin/hashlookup-analyser.py --bloomfilter <yourdir>/hashlookup-full.bloom --include-stats -d /bin

v0.5 (2021-12-14)

New

• [doc] add a quick documentation about the bloom filter feature. [Alexandre Dulaunoy]

• [bloomfilter] use a bloomfilter source instead of the live request. [Alexandre Dulaunoy]

  adulau@kolmogorov ~/git/hashlookup-forensic-analyser/bin (main)$ python3
  hashlookup-analyser.py --cache -d /home/adulau/sample/ --print-unknown
  --bloomfilter /tmp/x.bloom --include-stats
  hashlookup_result,filename,sha-1,size
  unknown,/home/adulau/sample/xxx,278DFA419DC12FB8CD9785A873956885BBE11779,17
  unknown,/home/adulau/sample/vi,DFA8A9B6417CE3C43D91B0E69E68C26A0230DD48,3194152
  stats,Analysed directory /home/adulau/sample/ on kolmogorov running
  Linux-5.10.0-1052-oem-x86_64-with-glibc2.29 at 2021-12-06
  06:48:56.892009+00:00- Found 1 on hashlookup.circl.lu
  (hashlookup-blomfilter)- Unknown files 2 - Excluded files 0

• New: [gh] based on the pull-request #6 from @signorrayan and feedback from @sthagen. [Alexandre Dulaunoy]

Changes

• [workflow] we have big screens nowadays. [Alexandre Dulaunoy]

• [hashlookup] black -S --target-version py38 [Alexandre Dulaunoy]

• [doc] --bloomfilter option added. [Alexandre Dulaunoy]

• [doc] updated with new --cache option. [Alexandre Dulaunoy]

• [bin] CSV now includes SHA-1 of the files. [Alexandre Dulaunoy]

• [bin] new --cache feature added. [Alexandre Dulaunoy]

  if cache is called, this will create a file per hash in a fixed
  directory. at each run, if --cache is called, it will check the
  existence of a file. The directory contains the JSON result saved
  from the previous hashlookup query of the server.

• [README] a status badge added. [Alexandre Dulaunoy]

• [bin] make isort happy. [Alexandre Dulaunoy]

• [bin] make black happy. [Alexandre Dulaunoy]

• [bin] consistency update fix #4. [Alexandre Dulaunoy]

Fix

• [doc] bloom filter link fixed. [Alexandre Dulaunoy]

• [bloomfilter] bug fix as bloomfilter just returns a True or False. [Alexandre Dulaunoy]

• [bin] remove ambiguity on file handle type. [Alexandre Dulaunoy]

• [bin] BinaryIO and not TextIO while reading file. [Alexandre Dulaunoy]

• [bin] blake formatting (need a post-hook ;-) [Alexandre Dulaunoy]

• [bin] skip type test on empty dict. [Alexandre Dulaunoy]

• [bin] incorrect version. [Alexandre Dulaunoy]

• [bin] as referenced in #6 - platform assignment can lead to some confusion. [Alexandre Dulaunoy]

• [REQUIREMENTS] pytz dep was missing fix #5. [Alexandre Dulaunoy]

hashlookup forensic analyser - version 0.2 release

hashlookup-forensic-analyser is a script to analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

What's Changed

• Clarification regarding the CSV output format
• Size of the file analysed added in the output
• Improved performance - reduce lines by @signorrayan in #1
• Pre-built Linux binary of the script added (if Python is not present on the target system to be analysed)

New Contributors

• @signorrayan made their first contribution in #1

Screenshot

Full Changelog: v0.1...v0.2