Additional hash algorithms

[richardlehane]

I had an email request to add blake3 support.

My preference is to just offer algorithms defined in the go standard library (crypto and hash packages). If there is a strong need, could consider adding algorithms from the x/crypto package too (e.g. blake2 algos are defined there).

There are other interesting hash algos out there e.g. xxhash. But I'm allergic to adding third-party dependencies to the project. Also, if you want to do fast hashing of files, siegfried is probably not the best vehicle for that as the way I'm doing IO probably isn't really optimised for the use case & you also have the cost of file format ID.

[ross-spencer]

Couple of thoughts:

• would be useful to get a more complete user story about the why.
• the principle of third-party dependencies in Siegfried is a strong one and has always felt in alignment with preservation principles and long-term support across a wide-range of user-bases.
• in archival use-cases for Siegfried checksums are rarely about security, rather they are about integrity across data-transfer boundaries (YMMV) but maybe there are other domains using Siegfried where this is more important? (but also, then, more specific tools exist to enable this -- not every tool need be a swiss army knife)

One question that comes to mind, is there a benchmark that can be easily run to test the different speeds of these algorithms when added to Siegfried? One project I follow uses these as an objective measure to begin these conversations if folks add/remove features like this?

[richardlehane]

Good points thanks Ross!

There's probably a good argument to use one of these fast and efficient algorithms in your regular fixity checking. And, if you have made that choice, maybe you want to use the same algorithm at every layer of your tech stack?

[ross-spencer]

maybe you want to use the same algorithm at every layer of your tech stack?

from my perspective that's fair. It can support other functions like content addressable storage, and it's also a more ergonomic experience for those looking at these because it is something recognizable and becomes easier to eyeball.

[kieranjol]

Same algo for each part of the stack including preingest or submissions themselves is very attractive! K

…

On Wed 28 Aug 2024 at 06:50, Ross Spencer ***@***.***> wrote: maybe you want to use the same algorithm at every layer of your tech stack? from my perspective that's fair. It can support other functions like content addressable storage, and it's also a more ergonomic experience for those looking at these because it is something recognizable and becomes easier to eyeball. — Reply to this email directly, view it on GitHub <#259 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAITFPS4JWNIQE33GKR6CSTZTVQLVAVCNFSM6AAAAABMZ3CDGGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBXGA2TSMI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** .com>