Add new UDC to perform an optimized git deep clone

[idodod]

Also:

• Moving non language UDCs under utils (+INSTALL_DIND should be used as ./utils/dind+INSTALL).
• Standardizing tests between ssh and git directories.
• Running dind+INSTALL tests for arm64 in a dedicated satellite.

[CLAassistant]

All committers have signed the CLA.

[idodod]

@vladaionescu should I maybe move git and ssh folders into a new tools folder
(and later also DIND_INSTALL into dind)?

[vladaionescu]

I wonder if we should move all three into such a common / misc / utitlities folder. We can keep DIND_INSTALL in the root too for backwards compatibility.

[idodod]

yep, that's what I was trying to suggest :)

[idodod]

I moved everything under utils, lmk if you prefer a different name.

[vladaionescu]

Looks good!

Should github.com/earthly/lib/utils expose the UDCs directly? (We can still have the implementation in subdirs)

[vladaionescu]

Hmm.. I wouldn't do it. It just adds more complexity to the language, with little benefit.

[idodod]

so we're left only with some variation of exposing the UDCs in utils/Earthfile, aren't we?

[vladaionescu]

I think that's the best option

[idodod]

This is the alternative

IMPORT github.com/earthly/lib/utils/git:<tag>
IMPORT github.com/earthly/lib/utils/ssh:<tag>

but this would probably mean cloning more than once, unless it's cached/deduped?

[vladaionescu]

It's a reasonable alternative - especially if sticking with the naming dind+INSTALL (as opposed to utils+INSTALL_DIND). The main downside of this is having to import each UDC separately. Arguably, you might not use multiple lib UDCs for this to be a problem. Maybe it can be in the future?

Having utils split up into subdirectories helps with segregating UDCs into different areas of concern nicely. But it also has the drawback that it's slightly more confusing what tag to use for each.

All in all, I'm ok with either option. There are pros and cons for each.

[idodod]

@vladaionescu
I'm not even sure we need to keep this for backwards compatibility since we're going to cut a new release/tag for utils and discontinue the global tag (which users can still use).
Thoughts?

[vladaionescu]

I would keep it, because we have many users of INSTALL_DIND who don't use a tag at all. (We should update our docs too to suggest using a tag)

I wonder if we should add a RUN echo <some-warning-message> when using the outter INSTALL_DIND, to invite the users to use the new version directly + a tag.

[idodod]

I think that'd a good idea