make some shell scripts use hermetic aspect toolchains

[GregBowyer]

This changes some of the shellscripts to use the aspect tools for coreutils to be more hermetic.

This also has a probably buggy rewrite of the embedded awk scripts to avoid needing an awk toolchain.

[jjmaestro]

At some point I also wanted to replace coreutils and other dependencies in the repo with something like rules_sh + rules_nixpkgs.

I thought it would be nicer to have one source of hermetic tooling for the shell scripts and for the rules, so there would be "one way and one source" of tools like bsdtar and zstd but also bzip, etc) instead of having to bring separate rules / repos.

@thesayyn what do you think?

[GregBowyer]

rules_nixpkgs will make everything depend on nixos in ways I dont think you want, it will also make it hard to do remote builds.

(FWIW I am a NixOs user)

[jjmaestro]

rules_nixpkgs will make everything depend on nixos in ways I dont think you want, it will also make it hard to do remote builds.

That's good to know! I just watched @aherrmann's "Hermetic shell scripts in Bazel" from Bazelcon 2022 and it seemed quite simple and abstracted from NixOs.

Can you give me more info / context and/or links? I'm interested in learning more about this!

[GregBowyer]

@jjmaestro probably this is the best current reference for at least the remote issue:

https://www.tweag.io/blog/2024-02-29-remote-execution-rules-nixpkgs/

Essentially nixos stores all its packages in the store under hashed paths, this assumes the store is:

1. Installed, so if you use pkgs nixos AFAIK your end users will have to install nix
2. figure out how to share the store (for example NFS mount it across builder nodes)

FWIW aspectlibs coreutils isnt the full gnu coreutils but the rust uutils reimplementation which makes it quite portable.

Sed is naturally missing from that as is awk, although goawk could be an alternative if you dont want to live in pure shell.

[jjmaestro]

Awesome, thanks a ton! This is really good info, I was also looking into rules_sh + rules_nixpkgs for other stuff but the RBE issue is definitely a roadblocker, 100%

[aherrmann]

FWIW rules_sh is orthogonal to rules_nixpkgs and can be used without it. We commonly use Nix to provide the actual shell tools, but they can come from other sources as well. E.g. there's an open issue to fetch and wrap a busibox toolchain.

[thesayyn]

i found awk scripts to be much more readable and easy to author for text editing purposes. however in this case we don't have awk built off the shelf.

I have bazel recipes in place to build GAWK for windows, darwin and linux but never made a release.

[thesayyn]

maybe it's better to introduce a gawk toolchain in bazel-lib and use that? i am not sure how much effort that would take.

[thesayyn]

Now that there is https://github.com/bazelbuild/rules_shell, it became much more confusing.

coreutils toolchain from bazel-lib uses rust reimplementation of coreutils like @GregBowyer said so it is portable to windows,darwin and linux.

back when i was evaluating rules_nixpkgs, windows support was a big blocker, so i don't think we can use that, we could probably use the toolchain types from rules_shell but let's keep it focused.

[thesayyn]

LGTM