Suggestion: Use the golang.org/dl API for version info

[docwhat]

Now that https://golang.org/dl?mode=json&include=all works, we can use this to fetch a complete list of go archives and installers.

`golang-versions.py`

#!/usr/bin/env python3

"""
Use this to pipe the JSON from 'https://golang.org/dl/?mode=json&include=all' to generate
shell compatable data for 'go-build'.

Example:

    $ { echo 'declare -a versions=()'; curl "https://golang.org/dl/?mode=json&include=all" | python3 this-script.py | grep '#arch:amd64#.*#kind:archive#.*#os:darwin#.*#stable#' | grep '^versions' | uniq; echo 'for v in "${versions[@]}"; do echo $v; done' } | /bin/bash
"""

import json
import sys
from shlex import quote


def emit_one_file(out, file, tags):
    for k, v in file.items():
        if k in ("filename", "size", "sha256", "version"):
            continue
        tags.append("{}:{}".format(k, v))
    tags.sort()
    for k in ("filename", "size", "sha256", "version"):
        out.write("{:80} ".format("{}={}".format(k, quote(str(file[k])))))
        out.write(" ".join(["#{}#".format(tag) for tag in tags]))
        out.write("\n")

    out.write("{:80} ".format("versions+=({})".format(quote(str(file["version"])))))
    out.write(" ".join(["#{}#".format(tag) for tag in tags]))
    out.write("\n")


def emit_one_version(out, one_version):
    tags = [
        "version:{}".format(one_version["version"]),
        "stable" if one_version["stable"] else "unstable",
    ]

    for file in one_version["files"]:
        emit_one_file(out, file, [t for t in tags])
    out.write("\n")


def emit_lookup_data(writer, versions):
    for one_version in versions:
        emit_one_version(writer, one_version)


if __name__ == "__main__":
    data = json.load(sys.stdin)
    emit_lookup_data(sys.stdout, data)

The generated file output can be used to find versions with grep.

# Note: grep terms need to be alphabetically sorted.
$ cat dbfile | grep '#arch:amd64#.*#kind:archive#.*#os:darwin#.*#stable#.*#version:go1\.12\.5#'
filename=go1.12.5.darwin-amd64.tar.gz                                            #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.12.5#
size=127612395                                                                   #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.12.5#
sha256=566d0b407f7d4aa5a1315988b562bbe4e9422a93ce2fbf27a664cddcb9a3e617          #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.12.5#
version=go1.12.5                                                                 #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.12.5#
versions+=(go1.12.5)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.12.5#

And you can find available versions by using the versions variable:

$ { echo 'declare -a versions=()'; curl "https://golang.org/dl/?mode=json&include=all" | python3 this-script.py | grep '#arch:amd64#.*#kind:archive#.*#os:darwin#.*#stable#' | grep '^versions' | uniq; echo 'for v in "${versions[@]}"; do echo $v; done' } | /bin/bash | head -n 20
declare -a versions=()
versions+=(go1.16.7)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.16.7#
versions+=(go1.15.15)                                                            #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.15#
versions+=(go1.16.6)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.16.6#
versions+=(go1.16.5)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.16.5#
versions+=(go1.16.4)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.16.4#
versions+=(go1.16.3)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.16.3#
versions+=(go1.16.2)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.16.2#
versions+=(go1.16.1)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.16.1#
versions+=(go1.16)                                                               #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.16#
versions+=(go1.15.14)                                                            #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.14#
versions+=(go1.15.13)                                                            #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.13#
versions+=(go1.15.12)                                                            #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.12#
versions+=(go1.15.11)                                                            #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.11#
versions+=(go1.15.10)                                                            #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.10#
versions+=(go1.15.9)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.9#
versions+=(go1.15.8)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.8#
versions+=(go1.15.7)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.7#
versions+=(go1.15.6)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.6#
versions+=(go1.15.5)                                                             #arch:amd64# #kind:archive# #os:darwin# #stable# #version:go1.15.5#

[docwhat]

The python code would be run periodically by a CI job. Though I guess we could include it as python3 is pretty common now... and fall back to the pre-generated version.

The DB is about 2.2Mb but compresses really well to about 183kb; we can just use zcat on it.

[syndbg]

It would be an awesome contribution.

[docwhat]

@syndbg Would it be okay to have a run-time dependency on python3? I can use any language or tool you want; it just needs to be able to translate JSON to text.

I could even download something like hairyhenderson/gomplate or docwhat/temple instead of depending on python3.

Or I could even build a tiny GoLang program to do the translation from JSON to the DB file format.

[drichelson]

Re: "Would it be okay to have a run-time dependency on python3?" This 'should just work' on many modern OSes, but I would prefer to avoid it since it adds extra complexity.

[pwmcintyre]

this would be an excellent feature - our CI uses goenv, but we have to rebuild the whole image just to refresh the list of supported golang version! Seems silly

consider: instead of throwing a definition not found error, first see if it exists remotely?

[syndbg]

Hey, sorry for the delayed reply, on a vacation 'till the end of the year.

So, answering the questions.

Would it be okay to have a run-time dependency on python3? I can use any language or tool you want; it just needs to be able to translate JSON to text.

I'd prefer not to depend on anything Python. If you're looking for a specific language to do it, I'd prefer Golang.

Just to be clear, this is only a feature for the CI.
My opinion on what goenv should solve is still the same - reliable, secure (as much as checksums are secure) and predictable golang version manager and installer.

I still prefer for this "version golang releases/versions discovery feature" to run in CI, create PR (if needed) and still require manual approval.

this would be an excellent feature - our CI uses goenv, but we have to rebuild the whole image just to refresh the list of supported golang version! Seems silly
consider: instead of throwing a definition not found error, first see if it exists remotely?

Being able to install versions fetched dynamically from remote might be amazing as a user experience, but I prefer it the other way - a specific version of Goenv can install only N versions of Golang from well-determined sources and expected checksums.

The reason why I prefer this is rather "lame". It's easier from a security perspective (InfoSec) to review and sign-off that this software is "secure". Enterprise/Grown-up friendly... in a way.

[docwhat]

The reason why I prefer this is rather "lame". It's easier from a security perspective (InfoSec) to review and sign-off that this software is "secure". Enterprise/Grown-up friendly... in a way.

Unless someone is reviewing each release, the only security benefit is that the window of attack is smaller (golang.org would have be compromised during a CI build vs. any time someone installs Go) and that a bad release can be removed (tho not revoked since goenv lacks a revoke system).

💡idea: Since we have to trust golang.org anyway, how about we pin the SSL certificate in the curl request? Whether we do it only in CI or in run-time.

Is there something we could do to increase the security of gathering the available go versions at run-time?

[ChronosMasterOfAllTime]

See #261..waiting on being able to add a PAT to the repo 😄

[ChronosMasterOfAllTime]

@ankitcharolia we use the go dev API for automation of daily version checks. I don't mind that you have your own version of goenv that depends on go. Our implementation doesn't depend on any external dependency and uses native shell to achieve the same outcomes.