Lessons learned from v1.0.0 release

[marc-white]

This issue is designed to record lessons learned from the v1.0.0 release process, and to track actions against those lessons.

• We should documented the suggested release process, to wit: (Post v1.0.0 release documentation updates #302)
  • Make a pre-release candidate build (see below)
  • Build a catalog
  • Make the full version release
• Need to document that pre-release candidates need an ordinal after the b, e.g., v1.0.0b0. Otherwise, the GitHub PyPI build action will append one, which breaks the relationship between the PyPI and Conda builds. (Post v1.0.0 release documentation updates #302)
• It occurs to me; what prevents a random user from triggering a new catalog build? I presume that a user needs to be a member of the xp65 writers to do so, but we should check. (And if being an xp65 writer is the requirement/safeguard, that should be documented).
• There appear to be no post-build tests suggested for the main catalog, beyond, "Run some random notebooks and see what breaks". Consideration should be given to defining some tests that a developer can run to check the health of the new catalog (perhaps even just a Jupyter notebook of assorted commands to check). Add 'sanity check' notebook for post-catalog build checks #303
• Document that an up-to-date code base is required in /g/data/xp65/admin/access-nri-intake-catalog, as that's where the build script sources the configuration YAML files from, and where the build script is read from. Should also consider if this script should be referenced as a command-line command instead. (Post v1.0.0 release documentation updates #302)
• Add metadata validation tests #285
• [BUG] Keeping test datasets aligned with Gadi data #290
• Add new storage paths and data ingest warnings to documentation #299
• Fail-resistant catalog build #307
• Catalog polling to infer & exclude broken datastores #308

[rbeucher]

Thanks @marc-white

[charles-turner-1]

There appear to be no post-build tests suggested for the main catalog, beyond, "Run some random notebooks and see what breaks". Consideration should be given to defining some tests that a developer can run to check the health of the new catalog (perhaps even just a Jupyter notebook of assorted commands to check).

Could we use/modify/extend the end to end tests for this?

[marc-white]

There appear to be no post-build tests suggested for the main catalog, beyond, "Run some random notebooks and see what breaks". Consideration should be given to defining some tests that a developer can run to check the health of the new catalog (perhaps even just a Jupyter notebook of assorted commands to check).

Could we use/modify/extend the end to end tests for this?

That's one possibility, although I was wondering how to set up a test suite that interrogates a 'live' catalog, rather than some minimal test case/mock.

What I actually had in mind was a basic Jupyter notebook that runs a lot of really basic sanity-check commands, and let's you look over the outputs for anything odd. For example, comparing the new catalog against a previous version (that's how I noticed all of the missing experiments in PR #281 ). Given each catalog update will have something unique to it, I'm not sure there's a way you can realistically automate testing of that.

[charles-turner-1]

Yeah, good call.

Elastic have a tool called nbtest which would be perfect for this situation - we build a notebook with a bunch of expected outputs, & we can use nbtest to ensure consistency.