When prepping a release, combine the dependency bump changelog entrie…

…s. (#2181)

* Add a note to get-dep-changes to alert folks that changing those formats might break other things.

* Create an awk script that will combine dependency changelog entries. Update prep-release to use it. Also apply a couple fixes that are alread in the release branch (and will be in main shortly). Also tweak the step 4 and 5 names to provide more context, and fix the verbose output header when recombining the sections.

* Add changelog entry.

* Clarify the new comment in get-dep-changes.sh.

* Update stuff that uses or talks about RELEASE_NOTES.md because it should actually be RELEASE_CHANGELOG.md. The SDK uses _NOTES but only puts a blurb in there, so it's not a changelog. But we include a changelog, so it makes sense to keep it named that way.

.changelog/README.md

@@ -230,7 +230,7 @@ If the entry isn't part of a release candidate, then in step 4 it gets moved to
 When preparing to mark a release, you should use the `.changelog/prep-release.sh` script.
 
 That script will:
-1. Create or Update the `RELEASE_NOTES.md` file.
+1. Create or Update the `RELEASE_CHANGELOG.md` file.
 2. Add the new version to the `CHANGELOG.md` file.
 3. Create a new version directory in the `.changelog/` folder with content from `unreleased/` and any rcs for this version.
 
@@ -240,14 +240,14 @@ That is, the `.changelog/<version>` directories should only ever exist on the `.
 This is primarily to reduce confusion if there is a discrepancy between the `CHANGELOG.md` content and an entry file's content.
 It also helps keep things tidy and file counts lower.
 
-If you need to make tweaks or clarifications to the content, you should make the changes in the `RELEASE_NOTES.md` file first, then copy/paste those into the `CHANGELOG.md` file.
+If you need to make tweaks or clarifications to the content, you should make the changes in the `RELEASE_CHANGELOG.md` file first, then copy/paste those into the `CHANGELOG.md` file.
 You should NOT update the changelog entry files, though (other than moving them).
 
-And to reiterate, the `RELEASE_NOTES.md` file and `.changelog/<version>` directories should never exist on `main`, only in the `.x` branch.
+And to reiterate, the `RELEASE_CHANGELOG.md` file and `.changelog/<version>` directories should never exist on `main`, only in the `.x` branch.
 
 If you can't, or don't want to use the `.changelog/prep-release.sh` script, here's how to do things manually.
 
-To manually create the new `RELEASE_NOTES.md` file:
+To manually create the new `RELEASE_CHANGELOG.md` file:
 
 1. If this is a full version and there were release candidates, move all the rc content into unreleased.
 2. Run `unclog build --unreleased` to get the preliminary content.

.changelog/combine-dep-lines.awk

@@ -0,0 +1,102 @@
+# This awk script will process the list of dependency changelog entries.
+# It will combine multiple bumps for the same library into a single line.
+#
+# The lines should have been generated by get-dep-changes.sh and relies on its formatting.
+# It's assumed that these entries are sorted by library (alphabetically), then by action ("added",
+# "bumped to", or "removed"), then by new version number (in version order).
+#
+# Example input:
+# * `lib1` bumped to v1.2.3 (from v1.2.2) <link1>.
+# * `lib1` bumped to v1.2.4 (from v1.2.3) <link2>.
+# * `lib2` bumped to v0.4.5 (from v0.3.1) <link3>.
+# Gets turned into this:
+# * `lib1` bumped to v1.2.4 (from v1.2.2) (<link1>, <link2>).
+# * `lib2` bumped to v0.4.5 (from v0.3.1) <link3>.
+#
+# If there were three (or more) lib1 lines, then, they'd all be combined similarly.
+#
+# This also accounts for a possible "but is still replaced by <other lib>" warning in the line, e.g.:
+# * `lib1` bumped to v1.2.3 (from v1.2.2) but is still replaced by <fork1> <link1>.
+# If the last line for a lib has that warning, the final line will too, but it's ignored on the other entries.
+#
+# It allows for the versions to include a replacement library too, e.g.
+# * `<lib>` bumped to v0.50.10-pio-1 of `<fork>` (from v0.50.7-pio-1 of `<fork>`) <link>.
+#
+# Here's the expeced bump line format:
+# * `<lib>` bumped to <version_new>[ of `<fork_new>`] (from <version_old>[ of `<fork_old>`])[ <warning>] <link>.
+# Where [] denotes an optional portion.
+# And <name> represents a string that we will call "name" (but doesn't necessarily reflect a variable).
+# And <link> has the format "[<text>](<address>)" (the [] and () in that are the actual expected characters).
+
+# printPrevLib prints the previous library's info and resets things.
+function printPrevLib() {
+    if (LibCount == 1) {
+        # If there was only one line for this library, just print it without any changes.
+        # The only real difference between this output and the combined one is the parenthases around the links.
+        # But I just felt like it'd be best to keep the original when there's nothing being combined.
+        print LibFirstLine
+    } else if (LibCount > 1) {
+        # If there were multiple lines for this library, create a new line for it with all the needed info.
+        print "* " PrevLib " bumped to " NewVer " (from " OrigVer ")" Warning " (" Links ")."
+    }
+    LibCount=0
+}
+
+{
+    # Expected bump line format: "* `<lib>` bumped to <new_version>[ of `<fork>`] (from <old_version>[ of `<fork>`])[ <warning>] <link>."
+    if (/^\* `.*` bumped to .* \(from .*\).* \[.*[[:digit:]]+\]\(.*\)/) {
+        # The library is the second thing in the line (first is the "*"). Get it including the ticks.
+        lib=$2
+
+        # The link is everything after "(from ...) " up to the last ")".
+        # This also accounts for a possible warning right before the link (the [^[]* part in this Rx).
+        link=$0
+        sub(/^.*\(from [^)]*\)[^[]* \[/,"[",link)
+        sub(/\)[^)]*$/,")",link)
+
+        if (lib != PrevLib) {
+            # This line's library is different from the previous. Print out the previous (if we have one).
+            printPrevLib()
+
+            # This lib is now the "previous" one and we restart the links list and counter.
+            PrevLib=lib
+            Links=link
+            LibFirstLine=$0
+            LibCount=1
+
+            # The original version is everything between "(from " and the next ")" of the first line for a library.
+            OrigVer=$0
+            sub(/^.*\(from /,"",OrigVer)
+            sub(/\).*$/,"",OrigVer)
+        } else {
+            # This line's library has the same library as the previous, add the link to the list and count it.
+            Links=Links ", " link
+            LibCount++
+        }
+
+        # The warning will be between "(from ...)" and the " [" of the link.
+        # It should end up with a leading space character.
+        # This resets every line so that we only get it from the last line for a library.
+        Warning=""
+        if (/\(from [^)]*\) .* \[/) {
+            Warning=$0
+            sub(/^.*\(from [^)]*\)/,"",Warning)
+            sub(/ \[.*$/,"",Warning)
+        }
+
+        # The version from this line is now the most recent (so far) for the current library.
+        # It's everything between "bumped to " and "(from ".
+        NewVer=$0
+        sub(/^.* bumped to /,"",NewVer)
+        sub(/ \(from .*$/,"",NewVer)
+    } else {
+        # Unknown line, print out the previous entry (if there is one), then print out this line.
+        printPrevLib()
+        print $0
+    }
+}
+
+# Make sure there isn't one left over.
+END {
+    printPrevLib()
+}

.changelog/get-dep-changes.sh

@@ -334,6 +334,7 @@ for lib in "${libs[@]}"; do
     [[ -n "$verbose" ]] && printf '[%d/%d]:   %s="%s"  %s="%s"  %s="%s"\n' "$i" "${#libs[@]}" 'new' "$new" 'was' "$was" 'warning' "$warning"
 
     # Now generate the changelog line for this library.
+    # There's stuff in combine-dep-lines.awk (used by prep-release.sh) that depends on these formats.
     if [[ -n "$new" && -n "$was" ]]; then
         if [[ "$new" != "$was" ]]; then
             [[ -n "$verbose" ]] && printf '[%d/%d]: Creating bump line.\n' "$i" "${#libs[@]}"

.changelog/prep-release.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 # This script will update the changelog stuff to mark a release.
+# The following script(s) must be in the same dir as this one:
+#   combine-dep-lines.awk
 
 show_usage () {
     cat << EOF
@@ -96,6 +98,15 @@ if [[ -z "$repo_root" ]]; then
 fi
 [[ -n "$verbose" ]] && printf ' Repo root dir: [%s].\n' "$repo_root"
 
+# Define the path to the dependency combiner awk script, allowing for it to be defined by env var instead.
+combine_dep_lines_awk="${COMBINE_DEP_LINES_AWK:-${where_i_am}/combine-dep-lines.awk}"
+if [[ ! -f "$combine_dep_lines_awk" ]]; then
+  printf 'Could not find the combine-dep-lines awk script: %s\n' "$combine_dep_lines_awk"
+  exit 1
+fi
+
+
+
 changelog_file="${repo_root}/CHANGELOG.md"
 if [[ ! -f "$changelog_file" ]]; then
     printf 'Could not find existing CHANGELOG.md file.\n'
@@ -452,17 +463,31 @@ while IFS="" read -r line || [[ -n "$line" ]]; do
     fi
 done < "$links_fixed_file"
 
-# Sort the entries of the dependencies section.
-# They have the format "* `<library>` <action> <version> ..." where <action> is one of "added at" "bumped to" or "removed at".
+# Clean, sort, and combine the entries of the dependencies section.
 # So, if we just sort them using version sort, it'll end up sorting them by library and version, which a handy way to view them.
 dep_file="${temp_dir}/3-section-dependencies.md"
 if [[ -f "$dep_file" ]]; then
-    [[ -n "$verbose" ]] && printf 'Sorting the dependency entries: [%s].\n' "$dep_file"
-    orig_dep_file="${dep_file}.orig"
+    # Move the existing one to a backup so we can create a new one in its place (but still have the original for reference).
+    orig_dep_file="${dep_file}.1_orig"
+    [[ -n "$verbose" ]] && printf 'Backing up original dependencies section file [%s] as [%s].\n' "$dep_file" "$orig_dep_file"
     mv "$dep_file" "$orig_dep_file"
-    head -n 2 "$orig_dep_file" > "$dep_file"
-    grep -E '^[[:space:]]*[-*]' "$orig_dep_file" | sed -E 's/^[[:space:]]+//; s/^(.)[[:space:]]+/\1 /;' | sort --version-sort >> "$dep_file"
-    printf '\n' >> "$dep_file"
+
+    # Entry format: "* `<library>` <action> <version> ..." where <action> is one of "added at" "bumped to" or "removed at".
+    # First, make sure that the bullet point is the first char and there's exactly one space between it and the text; also
+    # remove all trailing whitespace. Then, apply version sort to sort them by library (alphabetically), then
+    # action (alphabetically, i.e. "added" then "bumped" then "replaced"), then by version (in version order).
+    sorted_dep_file="${dep_file}.2_sorted"
+    [[ -n "$verbose" ]] && printf 'Cleaning and sorting dependencies into [%s].\n' "$sorted_dep_file"
+    grep -E '^[[:space:]]*[-*]' "$orig_dep_file" | sed -E 's/^[[:space:]]+//; s/^(.)[[:space:]]+/\1 /;' | sort --version-sort > "$sorted_dep_file"
+
+    combined_dep_file="${dep_file}.3_combined"
+    [[ -n "$verbose" ]] && printf 'Combining the sorted dependencies using [%s] into [%s].\n' "$combine_dep_lines_awk" "$combined_dep_file"
+    awk -f "$combine_dep_lines_awk" "$sorted_dep_file" > "$combined_dep_file"
+
+    [[ -n "$verbose" ]] && printf 'Creating final dependencies section file: [%s].\n' "$dep_file"
+    head -n 2 "$orig_dep_file" > "$dep_file" # Copy the original header line and following empty line.
+    cat "$combined_dep_file" >> "$dep_file"  # Append the cleaned, sorted, and combined entries.
+    printf '\n' >> "$dep_file"               # And add an empty line to the end of the section.
 fi
 
 [[ -n "$verbose" ]] && printf 'Determining desired order for sections.\n'
@@ -492,13 +517,13 @@ add_to_section_order top \
     dependencies
 [[ -n "$verbose" ]] && printf 'Including sections in this order (%d): [%s].\n' "${#section_order[@]}" "${section_order[*]}"
 
-new_cl_entry_file="${temp_dir}/4-release-notes.md"
+new_cl_entry_file="${temp_dir}/4-version-release-notes.md"
 [[ -n "$verbose" ]] && printf 'Re-combining sections with proper order: [%s].\n' "$new_cl_entry_file"
 
 s=0
 for section in "${section_order[@]}"; do
     s=$(( s + 1 ))
-    s_id="[${s}/${#}=${section}]"
+    s_id="[${s}/${#section_order[@]}=${section}]"
     s_file="${temp_dir}/3-section-${section}.md"
     if [[ ! -f "$s_file" ]]; then
         [[ -n "$verbose" ]] && printf '%s: No section file to include: [%s].\n' "$s_id" "$s_file"
@@ -525,8 +550,8 @@ clean_versions () {
 
 # If this is an rc and there's an existing release notes, append those to the end, removing any existing section for this version.
 # If it's not an rc, or there isn't an existing one, just use what we've already got.
-new_rl_file="${temp_dir}/5-release-notes.md"
-release_notes_file="${repo_root}/RELEASE_NOTES.md"
+new_rl_file="${temp_dir}/5-full-release-notes.md"
+release_notes_file="${repo_root}/RELEASE_CHANGELOG.md"
 cp "$new_cl_entry_file" "$new_rl_file"
 if [[ -n "$v_rc" && -f "$release_notes_file" ]]; then
     [[ -n "$verbose" ]] && printf 'Including existing release notes: [%s].\n' "$release_notes_file"

.changelog/unreleased/improvements/2181-combine-changelog-dep-lines.md

@@ -0,0 +1 @@
+* Update the prep-release script to combine dependency changelog entries [PR 2181](https://github.com/provenance-io/provenance/pull/2181).

CONTRIBUTING.md

@@ -310,8 +310,8 @@ You will need to create a new development branch for this and PR it back to the
 
 The `CHANGELOG.md` on the `.x` branch must be updated to reflect the new release.
 
-1. Run `.changelog/prep-release.sh <version>` to create/update `RELEASE_NOTES.md`, update `CHANGELOG.md`, and move things around in the `.changelog/` folder.
-2. Review the changes with extra attention on the new content of `CHANGELOG.md` and `RELEASE_NOTES.md`.
+1. Run `.changelog/prep-release.sh <version>` to create/update `RELEASE_CHANGELOG.md`, update `CHANGELOG.md`, and move things around in the `.changelog/` folder.
+2. Review the changes with extra attention on the new content of `CHANGELOG.md` and `RELEASE_CHANGELOG.md`.
 3. Stage and commit the changes.
 4. Push up your branch and create a PR for it to the `.x` branch. The PR title should be like `Mark v1.13.0`.
 5. Get the PR approved and merged.