Add Task V2; Use Dependabot-CLI to perform updates (#1318)

* Split extension task in to V1 and V2

* Basic support for running update using dependabot-cli

* Add missing update job configs

* Move update output processing to dedicated class; Add DevOps API client

* Codespell skip package-lock.json

* Clean-up

* Implement create pull request

* Implement groups, auto-complete, auto-approve, pull request properties, and updating existing pull requests

* Clean up

* Set task V1 as deprecated, task V2 as preview

* Restructure extension task to better support multiple version

* Fix typos

* Fix build

* Fix build

* Fix merge issues

* Remove unused task inputs

* Fix for 'convertPlaceholder' not accepted built-in DevOps variable names containing '.'

* Implement closing pull requests

* Implement updating pull requests

* Use default branch name if target branch not configured

* Implement approving pull requests

* Add task inputs for pr commit author email and name

* Implement open pull request limit config

* Cleanup temporary files after task completion

* Add configuration placeholders for dependabot component images

* Implement more config options

* Implement dependency list snapshots, which are stored in the DevOps project properties

* Add task input option for storing dependency list

* Implement experiments

* Implement requirements-update-strategy and lockfile-only configs

* Fix typo

* Implement PR reviewers, work item references, and labels

* Add start commands for each task version, use V2 by default

* Update V1 task.json version numbers when publishing the extension

* Update documentation

* Update documentation

* Update documentation

* Update documentation

* Typo

* Update documentation

* Fix reference to undefined `this.cachedUserIds`

* Use case insensitive comparision when parsing "System.Debug" variable

* Fix dependabot tool path detection in agents where `$PATH` does not contain `$GOPATH/bin`

* Add more logging

* Fix for task reporting success when pull request creation failed

* Add more logging; Fix formatting

* Fix 'labels' config parsing

* Implement "targetUpdateIds" task input option

* Fix error when using multiple update blocks in dependabot.yml with the same package manager

* Only install dependabot once; cache the tool path once known

* Add migration warning to complete V1 pull requests before migrating to V2

* Process updates synchronously when using multiple update blocks in dependabot.yml

* Fix typos

* Report the total number of failed update jobs in the task result

* Include stack trace when errors are logged, to help with diagnosing issues

* Fix inverted logic for "abandonUnwantedPullRequests"

* Fix error handling

.codespellrc

@@ -1,5 +1,5 @@
 [codespell]
-skip = .git,*.pdf,*.svg,pnpm-lock.yaml,yarn.lock
+skip = .git,*.pdf,*.svg,pnpm-lock.yaml,yarn.lock,package-lock.json
 # some modules, parts of regexes, and variable names to ignore, some
 # misspellings in fixtures/external responses we do not own
 ignore-words-list = caf,bu,nwo,nd,kernal,crate,unparseable,couldn,defintions

.github/workflows/extension.yml

@@ -76,9 +76,12 @@ jobs:
 
       - name: Update version numbers in task.json
         run: |
-          echo "`jq '.version.Major=${{ steps.gitversion.outputs.major }}' extension/tasks/dependabot/dependabotV1/task.json`" > extension/tasks/dependabot/dependabotV1/task.json
-          echo "`jq '.version.Minor=${{ steps.gitversion.outputs.minor }}' extension/tasks/dependabot/dependabotV1/task.json`" > extension/tasks/dependabot/dependabotV1/task.json
+          echo "`jq '.version.Major=1' extension/tasks/dependabot/dependabotV1/task.json`" > extension/tasks/dependabot/dependabotV1/task.json
+          echo "`jq '.version.Minor=34' extension/tasks/dependabot/dependabotV1/task.json`" > extension/tasks/dependabot/dependabotV1/task.json
           echo "`jq '.version.Patch=${{ github.run_number }}' extension/tasks/dependabot/dependabotV1/task.json`" > extension/tasks/dependabot/dependabotV1/task.json
+          echo "`jq '.version.Major=${{ steps.gitversion.outputs.major }}' extension/tasks/dependabot/dependabotV2/task.json`" > extension/tasks/dependabot/dependabotV2/task.json
+          echo "`jq '.version.Minor=${{ steps.gitversion.outputs.minor }}' extension/tasks/dependabot/dependabotV2/task.json`" > extension/tasks/dependabot/dependabotV2/task.json
+          echo "`jq '.version.Patch=${{ github.run_number }}' extension/tasks/dependabot/dependabotV2/task.json`" > extension/tasks/dependabot/dependabotV2/task.json
 
       - name: Create Extension (dev)
         run: >

docs/extension.md

@@ -3,21 +3,24 @@
 
 - [Using the extension](#using-the-extension)
 - [Development guide](#development-guide)
-  - [Getting the development environment ready](#getting-the-development-environment-ready)
-  - [Building the extension](#building-the-extension)
-  - [Installing the extension](#installing-the-extension)
-  - [Running the unit tests](#running-the-unit-tests)
+   * [Getting the development environment ready](#getting-the-development-environment-ready)
+   * [Building the extension](#building-the-extension)
+   * [Installing the extension](#installing-the-extension)
+   * [Running the task locally](#running-the-task-locally)
+   * [Running the unit tests](#running-the-unit-tests)
+- [Architecture](#architecture)
+   * [Task V2 high-level update process diagram](#task-v2-high-level-update-process-diagram)
+
 
 # Using the extension
 
-See the extension [README.md](../extension/README.md).
+Refer to the extension [README.md](../extension/README.md).
 
 # Development guide
 
 ## Getting the development environment ready
 
-First, ensure you have [Node.js](https://docs.docker.com/engine/install/) v18+ installed.
-Next, install project dependencies with npm:
+Install [Node.js](https://docs.docker.com/engine/install/) v18 or higher; Install project dependencies using NPM:
 
 ```bash
 cd extension
@@ -31,25 +34,86 @@ cd extension
 npm run build
 ```
 
-To generate the  Azure DevOps `.vsix` extension package for testing, you'll first need to [create a publisher account](https://learn.microsoft.com/en-us/azure/devops/extend/publish/overview?view=azure-devops#create-a-publisher) on the [Visual Studio Marketplace Publishing Portal](https://marketplace.visualstudio.com/manage/createpublisher?managePageRedirect=true). After this, override your publisher ID below and generate the extension with:
+To then generate the a Azure DevOps `.vsix` extension package for testing, you'll first need to [create a publisher account](https://learn.microsoft.com/en-us/azure/devops/extend/publish/overview?view=azure-devops#create-a-publisher) for the [Visual Studio Marketplace Publishing Portal](https://marketplace.visualstudio.com/manage/createpublisher?managePageRedirect=true). After this, use `npm run package` to build the package, with an override for your publisher ID:
 
 ```bash
 npm run package -- --overrides-file overrides.local.json --rev-version --publisher your-publisher-id-here
 ```
 
 ## Installing the extension
 
-To test the extension in Azure DevOps, you'll first need to build the extension `.vsix` file (see above). After this, [publish your extension](https://learn.microsoft.com/en-us/azure/devops/extend/publish/overview?view=azure-devops#publish-your-extension), then [install your extension](https://learn.microsoft.com/en-us/azure/devops/extend/publish/overview?view=azure-devops#install-your-extension).
+To test the extension in a Azure DevOps organisation:
+1. [Build the extension `.vsix` package](#building-the-extension)
+1. [Publish the extension to your publisher account](https://learn.microsoft.com/en-us/azure/devops/extend/publish/overview?view=azure-devops#publish-your-extension)
+1. [Share the extension with the organisation](https://learn.microsoft.com/en-us/azure/devops/extend/publish/overview?view=azure-devops#share-your-extension).
 
 ## Running the task locally
-
+To run the latest task version:
 ```bash
 npm start
 ```
 
+To run a specific task version:
+```bash
+npm run start:V1 # runs dependabotV1 task
+npm run start:V2 # runs dependabotV2 task
+```
 ## Running the unit tests
 
 ```bash
 cd extension
 npm test
 ```
+
+# Architecture
+
+## Task V2 high-level update process diagram
+High-level sequence diagram illustrating how the `dependabotV2` task performs updates using [dependabot-cli](https://github.com/dependabot/cli). For more technical details, see [how dependabot-cli works](https://github.com/dependabot/cli?tab=readme-ov-file#how-it-works).
+
+```mermaid
+ sequenceDiagram
+    participant ext as Dependabot DevOps Extension
+    participant agent as DevOps Pipeline Agent
+    participant devops as DevOps API
+    participant cli as Dependabot CLI
+    participant core as Dependabot Updater
+    participant feed as Package Feed
+
+    ext->>ext: Read and parse `dependabot.yml`
+    ext->>ext: Write `job.yaml`
+    ext->>agent: Download dependabot-cli from github
+    ext->>+cli: Execute `dependabot update -f job.yaml -o update-scenario.yaml`
+    cli->>+core: Run update for `job.yaml` with proxy and dependabot-updater docker containers
+    core->>devops: Fetch source files from repository
+    core->>core: Discover dependencies
+    loop for each dependency
+        core->>feed: Fetch latest version
+        core->>core: Update dependency files
+    end
+    core-->>-cli: Report outputs
+    cli->>cli: Write outputs to `update-sceario.yaml`
+    cli-->>-ext: Update completed
+
+    ext->>ext: Read and parse `update-sceario.yaml`
+    loop for each output
+      alt when output is "create_pull_request"
+        ext->>devops: Create pull request source branch
+        ext->>devops: Push commit to source branch
+        ext->>devops: Create pull request
+        ext->>devops: Set auto-approve
+        ext->>devops: Set auto-complete
+      end
+      alt when output is "update_pull_request"
+        ext->>devops: Push commit to pull request
+        ext->>devops: Update pull request description
+        ext->>devops: Set auto-approve
+        ext->>devops: Set auto-complete
+      end
+      alt when output is "close_pull_request"
+        ext->>devops: Create comment thread on pull request with close reason
+        ext->>devops: Abandon pull request
+        ext->>devops: Delete source branch
+      end
+    end
+
+```

docs/migrations/v1-to-v2.md

@@ -0,0 +1,64 @@
+
+> [!WARNING]
+> **:construction: Work in progress;** `dependabot@V2` is still under development and this document may change without notice up until general availability (GA).
+
+# Table of Contents
+- [Summary of changes V1 → V2](#summary-of-changes-v1-v2)
+- [Breaking changes V1 → V2](#breaking-changes-v1-v2)
+- [Todo before general availability](#todo-before-general-availability)
+
+# Summary of changes V1 → V2
+V2 is a complete re-write of the Dependabot task; It aims to:
+
+- Resolve the [numerous private feed/registry authentication issues](https://github.com/tinglesoftware/dependabot-azure-devops/discussions/1317) that currently exist in V1;
+- More closely align the update logic with the GitHub-hosted Dependabot service;
+
+The task now uses [Dependabot CLI](https://github.com/dependabot/cli) to perform dependency updates, which is the _[currently]_ recommended approach for running Dependabot. See [extension task architecture](../extension.md#architecture) for more details on the technical changes and impact to the update process.
+
+# Breaking changes V1 → V2
+
+> [!WARNING]
+> **It is strongly recommended that you complete (or abandon) all active Depedabot pull requests created in V1 before migrating to V2.** Due to changes in Dependabot dependency metadata, V2 pull requests are not compatible with V1 (and vice versa). Migrating to V2 before completing existing pull requests will lead to duplication of pull requests.
+
+### New pipeline agent requirements; "Go" must be installed
+Dependabot CLI requires [Go](https://go.dev/doc/install) (1.22+) and [Docker](https://docs.docker.com/get-started/get-docker/) (with Linux containers).
+If you use [Microsoft-hosted agents](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software), we recommend using the [ubuntu-latest](https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md) image, which meets all task requirements.
+For self-hosted agents, you will need to install Go 1.22+.
+
+### Security-only updates and "fixed vulnerabilities" are not implemented (yet)
+Using configuration `open-pull-requests-limit: 0` will cause a "not implemented" error. This is [current limitation of V2](../../README.md#unsupported-features-and-configurations). A solution is still under development and is expected to be resolved before general availability.
+See: https://github.com/dependabot/cli/issues/360 for more technical details.
+
+### Task Input `updaterOptions` has been renamed to `experiments`
+Renamed to match Dependabot Core/CLI terminology. The input value remains unchanged. See [configuring experiments](../../README.md#configuring-experiments) for more details.
+
+### Task Input `failOnException` has been removed
+Due to the design of Dependabot CLI, the update process can no longer be interrupted once the update has started. Because of this, the update will now continue on error and summarise all error at the end of the update process.
+
+### Task Input `excludeRequirementsToUnlock` has been removed
+This was a customisation/workaround specific to the V1 update script that can no longer be implemented with Dependabot CLI as it is not an official configuration option.
+
+### Task Input `dockerImageTag` has been removed
+This is no longer required as the [custom] [Dependabot Updater image](../updater.md) is no longer used.
+
+### Task Input `extraEnvironmentVariables` has been removed
+Due to the containerised design of Dependabot CLI, environment variables can no longer be passed from the task to the updater process. All Dependabot config must now set via `dependabot.yaml` or as task inputs. The following old environment variables have been converted to task inputs:
+
+| Environment Variable | New Task Input |
+|--|--|
+|DEPENDABOT_AUTHOR_EMAIL|authorEmail|
+|DEPENDABOT_AUTHOR_NAME|authorName|
+
+
+## Todo before general availability
+Before removing the preview flag from V2 `task.json`, we need to:
+ - [x] Open an issue in Dependabot-CLI, enquire how security-advisories are expected to be provided **before** knowing the list of dependencies. (https://github.com/dependabot/cli/issues/360)
+ - [ ] Convert GitHub security advisory client in `vulnerabilities.rb` to TypeScript code
+ - [ ] Implement `security-advisories` config once the answer the above is known
+ - [x] Review `task.json`, add documentation for new V2 inputs
+ - [x] Update `\docs\extension.md` with V2 docs
+ - [x] Update `\extension\README.MD` with V2 docs
+ - [x] Update `\README.MD` with V2 docs
+ - [ ] Do a general code tidy-up pass (check all "TODO" comments)
+ - [ ] Add unit tests for V2 utils scripts
+ - [ ] Investigate https://zod.dev/

docs/server.md

@@ -4,10 +4,10 @@
 - [Why should I use the server?](#why-should-i-use-the-server)
 - [Composition](#composition)
 - [Deployment](#deployment)
-  - [Single click deployment](#single-click-deployment)
-  - [Deployment Parameters](#deployment-parameters)
-  - [Deployment with CLI](#deployment-with-cli)
-  - [Service Hooks and Subscriptions](#service-hooks-and-subscriptions)
+   * [Single click deployment](#single-click-deployment)
+   * [Deployment Parameters](#deployment-parameters)
+   * [Deployment with CLI](#deployment-with-cli)
+   * [Service Hooks and Subscriptions](#service-hooks-and-subscriptions)
 - [Keeping updated](#keeping-updated)
 
 # Why should I use the server?
@@ -59,10 +59,12 @@ The deployment exposes the following parameters that can be tuned to suit the se
 |githubToken|Access token for authenticating requests to GitHub. Required for vulnerability checks and to avoid rate limiting on free requests|No|<empty>|
 |imageTag|The image tag to use when pulling the docker containers. A tag also defines the version. You should avoid using `latest`. Example: `1.1.0`|No|<version-downloaded>|
 
+> [!NOTE]
 > The template includes a User Assigned Managed Identity, which is used when performing Azure Resource Manager operations such as deletions. In the deployment it creates the role assignments that it needs. These role assignments are on the resource group that you deploy to.
 
 ## Deployment with CLI
 
+> [!IMPORTANT]
 > Ensure the Azure CLI tools are installed and that you are logged in.
 
 For a one time deployment, it is similar to how you deploy other resources on Azure.

docs/updater.md

@@ -1,24 +1,27 @@
 
+> [!WARNING]
+> **Deprecated;** Use of the Dependabot Updater image is no longer recommended since v2.0; The "updater" component is considered an internal to Dependabot and is not intended to be run directly by end-users. There are known limitations with this image, see [unsupported features and configuration](../README.md#unsupported-features-and-configurations) for more details.
+
 # Table of Contents
 
 - [Running the updater](#running-the-updater)
-  - [Environment variables](#environment-variables)
+   * [Environment Variables](#environment-variables)
 - [Development guide](#development-guide)
-  - [Getting the development environment ready](#getting-the-development-environment-ready)
-  - [Building the Docker image](#building-the-docker-image)
-  - [Running your code changes](#running-your-code-changes)
-  - [Running the code linter](#running-the-code-linter)
-  - [Running the unit tests](#running-the-unit-tests)
+   * [Getting the development environment ready](#getting-the-development-environment-ready)
+   * [Building the Docker image](#building-the-docker-image)
+   * [Running your code changes](#running-your-code-changes)
+   * [Running the code linter](#running-the-code-linter)
+   * [Running the unit tests](#running-the-unit-tests)
 
 # Running the updater
 
-First, you need to pull the docker image locally to your machine:
+[Build](#building-the-docker-image) or pull the docker image:
 
 ```bash
 docker pull ghcr.io/tinglesoftware/dependabot-updater-<ecosystem>
 ```
 
-Next create and run a container from the image. The full list of container options are detailed in [Environment variables](#environment-variables); at minimum the command should be:
+Create and run a container based on the image. The full list of container options are detailed in [environment variables](#environment-variables); at minimum the command should be:
 
 ```bash
 docker run --rm -t \
@@ -38,7 +41,8 @@ docker run --rm -t \
            ghcr.io/tinglesoftware/dependabot-updater-<ecosystem> update_script
 ```
 
-An example, for Azure DevOps Services:
+<details>
+<summary>Example, for Azure DevOps Services</summary>
 
 ```bash
 docker run --rm -t \
@@ -55,7 +59,10 @@ docker run --rm -t \
            ghcr.io/tinglesoftware/dependabot-updater-nuget update_script
 ```
 
-An example, for Azure DevOps Server:
+</details>
+
+<details>
+<summary>Example, for Azure DevOps Server</summary>
 
 ```bash
 docker run --rm -t \
@@ -75,9 +82,11 @@ docker run --rm -t \
            ghcr.io/tinglesoftware/dependabot-updater-nuget update_script
 ```
 
+</details>
+
 ## Environment Variables
 
-To run the script, some environment variables are required.
+The following environment variables are required when running the container.
 
 |Variable Name|Supported Command(s)|Description|
 |--|--|--|
@@ -135,10 +144,12 @@ To run the script, some environment variables are required.
 
 ## Getting the development environment ready
 
-First, ensure you have [Docker](https://docs.docker.com/engine/install/) and [Ruby](https://www.ruby-lang.org/en/documentation/installation/) installed.
-On Linux, you'll need the the build essentials and Ruby development packages too; These are typically `build-essentials` and `ruby-dev`.
+Install [Docker](https://docs.docker.com/engine/install/) and [Ruby](https://www.ruby-lang.org/en/documentation/installation/).
+
+> [!NOTE]
+> If developing in Linux, you'll also need the the build essentials and Ruby development packages; These are typically `build-essentials` and `ruby-dev`.
 
-Next, install project build tools with bundle:
+Install the project build tools using Bundle:
 
 ```bash
 cd updater
@@ -159,21 +170,23 @@ docker build \
     .
 ```
 
-In some scenarios, you may want to set `BASE_VERSION` to a specific version instead of "latest".
-See [updater/Dockerfile](../updater/Dockerfile) for a more detailed explanation.
+> [!TIP]
+> In some scenarios, you may want to set `BASE_VERSION` to a specific version instead of "latest".
+> See [updater/Dockerfile](../updater/Dockerfile) for a more detailed explanation.
 
 ## Running your code changes
 
-To test run your code changes, you'll first need to build the updater Docker image (see above), then run the updater Docker image in a container with all the required environment variables (see above).
+To test run your code changes, you'll first need to [build the Docker image](#building-the-docker-image), then run the Docker image in a container with all the [required environment variables](#environment-variables).
 
 ## Running the code linter
 
 ```bash
 cd updater
 bundle exec rubocop
-bundle exec rubocop -a # to automatically fix any correctable offenses
 ```
 
+> [!TIP]
+> To automatically fix correctable linting issues, use `bundle exec rubocop -a`
 ## Running the unit tests
 
 ```bash