Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix GitHub rate limiting when generating pull request descriptions #1362

Merged

Conversation

rhyskoedijk
Copy link
Contributor

@rhyskoedijk rhyskoedijk commented Sep 28, 2024

Refs #1356.

Dependabot hits api.github.com to fetch project info, release notes, and commit messages for dependencies when generating pull request descriptions. Currently these requests are unauthenticated because we do not set the GitHub access token in the job credentials, which leads to errors like:

2024-09-27T18:37:21.5657248Z   proxy | 2024/09/27 18:37:21 [952] Remote response: {"message":"API rate limit exceeded for xxx.xx.xx.x. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}
2024-09-27T18:37:21.5695270Z updater | 2024/09/27 18:37:21 ERROR <job_update_0_npm_and_yarn_all> Error while generating PR message: GET https://api.github.com/repos/angular/angular/contents/packages/language-service: 403 - API rate limit exceeded for xxx.xx.xx.x . (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) // See: https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting
2024-09-27T18:37:21.5697807Z updater | 2024/09/27 18:37:21 ERROR <job_update_0_npm_and_yarn_all> /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/octokit-7.2.0/lib/octokit/response/raise_error.rb:14:in `on_complete'
2024-09-27T18:37:21.5698373Z updater | /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/faraday-2.7.11/lib/faraday/middleware.rb:18:in `block in call'

This change adds a credential entry for github.com if the GitHub access token task input is set.

This change also forwards the Azure DevOps access token to dependabot via the LOCAL_AZURE_ACCESS_TOKEN environment variable, which is used here. It doesn't currently change any behaviour since the token is already supplied explicitly in the job credentials; Just added for consistency.

@rhyskoedijk rhyskoedijk marked this pull request as ready for review September 28, 2024 13:48
@mburumaxwell mburumaxwell merged commit 4f3ffc0 into tinglesoftware:main Sep 28, 2024
3 checks passed
@rhyskoedijk rhyskoedijk deleted the feature/fix-github-rate-limiting branch September 28, 2024 14:31
kzhuklinets added a commit to kirillcoso/dependabot-azure-devops that referenced this pull request Oct 4, 2024
* Update the server version/component to support directories and groups (tinglesoftware#1294)

* Update .gitignore

* Formatting fixes

* Update dependabot.yml

* Add prettier formatting (tinglesoftware#1304)

This will mostly format files in the extension but can format some other non-js/ts files that it supports like json, yaml, etc.

At the root `package.json` and `package-lock.json` files are added but a workspace is not used because we do not need it and it is preferred for the extension to be independent to ensure it is self-contained during deployment. This may change if we can use `tsup` to package it.

* Update packages to be included in the microsoft group

* Bump @types/node in /extension in the js-ts-types group (tinglesoftware#1296)

* Bump ts-jest from 29.2.4 to 29.2.5 in /extension in the jest group (tinglesoftware#1295)

* Bump axios from 1.7.4 to 1.7.5 in /extension (tinglesoftware#1298)

* Bump vcr from 6.2.0 to 6.3.1 in /updater (tinglesoftware#1302)

* Bump Azure.Monitor.Query from 1.4.0 to 1.5.0 in the azure group (tinglesoftware#1299)

* Add missing EFCore migration for directories and groups support (tinglesoftware#1305)

I added support for directories and groups in tinglesoftware#1294 but forgot to add the database migration. Possibly because I do not have unit tests against the actual database.

* Bump Microsoft.NET.Test.Sdk from 17.10.0 to 17.11.0 (tinglesoftware#1306)

* Bump dotnet-ef from 8.0.7 to 8.0.8 (tinglesoftware#1303)

Bumps [dotnet-ef](https://github.com/dotnet/efcore) from 8.0.7 to 8.0.8.
- [Release notes](https://github.com/dotnet/efcore/releases)
- [Commits](dotnet/efcore@v8.0.7...v8.0.8)

---
updated-dependencies:
- dependency-name: dotnet-ef
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump azure-pipelines-task-lib from 4.16.0 to 4.17.0 in /extension (tinglesoftware#1308)

Bumps [azure-pipelines-task-lib](https://github.com/Microsoft/azure-pipelines-task-lib) from 4.16.0 to 4.17.0.
- [Commits](https://github.com/Microsoft/azure-pipelines-task-lib/commits)

---
updated-dependencies:
- dependency-name: azure-pipelines-task-lib
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Test that token replacement works when padded with spaces

* Update docs and README.md

* Add swift to `ECOSYSTEM_LOOKUP` for vulnerabilities (tinglesoftware#1309)

* Improvement to credentials handling (tinglesoftware#1310)

- Handle setting `index-url` instead of `url` for `python_index` registries in the server version/component.
- Handle setting both `host` and `url` for `composer_repository` registries.
- Better logic around building extra credentials.

> This is likely the last modification before we try to move to the proxy used by `dependabot-cli`.

* Merge `mv:prod` script into `build:prod`

* Formatting fixes

* Exclude .idea folders from prettier

* Regenerate Gemfile.lock which should vulnerability in Rexml

* Register span processor for sentry to fix OTel export (tinglesoftware#1311)

This should fix the OpenTelemetry export issue.
It may be reverted in the future when `dependabot-core/updater` makes other changes.

See https://docs.sentry.io/platforms/ruby/tracing/instrumentation/opentelemetry/

* Update updater.yml

* Update extension task node version from v16 to v20.1 (tinglesoftware#1313)

* Update task node version from 16 to 20.1

* Bump minimum agent version

* Fix vNext AZURE_VIRTUAL_DIRECTORY is not working bug (tinglesoftware#1316)

* Bump @types/node (tinglesoftware#1322)

* Bump YamlDotNet from 16.0.0 to 16.1.0 (tinglesoftware#1325)

* Bump Microsoft.Data.SqlClient from 5.2.1 to 5.2.2 (tinglesoftware#1324)

* Bump axios from 1.7.5 to 1.7.7 in /extension (tinglesoftware#1323)

* Bump opentelemetry-exporter-otlp in /updater in the opentelemetry group (tinglesoftware#1319)

* Restructure extension to better support multiple task versions (tinglesoftware#1326)

* Restructure extension task to better support multiple version

* Fix typos

* Fix build

* Fix build

* Bump @types/node (tinglesoftware#1327)

Bumps the js-ts-types group with 1 update in the /extension directory: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node).


Updates `@types/node` from 22.5.2 to 22.5.3
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: js-ts-types
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump Microsoft.NET.Test.Sdk from 17.11.0 to 17.11.1 (tinglesoftware#1329)

Bumps [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest) from 17.11.0 to 17.11.1.
- [Release notes](https://github.com/microsoft/vstest/releases)
- [Changelog](https://github.com/microsoft/vstest/blob/main/docs/releases.md)
- [Commits](microsoft/vstest@v17.11.0...v17.11.1)

---
updated-dependencies:
- dependency-name: Microsoft.NET.Test.Sdk
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @types/node (tinglesoftware#1332)

Bumps the js-ts-types group with 1 update in the /extension directory: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node).


Updates `@types/node` from 22.5.3 to 22.5.4
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: js-ts-types
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dependabot-omnibus from 0.268.0 to 0.274.0 in /updater (tinglesoftware#1331)

Bumps [dependabot-omnibus](https://github.com/dependabot/dependabot-core) from 0.268.0 to 0.274.0.
- [Release notes](https://github.com/dependabot/dependabot-core/releases)
- [Changelog](https://github.com/dependabot/dependabot-core/blob/main/CHANGELOG_ARCHIVE_2019_TO_SWITCH_TO_GITHUB_RELEASES.md)
- [Commits](dependabot/dependabot-core@v0.268.0...v0.274.0)

---
updated-dependencies:
- dependency-name: dependabot-omnibus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Format fix

* Install ice and jq inside the @azure/cli task

* Bump typescript from 5.5.4 to 5.6.2 in /extension (tinglesoftware#1337)

* Bump azure-pipelines-task-lib from 4.17.0 to 4.17.2 in /extension (tinglesoftware#1338)

* Bump dependabot-omnibus from 0.274.0 to 0.275.0 in /updater (tinglesoftware#1339)

Bumps [dependabot-omnibus](https://github.com/dependabot/dependabot-core) from 0.274.0 to 0.275.0.
- [Release notes](https://github.com/dependabot/dependabot-core/releases)
- [Changelog](https://github.com/dependabot/dependabot-core/blob/main/CHANGELOG_ARCHIVE_2019_TO_SWITCH_TO_GITHUB_RELEASES.md)
- [Commits](dependabot/dependabot-core@v0.274.0...v0.275.0)

---
updated-dependencies:
- dependency-name: dependabot-omnibus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump YamlDotNet from 16.1.0 to 16.1.2 (tinglesoftware#1340)

Bumps [YamlDotNet](https://github.com/aaubry/YamlDotNet) from 16.1.0 to 16.1.2.
- [Release notes](https://github.com/aaubry/YamlDotNet/releases)
- [Commits](aaubry/YamlDotNet@v16.1.0...v16.1.2)

---
updated-dependencies:
- dependency-name: YamlDotNet
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @types/jest in /extension in the jest group across 1 directory (tinglesoftware#1341)

Bumps the jest group with 1 update in the /extension directory: [@types/jest](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jest).


Updates `@types/jest` from 29.5.12 to 29.5.13
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jest)

---
updated-dependencies:
- dependency-name: "@types/jest"
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: jest
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @types/node (tinglesoftware#1342)

Bumps the js-ts-types group with 1 update in the /extension directory: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node).


Updates `@types/node` from 22.5.4 to 22.5.5
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: js-ts-types
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Sync updater files to version 0.275.0 (tinglesoftware#1343)

* Bump Azure.ResourceManager.AppContainers in the azure group (tinglesoftware#1336)

Bumps the azure group with 1 update: [Azure.ResourceManager.AppContainers](https://github.com/Azure/azure-sdk-for-net).


Updates `Azure.ResourceManager.AppContainers` from 1.2.0 to 1.3.0
- [Release notes](https://github.com/Azure/azure-sdk-for-net/releases)
- [Commits](Azure/azure-sdk-for-net@Azure.ResourceManager.AppContainers_1.2.0...Azure.ResourceManager.AppContainers_1.3.0)

---
updated-dependencies:
- dependency-name: Azure.ResourceManager.AppContainers
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: azure
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Remove workaround for reading ContainerApp Job execution (tinglesoftware#1344)

The library has since been updated; we no longer need a workaround.

See also, Azure/azure-sdk-for-net#38385

* Add Task V2; Use Dependabot-CLI to perform updates (tinglesoftware#1318)

* Split extension task in to V1 and V2

* Basic support for running update using dependabot-cli

* Add missing update job configs

* Move update output processing to dedicated class; Add DevOps API client

* Codespell skip package-lock.json

* Clean-up

* Implement create pull request

* Implement groups, auto-complete, auto-approve, pull request properties, and updating existing pull requests

* Clean up

* Set task V1 as deprecated, task V2 as preview

* Restructure extension task to better support multiple version

* Fix typos

* Fix build

* Fix build

* Fix merge issues

* Remove unused task inputs

* Fix for 'convertPlaceholder' not accepted built-in DevOps variable names containing '.'

* Implement closing pull requests

* Implement updating pull requests

* Use default branch name if target branch not configured

* Implement approving pull requests

* Add task inputs for pr commit author email and name

* Implement open pull request limit config

* Cleanup temporary files after task completion

* Add configuration placeholders for dependabot component images

* Implement more config options

* Implement dependency list snapshots, which are stored in the DevOps project properties

* Add task input option for storing dependency list

* Implement experiments

* Implement requirements-update-strategy and lockfile-only configs

* Fix typo

* Implement PR reviewers, work item references, and labels

* Add start commands for each task version, use V2 by default

* Update V1 task.json version numbers when publishing the extension

* Update documentation

* Update documentation

* Update documentation

* Update documentation

* Typo

* Update documentation

* Fix reference to undefined `this.cachedUserIds`

* Use case insensitive comparision when parsing "System.Debug" variable

* Fix dependabot tool path detection in agents where `$PATH` does not contain `$GOPATH/bin`

* Add more logging

* Fix for task reporting success when pull request creation failed

* Add more logging; Fix formatting

* Fix 'labels' config parsing

* Implement "targetUpdateIds" task input option

* Fix error when using multiple update blocks in dependabot.yml with the same package manager

* Only install dependabot once; cache the tool path once known

* Add migration warning to complete V1 pull requests before migrating to V2

* Process updates synchronously when using multiple update blocks in dependabot.yml

* Fix typos

* Report the total number of failed update jobs in the task result

* Include stack trace when errors are logged, to help with diagnosing issues

* Fix inverted logic for "abandonUnwantedPullRequests"

* Fix error handling

* Check formatting (prettier) in the extension workflow (tinglesoftware#1345)

* Fix version for `azure-devops-node-api`

* Set `$schema` on vss-extension.json

* Reduce nesting of tasks by one level (tinglesoftware#1346)

This is likely to help with discovery which only checks upto 1 level

https://learn.microsoft.com/en-us/azure/devops/extend/develop/integrate-build-task?toc=%2Fazure%2Fdevops%2Fmarketplace-extensibility%2Ftoc.json&view=azure-devops#multiple-versions-layout

* Change identifier for v2 task

* Revert "Change identifier for v2 task"

This reverts commit c5d84ce.

* Change name for v2 task

* Bump prettier-plugin-organize-imports from 4.0.0 to 4.1.0 (tinglesoftware#1352)

* Bump xunit from 2.9.0 to 2.9.1 in the xunit group (tinglesoftware#1351)

* Bump the tingle group with 3 updates (tinglesoftware#1350)

* Add contributing guide; update developer guides (tinglesoftware#1347)

* Add contributing guide

* Waffling

* Update toc

* Fix broken link

* Bump dependabot-omnibus from 0.275.0 to 0.276.0 in /updater (tinglesoftware#1349)

Bumps [dependabot-omnibus](https://github.com/dependabot/dependabot-core) from 0.275.0 to 0.276.0.
- [Release notes](https://github.com/dependabot/dependabot-core/releases)
- [Changelog](https://github.com/dependabot/dependabot-core/blob/main/CHANGELOG_ARCHIVE_2019_TO_SWITCH_TO_GITHUB_RELEASES.md)
- [Commits](dependabot/dependabot-core@v0.275.0...v0.276.0)

---
updated-dependencies:
- dependency-name: dependabot-omnibus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Revert "Change name for v2 task"

This reverts commit 1e24c54.

* Set `argumentFormat`

* Do not override major version numbers in task.json files

* Fix undefined iteration

* Fix "cannot read property of undefined" error; Fix GitHub image pull rate-limiting (tinglesoftware#1353)

* Fix reading undefined reference when there is no dependency list stored for the project yet (i.e. a first-time run)

* Forward GitHub access token to dependabot-cli to avoid rate-limiting when pulling Docker images; Set Dependabot job id to aid with debugging

* Fix formatting

* Fix another "cannot read property of undefined" error (tinglesoftware#1354)

* Fix reading undefined reference when there are no existing pull requests to update yet

* Improved logging to ensure error stack traces are always dumped; helps with debugging

* Regenerate updater/Gemfile.lock, should fix some security concerns

* Sync updater files to version 0.276.0

* Attempt to fix brain fart from https://github.com/rhyskoedijk

See tinglesoftware#1354 (comment)

* Set `IncludeSourceRevisionInInformationalVersion=false` to allow GitVersion to work

See https://learn.microsoft.com/en-us/dotnet/core/compatibility/sdk/8.0/source-link

* Update known experiments list (tinglesoftware#1361)

* Fix "cannot read properties of null" (tinglesoftware#1359)

* Fix "undefined is not iterable" (tinglesoftware#1358)

* Fix GitHub rate limiting when generating pull request descriptions (tinglesoftware#1362)

* Set github registry credentials to avoid rate-limiting when generating pull request descriptions

* Set github registry credentials to avoid rate-limiting when generating pull request descriptions

* Run `go install dependabot` async so logs are streamed to task output instead of blocking until process has completed (tinglesoftware#1357)

* Log the number of pull requests that would have been updated when `skipPullRequests` is set (tinglesoftware#1360)

* Fix "cannot read properties of undefined" (reading 'filter') (tinglesoftware#1364)

* Fix "pull request cannot be activated because the source and/or the target branch no longer exists" error (tinglesoftware#1366)

* Strip reference prefix from the default branch name, the caller doesn't need to know this

* Use regex, match start of string, ignore case

* Upload flame graph report to pipeline timeline when `System.Debug` is `true` (tinglesoftware#1363)

* Attach flamegraph report to pipeline timeline when `System.Debug` is `true`

* Add documentation

* Fix formatting

* Bump xunit from 2.9.1 to 2.9.2 in the xunit group (tinglesoftware#1369)

* Bump Azure.Identity from 1.12.0 to 1.12.1 in the azure group (tinglesoftware#1368)

* Bump the sentry group in /updater with 2 updates (tinglesoftware#1373)

* Bump @types/node (tinglesoftware#1371)

* Bump YamlDotNet from 16.1.2 to 16.1.3 (tinglesoftware#1370)

* Bump azure-devops-node-api from 14.0.2 to 14.1.0 in /extension (tinglesoftware#1372)

* Bump dependabot-omnibus from 0.276.0 to 0.278.0 in /updater (tinglesoftware#1374)

Bumps [dependabot-omnibus](https://github.com/dependabot/dependabot-core) from 0.276.0 to 0.278.0.
- [Release notes](https://github.com/dependabot/dependabot-core/releases)
- [Changelog](https://github.com/dependabot/dependabot-core/blob/main/CHANGELOG_ARCHIVE_2019_TO_SWITCH_TO_GITHUB_RELEASES.md)
- [Commits](dependabot/dependabot-core@v0.276.0...v0.278.0)

---
updated-dependencies:
- dependency-name: dependabot-omnibus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix merge strategy parsing and default value (tinglesoftware#1376)

* Fix "unknown package manager" error (tinglesoftware#1377)

* Add support for the "pull-request-branch-name.separator" config option (tinglesoftware#1378)

* Add support for "pull-request-branch-name.separator" config

* Update migration documentation

* Apply suggestions from code review

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Maxwell Weru <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rhys Koedijk <[email protected]>
Co-authored-by: Yurii Yakymenko <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants