Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADO API responds with GitObjectDoesNotExistException when pushing file changes to branch #1413

Closed
grixxie opened this issue Oct 18, 2024 · 36 comments · Fixed by #1478
Closed

Comments

@grixxie
Copy link

grixxie commented Oct 18, 2024

Hi,

I am facing the following problem with my dependabot (v2) pipeline. My pipeline looks like so:

stages:
- stage: dependabot
  jobs:
  - job: dependabot
    pool:
      name: my-selfhosted-pool
    steps:
    - task: dependabot@2
      displayName: 'Run Dependabot'
      inputs:
        setAutoComplete: true
        abandonUnwantedPullRequests: false
        azureDevOpsAccessToken: $(System.AccessToken)
        gitHubAccessToken: $(GHTOKEN)

My dependabot.yml config looks like so:

version: 2
updates:
  - package-ecosystem: "nuget"
    directory: "/"
    open-pull-requests-limit: 10
    target-branch: 'dependabot/releases/release-15'
    schedule:
      interval: "daily"

My pipeline is erroring with the following error snippet, when it tries to call the /pushes endpoint of my repo via the ADO API to push the file changed. It is complaining that an object does not exist; I am not sure what this object is refering to. My release-15 branch mentioned in my dependabot.yml definitely exists and is a different ID to the one mentioned in the error...
The error:

Creating pull request 'Bump Microsoft.AspNetCore.SpaServices.Extensions from 8.0.8 to 8.0.10'...
 - Pushing 1 file change(s) to branch 'dependabot/nuget/dependabot/releases/release-15/Microsoft.AspNetCore.SpaServices.Extensions-8.0.10'...
🌎 🠊 [POST] https://dev.azure.com/ORG/PROJECT/_apis/git/repositories/REPO/pushes
🌎 🠈 [500] Internal Server Error
{"$id":"1","innerException":null,"message":"TF401035: The object '5eae8e97b8105c3b55acb7ed3d41233062e2bcf7' does not exist.","typeName":"Microsoft.TeamFoundation.Git.Server.GitObjectDoesNotExistException, Microsoft.TeamFoundation.Git.Server","typeKey":"GitObjectDoesNotExistException","errorCode":0,"eventId":3000}
##[error]Failed to create pull request: Error: Request to 'https://dev.azure.com/ORG/PROJECT/_apis/git/repositories/REPO/pushes' failed: 500 Internal Server Error
Error: Request to 'https://dev.azure.com/ORG/PROJECT/_apis/git/repositories/REPO/pushes' failed: 500 Internal Server Error
    at AzureDevOpsWebApiClient.restApiRequest (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/azure-devops/AzureDevOpsWebApiClient.js:521:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AzureDevOpsWebApiClient.restApiPost (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/azure-devops/AzureDevOpsWebApiClient.js:498:16)
    at async AzureDevOpsWebApiClient.createPullRequest (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/azure-devops/AzureDevOpsWebApiClient.js:156:26)
    at async DependabotOutputProcessor.process (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/dependabot-cli/DependabotOutputProcessor.js:67:42)
    at async DependabotCli.update (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/dependabot-cli/DependabotCli.js:113:51)
    at async run (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/index.js:75:57)

Please advise if I am doing something wrong. If you need any more information then please let me know and I will provide. Thank you.
If anyone else is also experiencing this issue please respond!

@rhyskoedijk
Copy link
Contributor

rhyskoedijk commented Oct 21, 2024

@grixxie thanks for the report;
do you have an existing branch named dependabot/releases/release-15? If so, I would guess that this is causing problems as Dependabot is wanting to use the dependabot/ prefix for the pull requests it is trying to create. If not, do you know what branch the commit 5eae8e97b8105c3b55acb7ed3d41233062e2bcf7 belongs to? Is it your target branch or another branch?

If you haven't already tried and it is possible to do so, I'd recommend renaming any branches starting with dependabot/; I'll look at adding a config option to change this prefix as it sounds like there are a few people who are already using this prefix.

EDIT: Branch name is not the cause of the issue; tested below

@rhyskoedijk
Copy link
Contributor

rhyskoedijk commented Oct 22, 2024

just a follow up to this; I've tried to reproduce your issue by doing the following:

  • Run update targeting default branch when an unrelated branch named dependabot/releases/release-15 already exists;
  • Run update targeting dependabot/releases/release-15 when dependabot/releases/release-15 already exists;
  • Run update targeting dependabot/releases/release-15 when dependabot/releases/release-15 does not exist;

All of these scenarios work without error; I was not able to reproduce the issue.

Are you able to provide any more information that would help reproduce your issue? Can you identify what branch git commit 5eae8e97b8105c3b55acb7ed3d41233062e2bcf7 belongs to?

@grixxie
Copy link
Author

grixxie commented Oct 22, 2024

Hi - thanks for the response.

I also tested rerunning v2 with changes to the target branch name and ran into my same issue.

To answer your question, I have no idea where that commit is coming from, I could not find it when I was doing my testing yesterday. I will be doing some more testing later today though so will let you know the outcome of that.

You not being able to reproduce this issue definitely makes it seem like I'm doing something not quite right though...

@DaleMckeown
Copy link

DaleMckeown commented Oct 22, 2024

I have also just tried to use v2 for the first time and and running into this issue.

Creating pull request 'Bump the npm group in /{folderName} with 17 updates'...
 - Pushing 1 file change(s) to branch 'dependabot/npm_and_yarn/dependabot/dependabot_grouping_v2/{folderName}/npm-d631ece976'...
🌎 🠊 [POST] https://dev.azure.com/{orgName}/{projectName}/_apis/git/repositories/{repoName}/pushes
🌎 🠈 [500] Internal Server Error
{"$id":"1","innerException":null,"message":"TF401035: The object 'abdceb5590885eeb4810012ade73d8125ae03864' does not exist.","typeName":"Microsoft.TeamFoundation.Git.Server.GitObjectDoesNotExistException, Microsoft.TeamFoundation.Git.Server","typeKey":"GitObjectDoesNotExistException","errorCode":0,"eventId":3000}
##[error]Failed to create pull request: Error: Request to 'https://dev.azure.com/{orggName}/{projectName}/_apis/git/repositories/{projectName}/pushes' failed: 500 Internal Server Error
##[debug]Processed: ##vso[task.issue type=error;source=TaskInternal;]Failed to create pull request: Error: Request to 'https://dev.azure.com/{orggName}/{projectName}/_apis/git/repositories/{repoName}/pushes' failed: 500 Internal Server Error
Error: Request to 'https://dev.azure.com/{orgName}/{projectName}/_apis/git/repositories/{repoName}/pushes' failed: 500 Internal Server Error
    at AzureDevOpsWebApiClient.restApiRequest (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/azure-devops/AzureDevOpsWebApiClient.js:521:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AzureDevOpsWebApiClient.restApiPost (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/azure-devops/AzureDevOpsWebApiClient.js:498:16)
    at async AzureDevOpsWebApiClient.createPullRequest (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/azure-devops/AzureDevOpsWebApiClient.js:156:26)
    at async DependabotOutputProcessor.process (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/dependabot-cli/DependabotOutputProcessor.js:67:42)
    at async DependabotCli.update (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/dependable-cli/DependabotCli.js:113:51)
    at async run (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/index.js:75:57)

I also cannot find that commit ID at all, it must be one being created locally by Dependabot? I've ran the job multiple times and each time the objectId is different.

The branch dependabot/dependabot_grouping_v2 definitely exists.

When running this, I have two package ecosystems. NuGet, which runs successfully, and then NPM, which fails.

In my dependabot config file, I am attempting to use groups:

version: 2
registries:
  nuget-internal-packages:
    type: nuget-feed
    url: https://pkgs.dev.azure.com/{orgName}/_packaging/{projectName}/nuget/v3/index.json
    token: 'PAT:${{PAT}}'
  nuget-public:
    type: nuget-feed
    url: https://api.nuget.org/v3/index.json
  npm-internal-packages:
    type: npm-registry
    url: https://pkgs.dev.azure.com/{orgName}/_packaging/{projectName}/npm/registry/
    token: 'PAT:${{PAT}}'
  npm-public:
    type: npm-registry
    url: https://registry.npmjs.org

updates:
  - package-ecosystem: nuget
    directory: '/'
    registries:
      - nuget-internal-packages
      - nuget-public
    schedule:
      interval: weekly
    open-pull-requests-limit: 50
    target-branch: targetBranchPlaceholder
    ignore:
      - dependency-name: 'Microsoft.AspNetCore'
        update-types: ['version-update:semver-major']
      - dependency-name: 'Microsoft.AspNetCore.*'
        update-types: ['version-update:semver-major']
      - dependency-name: 'UoL.*'
        update-types: ['version-update:semver-major']

  - package-ecosystem: npm
    directory: npmDirectoryPlaceholder
    registries:
      - npm-internal-packages
      - npm-public
    schedule:
      interval: weekly
    open-pull-requests-limit: 50
    target-branch: targetBranchPlaceholder
    groups:
      npm:
        applies-to: version-updates
        patterns:
        - "*"
        update-types:
        - "minor"
        - "patch"

@grixxie
Copy link
Author

grixxie commented Oct 22, 2024

@rhyskoedijk I have found the same as @DaleMckeown post above; running the command git branch -a --contains <SHA> returns no results. I also think that this commit is one being created locally by Dependabot, but has failed to push to remote. This would explain why ADO API returns 500 when trying to create the PR.

@DaleMckeown
Copy link

DaleMckeown commented Oct 22, 2024

If I flip the order of my package ecosystems to run NPM first and then NuGet, NPM runs successfully and NuGet errors instead. So it seems to be an issue related to processing multiple package ecosystems in the same Dependabot run.

Also, removing the grouping configuration doesn't have any effect, the same error is thrown.

Just tested again with the NPM config individually, and it works fine. So it's definitely related to having multiple package ecosystems in my case.

@rhyskoedijk
Copy link
Contributor

If I flip the order of my package ecosystems to run NPM first and then NuGet, NPM runs successfully and NuGet errors instead. So it seems to be an issue related to processing multiple package ecosystems in the same Dependabot run.

Thanks @DaleMckeown, that is a good lead and something that I have not tested yet; I will investigate.

@grixxie
Copy link
Author

grixxie commented Oct 22, 2024

If I flip the order of my package ecosystems to run NPM first and then NuGet, NPM runs successfully and NuGet errors instead. So it seems to be an issue related to processing multiple package ecosystems in the same Dependabot run.

Thanks @DaleMckeown, that is a good lead and something that I have not tested yet; I will investigate.

This may be a good lead to find the root of the problem. However, just to point out my dependabot.yml config in the description of this issue, I'm only using a single package ecosystem (NuGet) and still running into this issue!

@rhyskoedijk
Copy link
Contributor

This may be a good lead to find the root of the problem. However, just to point out my dependabot.yml config in the description of this issue, I'm only using a single package ecosystem (NuGet) and still running into this issue!

Understood; At this point I'm just looking for a way to reproduce the error, even if it isn't your original scenario.
I have just tried using a run with multiple package ecosystems (NPM and NuGet), but unfortunately this didn't fail for me either.

image

I am out of ideas for now; If anybody is able to produce a minimal reproduction of their dependabot.yml and *.csproj that causes the error, I will revisit this.

@DaleMckeown
Copy link

DaleMckeown commented Oct 23, 2024

@rhyskoedijk I've scoured the logs again, not sure if this helps at all, but here is the first time the errant commit sha appears in the logs:

updater | 2024/10/22 11:19:08 DEBUG Initializing the background worker with 2 threads
updater | 2024/10/22 11:19:10 INFO <job_update_1_npm_and_yarn_all> Starting job processing
updater | 2024/10/22 11:19:10 INFO <job_update_1_npm_and_yarn_all> Job definition: {"job":{"package-manager":"npm_and_yarn","allowed-updates":[{"dependency-type":"all"}],"debug":true,"dependency-groups":[{"name":"npm","applies-to":"version-updates","rules":{"patterns":["*"],"update-types":["minor","patch"]}}],"dependencies":null,"dependency-group-to-refresh":null,"existing-pull-requests":[],"existing-group-pull-requests":[],"experiments":null,"ignore-conditions":[],"lockfile-only":false,"requirements-update-strategy":null,"security-advisories":[],"security-updates-only":false,"source":{"provider":"azure","repo":"UoLIct/UniLincoln/_git/UniLincoln","directory":"/UniLincolnUI","branch":"dependabot/dependabot_grouping_v2","hostname":"dev.azure.com","api-endpoint":"[https://dev.azure.com:/"},"update-subdependencies":true,"updating-a-pull-request":false,"vendor-dependencies":false,"reject-external-code":false,"repo-private":false,"commit-message-options":null,"credentials-metadata":[{"host":"dev.azure.com","type":"git_source"},{"type":"nuget_feed","url":"https://pkgs.dev.azure.com/UoLIct/_packaging/Packages/nuget/v3/index.json"},{"type":"nuget_feed","url":"https://api.nuget.org/v3/index.json"},{"registry":"pkgs.dev.azure.com/UoLIct/_packaging/Packages/npm/registry/","type":"npm_registry"},{"registry":"registry.npmjs.org","type":"npm_registry"}],"max-updater-run-time":0}}](https://dev.azure.com/%22%7D,%22update-subdependencies%22:true,%22updating-a-pull-request%22:false,%22vendor-dependencies%22:false,%22reject-external-code%22:false,%22repo-private%22:false,%22commit-message-options%22:null,%22credentials-metadata%22:[%7B%22host%22:%22dev.azure.com%22,%22type%22:%22git_source%22%7D,%7B%22type%22:%22nuget_feed%22,%22url%22:%22https://pkgs.dev.azure.com/UoLIct/_packaging/Packages/nuget/v3/index.json%22%7D,%7B%22type%22:%22nuget_feed%22,%22url%22:%22https://api.nuget.org/v3/index.json%22%7D,%7B%22registry%22:%22pkgs.dev.azure.com/UoLIct/_packaging/Packages/npm/registry/%22,%22type%22:%22npm_registry%22%7D,%7B%22registry%22:%22registry.npmjs.org%22,%22type%22:%22npm_registry%22%7D],%22max-updater-run-time%22:0%7D%7D)
updater | 2024/10/22 11:19:11 INFO <job_update_1_npm_and_yarn_all> Base commit SHA: abdceb5590885eeb4810012ade73d8125ae03864
updater | 2024/10/22 11:19:11 INFO <job_update_1_npm_and_yarn_all> Finished job processing
updater | 2024/10/22 11:19:11 DEBUG Killing session flusher
updater | 2024/10/22 11:19:11 DEBUG Shutting down background worker
updater | 2024/10/22 11:19:12 DEBUG Initializing the background worker with 2 threads

It then appears after finding the first package that needs an update. In my case, Quasar:

updater | 2024/10/22 11:19:40 INFO <job_update_1_npm_and_yarn_all> Updating @quasar/extras from 1.16.11 to 1.16.12
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> Dependencies updated:
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all>  - @quasar/extras ( 1.16.11 to 1.16.12 )
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> Dependency files updated:
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all>   - /UniLincolnUI/package.json ( Changed 1 times )
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all>   - /UniLincolnUI/.npmrc 
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> Storing change to workspace: Updating @quasar/extras
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> [workspace] store_change - before: Initial SHA: abdceb5590885eeb4810012ade73d8125ae03864
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> [workspace] store_change - after: 90f51d7e Updating @quasar/extras

It then runs create_pull_request using this base commit successfully:

  proxy | 2024/10/22 11:23:08 [315] POST http://host.docker.internal:42793/update_jobs/update_1_npm_and_yarn_all/create_pull_request
{"data":{"base-commit-sha":"abdceb5590885eeb4810012ade73d8125ae03864","dependencies": ...}}
  proxy | 2024/10/22 11:23:08 [315] 200 http://host.docker.internal:42793/update_jobs/update_1_npm_and_yarn_all/create_pull_request

It repeats the above for each NPM package that needs an update.

It then marks the commit as processed, and outputs the changes detected:

{"data":{"base-commit-sha":"abdceb5590885eeb4810012ade73d8125ae03864"},"type":"mark_as_processed"}
  proxy | 2024/10/22 11:24:59 [290] PATCH http://host.docker.internal:42793/update_jobs/update_1_npm_and_yarn_all/mark_as_processed
  proxy | 2024/10/22 11:24:59 [290] 200 http://host.docker.internal:42793/update_jobs/update_1_npm_and_yarn_all/mark_as_processed
updater | 2024/10/22 11:24:59 INFO <job_update_1_npm_and_yarn_all> Finished job processing
updater | 2024/10/22 11:24:59 INFO Results:
updater | +------------------------------------------------------------------------------------------------------------------------------------+
updater | |                                                Changes to Dependabot Pull Requests                                                 |
updater | +---------+--------------------------------------------------------------------------------------------------------------------------+
updater | | created | @quasar/extras ( from 1.16.11 to 1.16.12 ), @uol/cwd ( from 7.0.0 to 7.0.1 ), @uol/cwd-fontawesome-pro ( from 7.0.0 t... |
updater | | created | @vueuse/core ( from 10.11.1 to 11.1.0 )                                                                                  |
updater | | created | vue-shepherd ( from 3.0.0 to 4.1.0 )                                                                                     |
updater | | created | @quasar/app-vite ( from 2.0.0-beta.14 to 2.0.0-beta.24 )                                                                 |
updater | | created | @types/node ( from 20.14.1 to 22.7.8 )                                                                                   |
updater | | created | @typescript-eslint/eslint-plugin ( from 7.12.0 to 8.11.0 )                                                               |
updater | | created | @typescript-eslint/parser ( from 7.12.0 to 7.18.0 )                                                                      |
updater | +---------+--------------------------------------------------------------------------------------------------------------------------+

It then outputs the create_pull_request response:

Processing output 'create_pull_request' with data: {
  'base-commit-sha': 'abdceb5590885eeb4810012ade73d8125ae03864',
...

Then it errors:

Creating pull request 'Bump the npm group in /UniLincolnUI with 17 updates'...
 - Pushing 1 file change(s) to branch 'dependabot/npm_and_yarn/dependabot/dependabot_grouping_v2/UniLincolnUI/npm-d631ece976'...
🌎 🠊 [POST] https://dev.azure.com/{orgName}/{projectName}/_apis/git/repositories/{repoName}/pushes
🌎 🠈 [500] Internal Server Error
{"$id":"1","innerException":null,"message":"TF401035: The object 'abdceb5590885eeb4810012ade73d8125ae03864' does not exist.","typeName":"Microsoft.TeamFoundation.Git.Server.GitObjectDoesNotExistException, Microsoft.TeamFoundation.Git.Server","typeKey":"GitObjectDoesNotExistException","errorCode":0,"eventId":3000}

@beytun
Copy link

beytun commented Oct 24, 2024

I am facing the exact issue. My pipeline was successful on 14th October. The next execution on 21st October failed with this issue. I see a new version of this task has been released meanwhile.

Successful pipeline

Task : Dependabot
Description : Automatically update dependencies and vulnerabilities in your code using Dependabot CLI
Version : 2.35.955
Author : Tingle Software
Help : https://github.com/tinglesoftware/dependabot-azure-devops/issues

Failed pipeline

Task : Dependabot
Description : Automatically update dependencies and vulnerabilities in your code using Dependabot CLI
Version : 2.36.1003
Author : Tingle Software
Help : https://github.com/tinglesoftware/dependabot-azure-devops/issues

Update:
I just tried pinning the version 2.35.955 in my pipeline and it worked fine. This can be used a workaround till the issue is found out and fixed.

@rhyskoedijk
Copy link
Contributor

rhyskoedijk commented Oct 28, 2024

@grixxie, @DaleMckeown, @beytun are git submodules used in your repositories facing this issue?

I was able to reproduce the same error message if my repo had a submodule and Dependabot had updated one of the files within the submodule. I will continue to investigate, but curious to know if this is the same scenario or just a different way of getting the same error.

@DaleMckeown
Copy link

DaleMckeown commented Oct 28, 2024

There are no git submodules in the project I have been working on that is experiencing this issue.

I still need to try and do a minimal repo to reproduce this issue. I'll try and get it done this week.

@Patrick-3000
Copy link

Same issue here:

We were using Dependabot for MS Azure Devops V1 but as this version got deprecated, we switched to V2.

However, Dependabot is not able to create Pull Requests as it fails with

Creating pull request '[npm-Updates]: Bump lru-cache from 11.0.0 to 11.0.1'...
 - Pushing 1 file change(s) to branch 'dependabot/npm_and_yarn/main/lru-cache-11.0.1'...
🌎 🠊 [POST] [https://***/***/***/_apis/git/repositories/***/pushes?api-version=5.0](https://%2A%2A%2A/***/***/_apis/git/repositories/***/pushes?api-version=5.0)
🌎 🠈 [500] Internal Server Error
{"$id":"1","innerException":null,"message":"TF401035: The object '450425a1a469b12373a3e49b8bd23bfca51a11e3' does not exist.","typeName":"Microsoft.TeamFoundation.Git.Server.GitObjectDoesNotExistException, Microsoft.TeamFoundation.Git.Server","typeKey":"GitObjectDoesNotExistException","errorCode":0,"eventId":3000}
##[error]Failed to create pull request: Error: Request to '[https://***/***/***/_apis/git/repositories/***/pushes?api-version=5.0'](https://%2A%2A%2A/***/***/_apis/git/repositories/***/pushes?api-version=5.0%27) failed: 500 Internal Server Error
Error: Request to '[https://***/***/***/_apis/git/repositories/***/pushes?api-version=5.0'](https://%2A%2A%2A/***/***/_apis/git/repositories/***/pushes?api-version=5.0%27) failed: 500 Internal Server Error
    at AzureDevOpsWebApiClient.restApiRequest (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.37.1025/utils/azure-devops/AzureDevOpsWebApiClient.js:551:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AzureDevOpsWebApiClient.restApiPost (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.37.1025/utils/azure-devops/AzureDevOpsWebApiClient.js:528:16)
    at async AzureDevOpsWebApiClient.createPullRequest (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.37.1025/utils/azure-devops/AzureDevOpsWebApiClient.js:177:26)
    at async DependabotOutputProcessor.process (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.37.1025/utils/dependabot-cli/DependabotOutputProcessor.js:81:42)
    at async DependabotCli.update (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.37.1025/utils/dependabot-cli/DependabotCli.js:116:55)
    at async run (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.37.1025/index.js:82:57)

For me this looks like a bug (especially considering that with V1 we could create PRs), but maybe anybody has a suggestion of what I could try as I am out of ideas...

@bene-tleilax-werdna
Copy link

Just chiming in that we also experience this issue with an npm package ecosystem that contains 2 registries (a public npm registry and a private one).

That said, things appear to work for a separate Python repository. I thought for a while that I was just doing things wrong and messed with some of the parameters but nothing appears to have made any difference.

I'll post my configs below, maybe it can help help someone with more knowledge.

Erroring Example

dependabot.yml

version: 2
updates:
  - package-ecosystem: "npm_and_yarn"
    directory: "/"
    registries:
      - private
      - public
    target-branch: "main"
    commit-message:
      prefix: "chore(deps)"
    versioning-strategy: increase-if-necessary
    allow:
      - dependency-type: "all"
    groups:
      all-dependencies:
        patterns:
          - "*"
        commit-message:
          prefix: "chore(deps)"
registries:
  private:
    type: npm-registry
    url: https://path.to.registry
    token: ${{TOKEN}}
  public:
    type: npm-registry
    url: https://registry.npmjs.org/

pipeline.yaml

jobs:
- job: DependabotJob
  displayName: "Run Dependabot"
  timeoutInMinutes: 240
  pool:
    vmImage: 'ubuntu-latest'
  steps:
  - script: echo "Branch name is $(Build.SourceBranch)"
    displayName: "Log branch name"
  - task: npmAuthenticate@0
    inputs:
        workingFile: .npmrc
  - task: dependabot@2
    displayName: Dependabot V2
    inputs:
      setAutoComplete: false
      mergeStrategy: squash
      autoApprove: false
    retryCountOnTaskFailure: 3
    timeoutInMinutes: 240
    env:
      TOKEN: $(System.AccessToken)

Successful Example

dependabot.yml

version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/"
    target-branch: "main"
    commit-message:
      prefix: "chore(deps)"
    allow:
      - dependency-type: "all"
    groups:
      all-dependencies:
        patterns:
          - "*"
        commit-message:
          prefix: "chore(deps)"

pipeline.yaml

jobs:
- job: DependabotJob
  displayName: "Run Dependabot"
  timeoutInMinutes: 240
  pool:
    vmImage: 'ubuntu-latest'
  steps:
  - script: echo "Branch name is $(Build.SourceBranch)"
    displayName: "Log branch name"
  - task: dependabot@2
    displayName: Dependabot V2
    inputs:
      setAutoComplete: true
      mergeStrategy: squash
      autoApprove: true
    retryCountOnTaskFailure: 3

@rhyskoedijk
Copy link
Contributor

rhyskoedijk commented Nov 10, 2024

I have not been able to reproduce this issue outside of using a git submodule to trick the updater in to updating files that don't exist in the source repo (similar to #504). However, this doesn't seem to be the same problem reported here.

I have submitted a change in #1452 that will add more debug info to failed DevOps API calls; Once released, it will hopefully provide more info in to the cause of this issue.

@Patrick-3000
Copy link

Update: I just tried pinning the version 2.35.955 in my pipeline and it worked fine. This can be used a workaround till the issue is found out and fixed.

@bhiggins14: Could you please explain how you managed to pin the Dependabot version? I do not see an option for this here: https://marketplace.visualstudio.com/items?itemName=tingle-software.dependabot

@beytun
Copy link

beytun commented Nov 12, 2024

@Patrick-3000

@Patrick-3000
Copy link

Patrick-3000 commented Nov 12, 2024

@beytun: thank you very very much, this actually helps us a lot!

I can confirm that with version 2.35.955 PRs can still be created, this means that the bug has to be inserted with a commit into the source code after this version.

@rhyskoedijk
Copy link
Contributor

Could somebody still facing this issue do an update using v1.38.1 with "Enable system diagnostics" selected on the pipeline run? This version should dump more info about the failed push that will hopefully provide some clues as to what is going on.

My assumption is that the error is caused by pushing a file that shouldn't be committed, but there could be more to it than just that.

@DaleMckeown
Copy link

DaleMckeown commented Nov 18, 2024

dependabotlog.txt
@rhyskoedijk Full log file attached.

@rhyskoedijk
Copy link
Contributor

@DaleMckeown thanks, that log is very helpful. It looks like on the 2nd update, the local git repo is not being reset correctly. It is likely that this issue was introduced by #1382, which would match with the comments above suggesting this issue started from v2.35.955.

If you add targetRepositoryName: '$(Build.Repository.Name)' to your task inputs, does the issue still happen?

@DaleMckeown
Copy link

@rhyskoedijk Do I Add that to the Dependabot task? i.e.:

      - task: dependabot@2
        displayName: Run Dependabot
        inputs:
          mergeStrategy: squash
          setAutoComplete: true
          targetRepositoryName: '$(Build.Repository.Name)'

@rhyskoedijk
Copy link
Contributor

Yes

@DaleMckeown
Copy link

DaleMckeown commented Nov 19, 2024

@rhyskoedijk

With that it fails to find my config file:

2024-11-19T09:35:35.2954465Z ##[debug]Evaluating condition for step: 'Run Dependabot'
2024-11-19T09:35:35.2955170Z ##[debug]Evaluating: SucceededNode()
2024-11-19T09:35:35.2955452Z ##[debug]Evaluating SucceededNode:
2024-11-19T09:35:35.2955897Z ##[debug]=> True
2024-11-19T09:35:35.2956221Z ##[debug]Result: True
2024-11-19T09:35:35.2956650Z ##[section]Starting: Run Dependabot
2024-11-19T09:35:35.2963373Z ==============================================================================
2024-11-19T09:35:35.2963537Z Task         : Dependabot
2024-11-19T09:35:35.2963610Z Description  : Automatically update dependencies and vulnerabilities in your code using [Dependabot CLI](https://github.com/dependabot/cli)
2024-11-19T09:35:35.2963761Z Version      : 2.38.1062
2024-11-19T09:35:35.2963853Z Author       : Tingle Software
2024-11-19T09:35:35.2963928Z Help         : https://github.com/tinglesoftware/dependabot-azure-devops/issues
2024-11-19T09:35:35.2964031Z ==============================================================================
2024-11-19T09:35:35.3410041Z ##[debug]Using node path: /home/vsts/agents/3.246.0/externals/node20_1/bin/node
2024-11-19T09:35:35.4565056Z ##[debug]system.debug=True
2024-11-19T09:35:35.4567181Z ##[debug]DistributedTask.Tasks.Node.SkipDebugLogsWhenDebugModeOff=True
2024-11-19T09:35:35.4641617Z ##[debug]agent.TempDirectory=/home/vsts/work/_temp
2024-11-19T09:35:35.4651554Z ##[debug]loading inputs and endpoints
2024-11-19T09:35:35.4656029Z ##[debug]loading INPUT_SKIPPULLREQUESTS
2024-11-19T09:35:35.4675942Z ##[debug]loading INPUT_ABANDONUNWANTEDPULLREQUESTS
2024-11-19T09:35:35.4676980Z ##[debug]loading INPUT_COMMENTPULLREQUESTS
2024-11-19T09:35:35.4683743Z ##[debug]loading INPUT_SETAUTOCOMPLETE
2024-11-19T09:35:35.4685067Z ##[debug]loading INPUT_MERGESTRATEGY
2024-11-19T09:35:35.4685962Z ##[debug]loading INPUT_AUTOAPPROVE
2024-11-19T09:35:35.4686628Z ##[debug]loading INPUT_STOREDEPENDENCYLIST
2024-11-19T09:35:35.4692321Z ##[debug]loading INPUT_TARGETREPOSITORYNAME
2024-11-19T09:35:35.4693074Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2024-11-19T09:35:35.4696097Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2024-11-19T09:35:35.4698918Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2024-11-19T09:35:35.4707849Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
2024-11-19T09:35:35.4709941Z ##[debug]loaded 12
2024-11-19T09:35:35.4717025Z ##[debug]Agent.ProxyUrl=undefined
2024-11-19T09:35:35.4717903Z ##[debug]Agent.CAInfo=undefined
2024-11-19T09:35:35.4718539Z ##[debug]Agent.ClientCert=undefined
2024-11-19T09:35:35.4719240Z ##[debug]Agent.SkipCertValidation=undefined
2024-11-19T09:35:35.8788959Z ##[debug]Checking for `docker` install...
2024-11-19T09:35:35.8789717Z ##[debug]which 'docker'
2024-11-19T09:35:35.8820058Z ##[debug]found: '/usr/bin/docker'
2024-11-19T09:35:35.8820627Z ##[debug]Checking for `go` install...
2024-11-19T09:35:35.8821070Z ##[debug]which 'go'
2024-11-19T09:35:35.8827287Z ##[debug]found: '/usr/bin/go'
2024-11-19T09:35:35.8829360Z ##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/{redacted}/
2024-11-19T09:35:35.8839688Z ##[debug]No virtual directory detected; Running for Azure DevOps Services.
2024-11-19T09:35:35.8840340Z ##[debug]System.TeamProjectId=bc32d8ef-8cfc-42cc-a120-d205f5436004
2024-11-19T09:35:35.8840814Z ##[debug]System.TeamProject=UniLincoln
2024-11-19T09:35:35.8845968Z ##[debug]targetRepositoryName=UniLincoln
2024-11-19T09:35:35.8847436Z ##[debug]Custom repository provided; Running update for remote repository.
2024-11-19T09:35:35.8848906Z ##[debug]gitHubAccessToken=undefined
2024-11-19T09:35:35.8849368Z ##[debug]gitHubConnection=undefined
2024-11-19T09:35:35.8849783Z ##[debug]azureDevOpsUser=undefined
2024-11-19T09:35:35.8850517Z ##[debug]azureDevOpsAccessToken=undefined
2024-11-19T09:35:35.8851090Z ##[debug]azureDevOpsServiceConnection=undefined
2024-11-19T09:35:35.8851947Z ##[debug]No custom token provided. The SystemVssConnection's AccessToken shall be used.
2024-11-19T09:35:35.8854534Z ##[debug]SystemVssConnection auth param AccessToken = ***
2024-11-19T09:35:35.8855864Z ##[debug]authorEmail=undefined
2024-11-19T09:35:35.8856226Z ##[debug]authorName=undefined
2024-11-19T09:35:35.8857199Z ##[debug]setAutoComplete=true
2024-11-19T09:35:35.8857533Z ##[debug]mergeStrategy=squash
2024-11-19T09:35:35.8857895Z ##[debug]autoCompleteIgnoreConfigIds=undefined
2024-11-19T09:35:35.8859073Z ##[debug]storeDependencyList=false
2024-11-19T09:35:35.8860218Z ##[debug]autoApprove=false
2024-11-19T09:35:35.8860847Z ##[debug]autoApproveUserToken=undefined
2024-11-19T09:35:35.8861221Z ##[debug]experiments=undefined
2024-11-19T09:35:35.8861564Z ##[debug]System.Debug=True
2024-11-19T09:35:35.8862299Z ##[debug]targetUpdateIds=undefined
2024-11-19T09:35:35.8862653Z ##[debug]securityAdvisoriesFile=undefined
2024-11-19T09:35:35.8863013Z ##[debug]skipPullRequests=false
2024-11-19T09:35:35.8864348Z ##[debug]commentPullRequests=false
2024-11-19T09:35:35.8865497Z ##[debug]abandonUnwantedPullRequests=false
2024-11-19T09:35:35.8874865Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8875942Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8876836Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8877531Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8878428Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8879972Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8880796Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8881433Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8884046Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8886236Z ##[debug]Processed: ##vso[task.setsecret]***
2024-11-19T09:35:35.8886661Z ##[debug]Attempting to fetch configuration file via REST API ...
2024-11-19T09:35:35.8887154Z ##[debug]GET https://***/***/***/_apis/git/repositories/***/items?path=/.azuredevops/dependabot.yml
2024-11-19T09:35:36.0638630Z ##[debug]No configuration file at 'https://***/***/***/_apis/git/repositories/***/items?path=/.azuredevops/dependabot.yml'
2024-11-19T09:35:36.0639916Z ##[debug]GET https://***/***/***/_apis/git/repositories/***/items?path=/.azuredevops/dependabot.yaml
2024-11-19T09:35:36.2007510Z ##[debug]No configuration file at 'https://***/***/***/_apis/git/repositories/***/items?path=/.azuredevops/dependabot.yaml'
2024-11-19T09:35:36.2010322Z ##[debug]GET https://***/***/***/_apis/git/repositories/***/items?path=/.github/dependabot.yaml
2024-11-19T09:35:36.3672482Z ##[debug]No configuration file at 'https://***/***/***/_apis/git/repositories/***/items?path=/.github/dependabot.yaml'
2024-11-19T09:35:36.3673793Z ##[debug]GET https://***/***/***/_apis/git/repositories/***/items?path=/.github/dependabot.yml
2024-11-19T09:35:36.5106337Z ##[debug]No configuration file at 'https://***/***/***/_apis/git/repositories/***/items?path=/.github/dependabot.yml'
2024-11-19T09:35:36.5109411Z ##[debug]task result: Failed
2024-11-19T09:35:36.5151998Z ##[error]Configuration file not found at possible locations: /.azuredevops/dependabot.yml, /.azuredevops/dependabot.yaml, /.github/dependabot.yaml, /.github/dependabot.yml
2024-11-19T09:35:36.5160218Z ##[debug]Processed: ##vso[task.issue type=error;source=TaskInternal;]Configuration file not found at possible locations: /.azuredevops/dependabot.yml, /.azuredevops/dependabot.yaml, /.github/dependabot.yaml, /.github/dependabot.yml
2024-11-19T09:35:36.5162070Z ##[debug]Processed: ##vso[task.complete result=Failed;]Configuration file not found at possible locations: /.azuredevops/dependabot.yml, /.azuredevops/dependabot.yaml, /.github/dependabot.yaml, /.github/dependabot.yml
2024-11-19T09:35:36.5163326Z ##[error]An unhandled exception occurred: Error: Configuration file not found at possible locations: /.azuredevops/dependabot.yml, /.azuredevops/dependabot.yaml, /.github/dependabot.yaml, /.github/dependabot.yml
2024-11-19T09:35:36.5163906Z ##[debug]Processed: ##vso[task.issue type=error;source=TaskInternal;]An unhandled exception occurred: Error: Configuration file not found at possible locations: /.azuredevops/dependabot.yml, /.azuredevops/dependabot.yaml, /.github/dependabot.yaml, /.github/dependabot.yml
2024-11-19T09:35:36.5165109Z Error: Configuration file not found at possible locations: /.azuredevops/dependabot.yml, /.azuredevops/dependabot.yaml, /.github/dependabot.yaml, /.github/dependabot.yml
2024-11-19T09:35:36.5166141Z     at parseConfigFile (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.38.1062/utils/dependabot/parseConfigFile.js:92:15)
2024-11-19T09:35:36.5166480Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-11-19T09:35:36.5166878Z     at async run (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.38.1062/index.js:30:34)
2024-11-19T09:35:36.5246634Z ##[section]Finishing: Run Dependabot

@rhyskoedijk
Copy link
Contributor

Ah sorry, you'll need the DevOps access token input set too, e.g:

      - task: dependabot@2
        inputs:
          targetRepositoryName: '$(Build.Repository.Name)'
          azureDevOpsAccessToken: '$(System.AccessToken)' # or whatever access token you normally use for running Dependabot as

@DaleMckeown
Copy link

Ah sorry, you'll need the DevOps access token input set too, e.g:

      - task: dependabot@2
        inputs:
          targetRepositoryName: '$(Build.Repository.Name)'
          azureDevOpsAccessToken: '$(System.AccessToken)' # or whatever access token you normally use for running Dependabot as

Apologies, got distracted with other work stuff. I'll give this a go tomorrow.

@bhiggins14
Copy link

hey I'm someone who has been following the thread with the same issue.

Trying to do the fix and I'm having issues passing my pat to the dependabot.yaml config from azure pipeline

Here's my pipeline

 - name: ADO_PAT
   displayName: ADO_PAT
   default: ""
   type: string

pool:
  vmImage: 'ubuntu-latest'

variables:
  System.Debug: false

jobs:
- job: Build
  variables:
     ADO_PAT: ${{ parameters.ADO_PAT }}
  pool:
    vmImage: 'ubuntu-latest'
  steps:
  - task: dependabot@2
    displayName: DependaBot
    env:
      ADO_PAT: ${{ variables.ADO_PAT }}
    inputs:
      targetRepositoryName: '$(Build.Repository.Name)'
      skipPullRequests: false
      azureDevOpsAccessToken: '$(System.AccessToken)' 

Here's my dependabot.yaml

version: 2
registries:
  private:
    type: nuget-feed
    url: https://pkgs.dev.azure.com/***/***/_packaging/***/nuget/v3/index.json
    token: PAT:${{  variables.ADO_PAT  }}
    key: contract-feed
updates:
- package-ecosystem: nuget
  directory: "/"
  registries:
    - private
  schedule:
    interval: daily
  open-pull-requests-limit: 25

Getting 401s when trying to access my feed

updater | +---------------------------------------+
updater | | Errors |
updater | +---------------------------------------+
updater | | private_source_authentication_failure |
updater | +---------------------------------------+

@rhyskoedijk
Copy link
Contributor

@bhiggins14 you probably need to replace token: PAT:${{ variables.ADO_PAT }} with token: PAT:${{ADO_PAT}}; You can only reference environment variables. See the purple tips in configuring private feeds and registries.

@DaleMckeown
Copy link

Ah sorry, you'll need the DevOps access token input set too, e.g:

      - task: dependabot@2
        inputs:
          targetRepositoryName: '$(Build.Repository.Name)'
          azureDevOpsAccessToken: '$(System.AccessToken)' # or whatever access token you normally use for running Dependabot as

Sorry, I got distracted with other stuff. I'll give this a go tomorrow.

@DaleMckeown
Copy link

@rhyskoedijk

I've tried this a few times, I can't get it working. It keeps telling me that I don't have an access token, despite the debug output above stating that I do:

##[debug]azureDevOpsAccessToken provided, using for authenticating
##[debug]authorEmail=undefined
##[debug]authorName=undefined
##[debug]setAutoComplete=true
##[debug]mergeStrategy=squash
##[debug]autoCompleteIgnoreConfigIds=undefined
##[debug]storeDependencyList=false
##[debug]autoApprove=false
##[debug]autoApproveUserToken=undefined
##[debug]experiments=undefined
##[debug]System.Debug=True
##[debug]targetUpdateIds=undefined
##[debug]securityAdvisoriesFile=undefined
##[debug]skipPullRequests=false
##[debug]commentPullRequests=false
##[debug]abandonUnwantedPullRequests=false
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Processed: ##vso[task.setsecret]***
##[debug]Attempting to fetch configuration file via REST API ...
##[debug]GET [https://***/***/***/_apis/git/repositories/***/items?path=/.azuredevops/dependabot.yml](https://%2A%2A%2A/***/***/_apis/git/repositories/***/items?path=/.azuredevops/dependabot.yml)
##[debug]task result: Failed
##[error]No access token has been provided to access '[https://***/***/***/_apis/git/repositories/***/items?path=/.azuredevops/dependabot.yml'](https://%2A%2A%2A/***/***/_apis/git/repositories/***/items?path=/.azuredevops/dependabot.yml%27)

@rhyskoedijk
Copy link
Contributor

@DaleMckeown no worries, I did some very rough testing the other day and I am fairly confident that reverting #1382 would fix the issue. I hope to have a PR for it this weekend.

@bhiggins14
Copy link

hey @rhyskoedijk the settings you gave me worked worked and I was able to generate PRs successfully with

Task         : Dependabot
Description  : Automatically update dependencies and vulnerabilities in your code using [Dependabot CLI](https://github.com/dependabot/cli)
Version      : 2.38.1062

@rhyskoedijk
Copy link
Contributor

This should be fixed by #1478.

@rhyskoedijk
Copy link
Contributor

Could everyone impacted by this please test again using 1.38.2? I know at least one instance of this error is fixed, but if there are still scenarios which aren't fixed, it would be good to get a copy of your logs so I can investigate further. Thanks.

@DaleMckeown
Copy link

@rhyskoedijk FYI It fixed my case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
7 participants