Gitextractor fails - Plain clone git error

[markdand]

Good morning!

I wanted to find out if anyone had this issue and has resolved it before opening a bug. I can't find anything related to this specific error.
Running Devlake using Docker / Docker-Compose, during the gitextractor step of the pipeline, fails with the error:
subtask cloneGitRepo ended unexpectedly Wraps: (2) plain clone git error Wraps: (3) plain clone git error Error types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *errors.errorString

The rest of the git data is collected correctly, so we can view deployments, PR's, etc, however, data related to gitextractor like file-level metrics, is missing.
We are using a self-hosted Gitlab instance, on version 16.9.
Our Devlake version is v0.21.0@ac4f7fe.

[durdekm]

Hi, we have the same Problem with the sam setup.

Thanks for troubleshooting and solution tips...

[markdand]

Hello!

Yes, I managed to resolve this for our setup. The reason for the error was that we don't allow clone over HTTP/HTTPS for git, only SSH. The default configuration for gitextractor tries to clone with HTTP. I discovered that it does allow for SSH cloning, but you have to use advanced blueprint mode. These were the steps that I followed:

1. Upgraded to V1.0.0-beta8 (not sure if this is required, but it was the latest version at the time I resolved it)

2. Changed the blueprint mode from 'NORMAL' to 'ADVANCED' in the devlake database.
   UPDATE _devlake_blueprints SET mode = 'ADVANCED' where name = '<blueprint_name>';

3. In the Blueprint configuration on the Devlake UI, I edited the gitextractor plugin section to have the following:
   { "plugin": "gitextractor", "subtasks": null, "options": { "fullName": "Name", "name": "App", "privateKey": "/path/to/private_key", "proxy": "", "repoId": "gitlab:GitlabProject:1:<id>", "url": "ssh://git@gitURL/group/repo.git" }

Most of the items will be entered for you already, like name, repoid, etc. You will have to add the privateKey section, and edit the URL from https to ssh.

Hope that helps!

[durdekm]

Hello. Thank you for your information. I'll check it...

[jphelpsqred]

Having a similar problem as we also only allow ssh cloning. Since I run devlake as a container and wouldn't be able to store a pem file on a persistent filesystem.
I see that you're referencing privateKey as a file path, would that be the absolute path to the pem file on the server running Devlake?

[markdand]

Hi,

Yes that's correct. I created a key-pair with read only permissions, then added it to the Devlake container by building a custom image. I simply used the devlake image as the base and copied the file into the image. The path to the private key is where I copied it to in the image (e.g. /data/id_rsa)

Hope that helps!

[klesh]

I think a better solution is to:

1. store ssh_url_to_repo alongside http_url_to_repo in Gitlab Project table.
2. add an option to the GitLab connection table to allow users to specify which URL to use for git cloning.
3. update api/blueprint_v200.go to generate the gitextractor task based on the above information.