Consider previous deployments as successful

[VictorChacon-Ada]

Hi there. We currently have a workflow running on github actions that deploys to a staging environment (actually only pushes an image, and argocd does the rest), and then awaits manual approval to deploy to production.

The thing is, if multiple people are making changes to the same repo, and testing on our staging environment, sometimes a later deployment process is approved, and takes the previous changes with it. Then, we never actually approve the previous deployment, and it ends up failing after the 30 days timeout.

This ends up causing 2 things:

• We end up with a higher than the actual failure rate for deployments
• We get a wrong lead time for changes. For instance, in specific periods with high amount of these types of deployments, we get a lead time for changes of 720 hours (since a lot of deployments are considered with the 30 days timeout)

Below, I have 2 screeshots, 1 for a pipeline that is currently waiting for deployment, but has been superseded, and another one where the same thing happened, but timed out after 30 days.

Is there any way I can treat these specific cases on devlake? I tried reading the code for the lead time calculation in the github connector, but couldn't think of any way to reconcile the data. If not, any suggestions on how I can calculate these Dora Metrics accurately?

[klesh]

@Startrekzky Need your input. Thanks

[Startrekzky]

Hi @VictorChacon-Ada , thank you for the feedback.

We end up with a higher than the actual failure rate for deployments.

What do you mean by failure rate for deployments? Is it the Change Failure Rate in DORA or a metric that you measured on your own?

We get a wrong lead time for changes. For instance, in specific periods with high amount of these types of deployments, we get a lead time for changes of 720 hours (since a lot of deployments are considered with the 30 days timeout)

If the previous deployment is d-1, the later deployment is d-2.

As far as I know,

• First, the change lead time is calculated based on the association between PR and the SUCCESS deployment which is d-2, not d-1. Thus, d-2's deployment shouldn't be taken into calculation.
• Second, the duration of deployments will not affect the change lead time according to the algorithm. See doc https://devlake.apache.org/docs/Metrics/LeadTimeForChanges#how-is-it-calculated

Hence, I'm confused why your change lead time has been affected.

cc. @klesh

[VictorChacon-Ada]

Hello guys, thanks for the help!

The metric I was talking about was the DORA Change Failure Rate, but I had gotten to the conclusion that these deployments were considered failures by querying the data on SQL. On checking my grafana dashboards, that is not true (I was probably missing a step when querying manually), so we can ignore that first point.

For the second point, it is true that the d-2 deployment does not affect the d-1 lead time. But exactly for that reason, we get the situation where one of the deployments has a short Lead Time, and another has a very long Lead Time. If I get enough of these "superseded" deployments in a project in a month, I get a median Lead Time of 721 hours, a little bit over the 30 days when the deployment times out.

I have an example of a timed out deployment that affects the lead time. If I run the query below on the SQL backend, I get that result.

SELECT 
	pr.id as pr_id,
	pr.status as pr_status,
	pr.merge_commit_sha as pr_merge_commit_sha,
	pr.base_commit_sha  as pr_base_commit_sha,
	pr.head_commit_sha as pr_head_commit_sha,
	ppm.first_commit_sha as ppm_first_commit_sha,
	prc.commit_sha as prc_commit_sha,
	ppm.pr_coding_time as coding_time,
	ppm.deployment_commit_id as ppm_deployment_commit_id,
	ppm.pr_deploy_time as deploy_time,
	ppm.pr_cycle_time as cycle_time,
	cdc.`result` as cdc_result,
	cdc.status as cdc_status,
	pr.merged_date as pr_merged_date,
	cdc.finished_date as cdc_finished_date,
	cdc.duration_sec as cdc_duration_sec
FROM 
pull_requests pr
INNER JOIN project_pr_metrics ppm ON ppm.id = pr.id
INNER JOIN pull_request_commits prc ON prc.pull_request_id = pr.id
INNER JOIN cicd_deployment_commits cdc ON cdc.commit_sha = pr.merge_commit_sha 
WHERE pr.id = "github:GithubPullRequest:1:1475009734"

pr_id	pr_status	pr_merge_commit_sha	pr_base_commit_sha	pr_head_commit_sha	ppm_first_commit_sha	prc_commit_sha	coding_time	ppm_deployment_commit_id	deploy_time	cycle_time	cdc_result	cdc_status	pr_merged_date	cdc_finished_date	cdc_duration_sec
github:GithubPullRequest:1:1475009734	MERGED	e40f2c2d9cf4aea56784353f6b890e9d0188a2c9	e64fbb5170e83e9bf3e7f5107f3960839b48c02f	7d5462cbf6009579a46c46e207fe5a5d99fc6363	7d5462cbf6009579a46c46e207fe5a5d99fc6363	7d5462cbf6009579a46c46e207fe5a5d99fc6363	1	github:GithubRun:1:673988831:5860050089:https://github.com/Ada-Tech-Br/dev_ada_async	43209	43216	FAILURE	DONE	2023-08-14 16:56:32	2023-09-13 17:04:54	2592499

As you can see, the fact that the cdc duration is so long, ends up on the pr_cycle_time. From the dora metric plugin code, I can see that the deployment is being used to calculate the Lead Time.

Maybe I have some inconsistencies in my data that are causing stuff like that to not be filtered, but as it stands, the median lead time for changes metric is being polluted by the 721 hours deployments, and in the worst cases, completely overtaken, making the median 721 hours.

I appreciate any help I can get in understanding this problem.

[Startrekzky]

Hi @klesh ,

1. Will a 'FAILURE' deployment be written into table.cicd_deployment_commits ?
2. If so, will it be calculated in CFR?

It seems there's a bug here.

@VictorChacon-Ada BTW, What DevLake version are you on?

[VictorChacon-Ada]

I'm running the latest version, 0.18.0.

[klesh]

@Startrekzky

1. Yes
2. No

@VictorChacon-Ada
This is weird, Could you check the cicd_deployment_commits table and see if the prev_success_deployment_commit_id are associated correctly? meaning that they should not be pointing to the failed one.

[VictorChacon-Ada]

I just checked, and for that specific run, the prev_success_deployment_commit_id is empty, is that expected?

[klesh]

@VictorChacon-Ada Interesting, I might know what is wrong. Seems like the commits_diffs of the failed deployment commit get computed and because of

incubator-devlake/backend/plugins/dora/tasks/change_lead_time_calculator.go

Line 209 in 39e9751

dal.Where("pm.project_name = ? AND cd.commit_sha = ?", projectName, mergeSha),

failed to filter the deployment commits by status leads to the problem.

[klesh]

So, deployment commit with id github:GithubRun:1:673988831:5860050089:https://github.com/Ada-Tech-Br/dev_ada_async was failed (timeouted)?

[VictorChacon-Ada]

Yes, it is considered a failed deployment, but practically, it was just superseded by a later deployment that took the changes together. That is the main problem. In the best possible cenario, I'd like to be able to consider the deployment for github:GithubPullRequest:1:1475009734 as completed when a child commit is succesfully deployed, but as a simpler and more immediate solution, I think failed deployments shouldn't count for Lead Time, since, at least in theory, they never went to production.

[klesh]

@VictorChacon-Ada Totally, agreed.
However, it shouldn't be happening by the current logic. We need to dig into the data and see which step went wrong, would you like to do so?

[klesh]

It might be caused by the status not being processed properly, similar to #6233

[klesh]

Hi, @VictorChacon-Ada I found another possible cause and it should be fixed by #6270
Will release a new beta , please try it out and see if it fixes your problem.

[abeizn]

@VictorChacon-Ada We have released two versions(https://github.com/apache/incubator-devlake/releases/tag/v0.18.1-beta2 or https://github.com/apache/incubator-devlake/releases/tag/v0.19.0-beta5), you can upgrade to one of them to see if the problem is solved.

[VictorChacon-Ada]

That seems to have fixed it. Thank you all so much for the help!