Mirrorstats report wrong results

[NotMyFault]

Service(s)

get.jenkins.io

Summary

Hey,

while I was checking something and opened the mirrorstats page for 2.361.1, I noticed that 1/4 of the mirrors report that they are either offline or updated a long time ago: https://get.jenkins.io/war/2.361.1/jenkins.war?mirrorstats

Although, the results are not correct for those mirrors. For example, mirrorstats reports serverion was updated 287 days ago, when its timestamp reports the last update happened a few hours ago, according to https://mirror.serverion.com/jenkins/TIME.

Reproduction steps

No response

[MarkEWaite]

The mirror at serverion had some reliability issues that caused the infrastructure team to remove it from the mirror list.

https://get.jenkins.io/war/2.368/jenkins.war?mirrorlist shows that it is not in the current list of contributing mirrors.

[NotMyFault]

Oh I see, thanks for the clarification!

[lemeurherve]

I'll check if we have datadog probes configured on them. EDIT: we don't have any.

[dduportal]

Status: we (jenkins-infra team) need to check the status of the 4 following mirrors:

• esuni.jp
• serverion.com
• aliyun.com
• ftp.belnet.be
   
  and need to check why archives.jenkins.io is not the last proposal ?

[dduportal]

Update as per https://groups.google.com/g/jenkins-infra/c/C7cW3MKwR0I/m/lVNM4ymTBAAJ, it looks like belnet could be re-enabled and synced again.

Adding the issue to our current milestone so we can work on it

[dduportal]

I'm stuck trying to enable the ftp.belnet.be mirror and it seems an issue is needed on the mirrorbits project.

What I tried:

• Scan + Enable the mirror. Works (no error, exit code 0):

  $ mirrorbits scan -enable ftp.belnet.be
  Scanning ftp.belnet.be... done
    ∟ Enabled

• Corresponding log, with no error message

  2023/09/20 17:10:17.615 UTC [ftp.belnet.be] Requesting file list via rsync...
  2023/09/20 17:10:33.584 UTC [ftp.belnet.be] Indexed 37910 files (37908 known), 0 removed
  

But the mirror stays in state "down" with no reason.

I've tracked down the function MarkMirrorDown in mirrorbits's code (https://github.com/etix/mirrorbits/blob/e83e56ac6496a3643d18a731324bd266f75f7b32/mirrors/mirrors.go#L180-L183) which is called during the mirrors healthchecks:

• If an error is returned by the HTTP client https://github.com/etix/mirrorbits/blob/9189dc7c20b389b5921de4fc85273cdbe7b52356/daemon/monitor.go#L528-L539
• If the mirrors answers a 404: https://github.com/etix/mirrorbits/blob/9189dc7c20b389b5921de4fc85273cdbe7b52356/daemon/monitor.go#L554
• If the status code is unknwown: https://github.com/etix/mirrorbits/blob/9189dc7c20b389b5921de4fc85273cdbe7b52356/daemon/monitor.go#L566

But these 4 occurences provides a reason which should be logged as error. So maybe the log.Errorf does not write to the container stdout (I'm checking the default log files) or there is an unexpected code path case.

[dduportal]

https://github.com/jenkins-infra/docker-mirrorbits/blob/daf0a7c7b3eebbe9f97561aa4285f01074e94cce/config/mirrorbits.conf#L19 => logsDir directive is disabled, which means we might not be using "download" logs with th expected content (https://github.com/etix/mirrorbits/blob/9189dc7c20b389b5921de4fc85273cdbe7b52356/logs/logs.go#L137-L139)

[dduportal]

Update:

• The ftp.belnet.be mirror is now seen online (see https://get.jenkins.io/war/2.423/jenkins.war?mirrorlist).
  (I believe the explanation at scan mirror before enabling after down/unreachable etix/mirrorbits#20 (comment) from mirrorbit's author explains why it worked after a few hours.)
• serverion.com is still disabled (gotta check with them but I believe we should not continue using it given the amount of issues in the past wdyt @MarkEWaite ) => to be discussed during next team meeting
• esuni.jp is disabled because it only provides an HTTP URL. As such, mirrorbits cannot scan it. I've asked our contact in https://groups.google.com/g/jenkins-infra/c/ZVwySY7rjoA/m/EZSy4noRCQAJ let's way for an answer. If no answer or if a refusal, then we'll have to remove this mirror.
• for aliyun: Add Alibaba mirror #3759

[MarkEWaite]

• serverion.com is still disabled (gotta check with them but I believe we should not continue using it given the amount of issues in the past wdyt @MarkEWaite ) => to be discussed during next team meeting

I agree that we should not continue using serverion.com due to the number of issues we've encountered in the past.

[dduportal]

Since this is easily revertable, and as we received no answers, I've removed the esuni.jp and aliyun (Alibaba) mirrors.
We can add them back if an rsync or ftp server is provided in the future.

I agree that we should not continue using serverion.com due to the number of issues we've encountered in the past.

Same: I've removed serverion.com, unless they contact us back (in that case we'll re-evaluate and add back the mirror , its is one command away).

[dduportal]

For info, mirrors are bein rescanned fully, which is having the following impacts:

• OSUOSL are used as fallback until all scan are performed
• A lot of Azure file account operation are being done

[dduportal]

Update (before being able to close this issue):

• Cleanup removed mirrors from datadog monitoring
• Updated runbook (private link: https://github.com/jenkins-infra/runbooks/commit/ffec7f6c6712cf9195f97218cd03ce8afe999e73)
• Cleaned up duplicated (and unused) doc in helm charts of mirrorbits: docs(mirrorbits,mirrorbits-lite) move Jenkins production custom documentation to only runbooks helm-charts#779