chore: Temporarily disable the license report generation to speed up build #704

Roiocam · 2023-10-11T11:29:35Z

Motivation

Refs: original from #326 and #332, then #1019

This PR improves the speed of paradox build by disabling sbt-license-report on specific build.

PTAL, @mdedetrich @He-Pin

Roiocam

Approach that skip license-report

Roiocam · 2023-10-11T11:39:00Z

.github/workflows/link-validator.yml

@@ -35,7 +35,7 @@ jobs:
        uses: coursier/[email protected]

      - name: Create the Pekko site
-        run: sbt -Dpekko.genjavadoc.enabled=true "Javaunidoc/doc; Compile/unidoc; docs/paradox"
+        run: sbt -Dpekko.genjavadoc.enabled=true -Dpekko.genlicensereport.enabled=false "Javaunidoc/doc; Compile/unidoc; docs/paradox"


skip the license-report page generation in link-validator step.

As this is a ci, can we keep it? or we add a dedicated task for genlicensereport

project/ProjectIndexGenerator.scala

scripts/link-validator.conf

Roiocam · 2023-10-12T11:01:32Z

Looks like last CI has successfully passed. And since the PR was opened, every Link Validator job in each commit has been successfully passed. It seems that this approach can address the issues caused by the license report.

One more commit for documentation clarify.

@pjfanning could you trigger the CI again? Thanks.

mdedetrich · 2023-10-13T15:53:33Z

I had another way of solving this issue which involved adding an ignore-files config to my forked https://github.com/mdedetrich/site-link-validator (the fork is also there so we don't rely on akka). Such a configuration would provide a far simpler solution as you would just do something like

ignore-files = [
  "link-validator.html"
]

in scripts/link-validator.conf

Which makes this really annoying because a lot of time was spent on this :/

Roiocam · 2023-10-13T16:28:03Z

I had another way of solving this issue which involved adding an ignore-files config to my forked https://github.com/mdedetrich/site-link-validator (the fork is also there so we don't rely on akka).

I have to say that this approach also crossed my mind initially. However, considering the additional maintenance cost of another repo, I have decided to abandon this solution for this issue and consider a simpler approach.

This PR approach more than ignores license report files and could skip the task of license report generation, it can make PR checks run faster.

In addition, maintaining a new repo could provide ongoing solutions for future issues that may arise. However, the biggest obstacle for me is the sustainability of maintaining an additional repo. Throughout my career, I have encountered situations where repositories were no longer updated due to a lack of maintenance personnel, such as Apache Common DButils. (It's just my stubborn mindset.)

Anyway, I am willing to accept whichever solution is ultimately accepted because both of them address the problem.

Best regards.

mdedetrich · 2023-10-13T18:57:40Z

So my biggest apprehension is the complexity of the build which this PR does increase, I have an overarching goal in my head to reduce to try and gradually simplify the sbt build (I haven't really formalized this yet) as I see that sbt's build complexity is creating a barrier of entry to those trying to contribute.

The reduction of the build time I don't see as a critical priority in this specific case, also do note that there will be movement in the future to try and procure our own hardware.

Roiocam · 2023-10-14T07:01:23Z

@mdedetrich Another viewpoint argues that those repo(link-validator) is not significant in scale and does not require regular updates. Therefore, the lack of maintenance is not a significant issue to consider.

Scala is not my primary language, but I still can make the commit like this PR. I acknowledge that the sbt build can be a bit complex, but also offers a lot of possibilities. Considering that sbt builds are not something that needs frequent changes, I believe it won't hinder those who are trying to contribute.

Perhaps we can merge this PR first to avoid those annoying check failure. Once the PR for the other repo is merged and released, we can remove the redundant build task from this PR (This PR also includes a fix part for the broken links). Unless the repo of https://github.com/mdedetrich/site-link-validator was proceeding well.

Roiocam · 2023-10-25T06:08:20Z

@mdedetrich any feedback of this issue will be great.

He-Pin · 2024-01-22T06:20:05Z

project/ProjectIndexGenerator.scala

+      "links.md")
+
+    val markdownFiles = if (CliOptions.generateLicenseReportEnabled.get) {
+      markdownFilesBeforeLicense ++ Seq("license-report.md") ++ markdownFilesAfterLicense


I think we can add a paradoxFast and paradoxFastBrowse task which turn this off?

Have tried those approach:

(Compile / paradox / javaOptions += "-Dpekko.genlicensereport.enabled=false")

(javaOptions += "-Dpekko.genlicensereport.enabled=false")

Compile / paradoxMarkdownToHtml / sourceGenerators := { (Compile / paradoxMarkdownToHtml / sourceGenerators).value.filterNot(_ == Paradox.licenseReportGeneratorTask.taskValue) }

Compile / paradoxMarkdownToHtml / sourceGenerators := { Seq.empty }

None of them can be filtered in the license report generated, will keep find a new approach tomorrow.

docs/src/main/paradox/project/project-index.md

mdedetrich · 2024-01-22T07:05:59Z

So I am not sure how this PR relates to #1019 specifically but as explained here the majority of time is taken up in sbt-license-report's updateLicenses task.

I would divert effort into solving that fundamental issue

pjfanning · 2024-01-22T17:32:21Z

@Roiocam could I ask you to get an iCLA? https://www.apache.org/licenses/contributor-agreements.html

It makes the team's life easier if regular contributors have iCLAs

Roiocam · 2024-01-22T17:47:53Z

@Roiocam could I ask you to get an iCLA? https://www.apache.org/licenses/contributor-agreements.html

It makes the team's life easier if regular contributors have iCLAs

I was sent the ICLA submission on 2023/9/23, is any thing I have missed?

pjfanning · 2024-01-22T17:51:23Z

@Roiocam thanks - I found your iCLA.

mdedetrich · 2024-01-22T21:09:08Z

So I am fine with this being a temporary fix but when sbt/sbt-license-report#87 is resolved I would suggest to revert this commit because there isn't any reason to have this feature disable i.e. its currently a workaround for that bug.

This is because when sbt/sbt-license-report#87 is resolved, the updateLicenses task should be near instant because it will actually re-use all of the dependency resolution that was already done when sbt is loaded.

Roiocam · 2024-01-23T02:07:01Z

So I am fine with this being a temporary fix but when sbt/sbt-license-report#87 is resolved I would suggest to revert this commit because there isn't any reason to have this feature disable i.e. its currently a workaround for that bug.

Yes, that's why we still need another ISSUE #1019 to associate the problem.

This is because when sbt/sbt-license-report#87 is resolved, the updateLicenses task should be near instant because it will actually re-use all of the dependency resolution that was already done when sbt is loaded.

No doubt it, this PR is just sugar for contributors now.

mdedetrich

Okay I am going to approve it, would be good if @pjfanning and @He-Pin also looked at it. @Roiocam Can you make an standard issue to revert this PR once sbt/sbt-license-report#87 is solved so we don't forget.

Thanks for the work!

Roiocam · 2024-01-23T02:26:54Z

Okay I am going to approve it, would be good if @pjfanning and @He-Pin also looked at it. @Roiocam Can you make an standard issue to revert this PR once sbt/sbt-license-report#87 is solved so we don't forget.

Yes, I will stay updated on the progress.

He-Pin

Thanks

He-Pin · 2024-01-23T07:49:58Z

@pjfanning Would you like to take a look at this, I think with this , all our contributors can have a faster local development exprirence.

Roiocam commented Oct 11, 2023

View reviewed changes

Roiocam marked this pull request as draft October 12, 2023 08:32

Roiocam force-pushed the flaky-link-validator branch from a93ce8a to 687d3f6 Compare October 12, 2023 09:37

Roiocam marked this pull request as ready for review October 12, 2023 09:38

Roiocam force-pushed the flaky-link-validator branch 3 times, most recently from 4681235 to 0c964fe Compare December 21, 2023 11:14

Roiocam mentioned this pull request Jan 22, 2024

Long term: sbt-license-report took so much time to compile (upstream problem) #1019

Open

2 tasks

Roiocam force-pushed the flaky-link-validator branch from 0c964fe to 35f2688 Compare January 22, 2024 06:03

Roiocam changed the title ~~fix: flaky link validator test~~ feat: improve the speed of paradox build Jan 22, 2024

He-Pin reviewed Jan 22, 2024

View reviewed changes

docs/src/main/paradox/project/project-index.md Show resolved Hide resolved

Roiocam and others added 11 commits January 23, 2024 01:08

fix: using project/index generator

3e21d98

fix: broken link

d2f7aad

chore: reformat code

b8fff55

simplify generator and build fixes

0851d93

additional documentation clarify

ebade56

code format chore

c4bdbab

chore: reducing cr cost

af2b404

chore: remove unnecessary

b95af29

reset link configuration

40a317f

keep index doc

d9fda4e

disable license report by default

dd78d2f

reduce license because disable by default

988c1c2

Roiocam force-pushed the flaky-link-validator branch from 989218f to 988c1c2 Compare January 22, 2024 17:08

Change the description of build paradox

a2e50f5

chore: code fmt

c8499d4

Roiocam changed the title ~~feat: improve the speed of paradox build~~ chore: Temporarily disable the license report generation to speed up build Jan 23, 2024

mdedetrich approved these changes Jan 23, 2024

View reviewed changes

He-Pin approved these changes Jan 23, 2024

View reviewed changes

He-Pin requested a review from pjfanning January 23, 2024 11:11

He-Pin merged commit 108f5d1 into apache:main Jan 23, 2024
18 checks passed

Roiocam mentioned this pull request Jan 26, 2024

Build script/plugin issue tracker #1052

Open

32 tasks

Roiocam mentioned this pull request Jul 2, 2024

Simplify link validator config #1381

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Temporarily disable the license report generation to speed up build #704

chore: Temporarily disable the license report generation to speed up build #704

Roiocam commented Oct 11, 2023 •

edited

Loading

Roiocam left a comment

Roiocam Oct 11, 2023

He-Pin Jan 22, 2024

Roiocam commented Oct 12, 2023 •

edited

Loading

mdedetrich commented Oct 13, 2023 •

edited

Loading

Roiocam commented Oct 13, 2023 •

edited

Loading

mdedetrich commented Oct 13, 2023

Roiocam commented Oct 14, 2023 •

edited

Loading

Roiocam commented Oct 25, 2023

He-Pin Jan 22, 2024

Roiocam Jan 22, 2024

mdedetrich commented Jan 22, 2024

pjfanning commented Jan 22, 2024

Roiocam commented Jan 22, 2024

pjfanning commented Jan 22, 2024

mdedetrich commented Jan 22, 2024

Roiocam commented Jan 23, 2024

mdedetrich left a comment

Roiocam commented Jan 23, 2024

He-Pin left a comment

He-Pin commented Jan 23, 2024

chore: Temporarily disable the license report generation to speed up build #704

chore: Temporarily disable the license report generation to speed up build #704

Conversation

Roiocam commented Oct 11, 2023 • edited Loading

Motivation

Roiocam left a comment

Choose a reason for hiding this comment

Roiocam Oct 11, 2023

Choose a reason for hiding this comment

He-Pin Jan 22, 2024

Choose a reason for hiding this comment

Roiocam commented Oct 12, 2023 • edited Loading

mdedetrich commented Oct 13, 2023 • edited Loading

Roiocam commented Oct 13, 2023 • edited Loading

mdedetrich commented Oct 13, 2023

Roiocam commented Oct 14, 2023 • edited Loading

Roiocam commented Oct 25, 2023

He-Pin Jan 22, 2024

Choose a reason for hiding this comment

Roiocam Jan 22, 2024

Choose a reason for hiding this comment

mdedetrich commented Jan 22, 2024

pjfanning commented Jan 22, 2024

Roiocam commented Jan 22, 2024

pjfanning commented Jan 22, 2024

mdedetrich commented Jan 22, 2024

Roiocam commented Jan 23, 2024

mdedetrich left a comment

Choose a reason for hiding this comment

Roiocam commented Jan 23, 2024

He-Pin left a comment

Choose a reason for hiding this comment

He-Pin commented Jan 23, 2024

Roiocam commented Oct 11, 2023 •

edited

Loading

Roiocam commented Oct 12, 2023 •

edited

Loading

mdedetrich commented Oct 13, 2023 •

edited

Loading

Roiocam commented Oct 13, 2023 •

edited

Loading

Roiocam commented Oct 14, 2023 •

edited

Loading