Add BOM / dependency management support #3924

alexarchambault · 2024-11-08T16:05:31Z

This PR adds support for user-specified BOMs and dependency management in Mill.

BOM support allows users to pass the coordinates of an existing Maven "Bill of Material" (BOM), such as this one, that contains versions of dependencies, meant to override those pulled during dependency resolution. (They can also add exclusions to dependencies.)

def bomDeps = Agg(
  ivy"io.quarkus:quarkus-bom:3.17.0"
)

It also allows users to specify the coordinates of a parent POM, which are taken into account just like a BOM:

def parentDep = ivy"org.apache.spark::spark-parent:3.5.3"

(in line with PublishModule#pomParentProject that's been added recently)

It allows users to specify "dependency management", which act like the dependencies listed in a BOM: versions in dependency management override those pulled transitively during dependency resolution, and exclusions in its dependencies are added to the same dependencies during dependency resolution.

def dependencyManagement = Agg(
  ivy"com.google.protobuf:protobuf-java:4.28.3",
  ivy"org.java-websocket:Java-WebSocket:_" // placeholder version - this one only adds exclusions, no version override
        .exclude(("org.slf4j", "slf4j-api"))
)

BOM and dependency management also allow for "placeholder" versions: users can use _ as version in their ivyDeps, and the version of that dependency will be picked either in dependency management or in BOMs:

def bomDeps = Agg(
  ivy"com.google.cloud:libraries-bom:26.50.0"
)
def ivyDeps = Agg(
  ivy"com.google.protobuf:protobuf-java:_"
)

A tricky aspect of that PR is that details about BOMs and dependency management have to be passed around via several paths:

in the current module: BOMs and dependency management have to be taken into account during dependency resolution of the module they're added to
via moduleDeps: BOMs and dependency management of module dependencies have to be applied to the dependencies of the module they come from
to transitive modules pulled via moduleDeps: BOMs and dependency management of a module dependency have to be applied to the dependencies of modules they pull transitively (if A depends on B and B depends on C, from A, the BOMs and dep mgmt of B apply to C's dependencies too) (worked out-of-the-box with the previous point, via transitiveIvyDeps)
via ivy.xml: when publishing to Ivy repositories (like during pubishLocal), BOMs and dep mgmt details need to be written in the ivy.xml file, so that they're taken into account when resolving that module from the Ivy repo
via POM files: when publishing to Maven repositories, BOMs and dep mgmt details need to be written to POMs, so that they're taken into account when resolving that module from the Maven repo

Fixes #1975

himanshumahajan138 · 2024-11-09T07:16:19Z

Sir @alexarchambault, will this add bom support for jetpack compose libs also?

alexarchambault · 2024-11-09T14:42:38Z

If it only relies on a "BOM", that should yes. But this PR isn't about the Gradle module metadata stuff, that was mentioned too in one of your PRs.

himanshumahajan138 · 2024-11-09T14:48:58Z

Actually i think this will work(not sure) lets see after it gets merged then i will try this once

lefou

Don't we need to take transitive bomDeps into account? In it's current draft, bomDeps is only used intransitively.

As a rule of thumb, transitive moduleDeps should behave the same as transitive ivyDeps. If an upstream dependency import a BOM and that BOM is considered by coursier when resolving dependencies, it should also work, if that upstream dependency is a local module.

alexarchambault · 2024-11-11T23:09:37Z

@lefou Yes, that's what I meant by "make things working for multi-module projects". In the ITs, I intend to publish locally things in a directory, and check that resolution by coursier of stuff published locally gives the same class path as Mill computes internally.

Conflicts: build.mill

Conflicts: scalalib/src/mill/scalalib/PublishModule.scala scalalib/src/mill/scalalib/publish/Pom.scala

Conflicts: build.mill main/util/src/mill/util/CoursierSupport.scala scalalib/src/mill/scalalib/CoursierModule.scala scalalib/src/mill/scalalib/JavaModule.scala scalalib/src/mill/scalalib/Lib.scala

alexarchambault · 2024-11-22T10:36:13Z

@lihaoyi @lefou This is almost ready for review, you can already have a look if you wish.

There a two things I'd like to tweak though: some examples should be added in the doc ~~, and there's an aspect of the description above that I'd like to check ("transitive modules pulled via moduleDeps")~~

This PR adds many features. I have a branch with these changes with a cleaner git history here. If needed, to ease review, I can drop some features (and send subsequent PRs later…), so that these can be reviewed one-by-one.

lefou

Not a full review, but some remarks.

lefou · 2024-11-23T09:47:58Z

main/util/src/mill/util/CoursierSupport.scala

@@ -31,6 +31,20 @@ trait CoursierSupport {
        ctx.fold(cache)(c => cache.withLogger(new TickerResolutionLogger(c)))
      }

+  def isLocalTestDep(dep: Dependency): Option[Seq[PathRef]] = {


Can we make this private or private[mill]?

Yes, private actually (with minor refactoring so that it's not used elsewhere)

lefou · 2024-11-23T15:26:34Z

scalalib/src/mill/scalalib/JavaModule.scala

+  def allIvyDeps: T[Agg[Dep]] = Task {
+    val bomDeps0 = allBomDeps().toSeq
+    val rawDeps = ivyDeps() ++ mandatoryIvyDeps()
+    val depsWithBoms =
+      if (bomDeps0.isEmpty) rawDeps
+      else rawDeps.map(dep => dep.copy(dep = dep.dep.addBoms(bomDeps0)))
+    val depMgmt = dependencyManagementDict()
+    if (depMgmt.isEmpty)
+      depsWithBoms
+    else {
+      lazy val depMgmtMap = depMgmt.toMap
+      depsWithBoms
+        .map(dep => dep.copy(dep = dep.dep.withOverrides(dep.dep.overrides ++ depMgmt)))
+        .map { dep =>
+          val key = DependencyManagement.Key(
+            dep.dep.module.organization,
+            dep.dep.module.name,
+            coursier.core.Type.jar,
+            dep.dep.publication.classifier
+          )
+          val versionOverride =
+            if (dep.dep.version == "_") depMgmtMap.get(key).map(_.version)
+            else None
+          val exclusions = depMgmtMap.get(key)
+            .map(_.minimizedExclusions)
+            .map(dep.dep.minimizedExclusions.join(_))
+            .getOrElse(dep.dep.minimizedExclusions)
+          dep.copy(
+            dep = dep.dep
+              .withVersion(versionOverride.getOrElse(dep.dep.version))
+              .withMinimizedExclusions(exclusions)
+          )
+        }
+    }
+  }


This large-ish task is hard to digest. Maybe, we can make it more compact and use some def s with meaningful names. Or add some in-line comments.

Maybe this logic could be moved into a helper method?

lefou · 2024-11-23T15:30:43Z

scalalib/src/mill/scalalib/JavaModule.scala

+   * Data from dependencyManagement, converted to a type ready to be passed to coursier
+   * for dependency resolution
+   */
+  def dependencyManagementDict: Task[Seq[(DependencyManagement.Key, DependencyManagement.Values)]] =


This looks a bit like utility task we could move to CoursierSupport (might not work as-is due to uses of Dep) or CoursierModule.

lihaoyi · 2024-11-24T01:13:16Z

integration/feature/bom/resources/build.mill

+      ivy"com.google.cloud:libraries-bom:26.50.0"
+    )
+    def ivyDeps = Agg(
+      ivy"com.google.protobuf:protobuf-java:_"


Can we leave out the :_ and just use ivy"com.google.protobuf:protobuf-java"? That would make our syntax consistent with Maven and Gradle, both of which just leave out the version segment without any placeholder

Trying that

Leaving out the version entirely would also be consistent with how we currently handle missing versions for contrib plugins. We could remove the special handling and instead auto-include a BOM configuring all contrib plugins.

lihaoyi · 2024-11-24T01:14:14Z

integration/feature/bom/resources/build.mill

@@ -0,0 +1,200 @@
+package build


I don't think this needs to be an integration test. Let's cover all the edge cases in a unit test, and have a single example test under example/{lang}lib/dependencies/ as both documentation for the happy paths and end-to-end verification that the feature works from a user point of view

If I'm not mistaken, there are a few places we should assert that the BOM takes effect end-to-end in a JavaModule/ScalaModule/KotlinModule:

compile

run

jar

assembly

launcher

publishArtifacts

We can run these tasks in the example module for each language and verify that they work (e.g. library resolves and the right version is used)

lihaoyi · 2024-11-24T01:18:59Z

scalalib/src/mill/scalalib/JavaModule.scala

@@ -159,6 +194,109 @@ trait JavaModule
   */
  def runIvyDeps: T[Agg[Dep]] = Task { Agg.empty[Dep] }

+  def parentDep: T[Option[Dep]] = Task { None }


Let's call this bomParentDep? To make it clear what it's used for

lihaoyi · 2024-11-24T01:37:58Z

Left some comments. Some high level questions

How do BOMs interact with moduleDeps? Do they get aggregated and combined somehow, or do they need to be added to each individual module to apply?
How do BOMs interact with runClasspath and compileClasspath? Do they apply universally to both/all? Would it be necessary to have a run-specific BOM and compile-specific BOM?
Do BOMs interact with dependencyManagement sections of third-party ivyDeps?

alexarchambault · 2024-11-25T15:39:03Z

How do BOMs interact with moduleDeps? Do they get aggregated and combined somehow, or do they need to be added to each individual module to apply?

They apply only to the module they're added to. If one wants the BOMs to apply to dependencies added in a second module that depends on the first, the BOMs have to be added to it too. But the BOMs of the first module still apply to dependencies pulled via it by the second module.

alexarchambault · 2024-11-27T20:35:46Z

(Last push may not need reviews yet. It needs a bit of cleanup, I'm just checking if the CI passes)

alexarchambault added 3 commits November 8, 2024 17:03

Remove trailing space

910e8a4

NIT Use allIvyDeps task

4f5f070

WIP Add BOM support

3e8643d

lefou reviewed Nov 11, 2024

View reviewed changes

alexarchambault added 4 commits November 13, 2024 15:19

Merge branch 'main' into pr/bom-support

f796d0c

Conflicts: build.mill

more

bc5e1b3

Merge branch 'main' into pr/bom-support

425b4aa

Conflicts: scalalib/src/mill/scalalib/PublishModule.scala scalalib/src/mill/scalalib/publish/Pom.scala

more

732c659

This comment was marked as outdated.

Sign in to view

alexarchambault added 4 commits November 19, 2024 20:26

fixup

17259c0

more

a8b86e3

Merge branch 'main' into pr/bom-support

67ff17a

Conflicts: build.mill main/util/src/mill/util/CoursierSupport.scala scalalib/src/mill/scalalib/CoursierModule.scala scalalib/src/mill/scalalib/JavaModule.scala scalalib/src/mill/scalalib/Lib.scala

more

067d84d

alexarchambault force-pushed the bom-support branch 2 times, most recently from 2843192 to 067d84d Compare November 20, 2024 18:33

This comment was marked as outdated.

Sign in to view

alexarchambault added 10 commits November 21, 2024 11:37

more

17c3084

Merge branch 'main' into pr/bom-support

c143182

fix

a45ddd6

Revert UnidocModule stuff

d601c44

Merge branch 'main' into pr/bom-support

20f59c5

Merge branch 'main' into pr/bom-support

8c00ccb

tweaking

f53fd8d

…

c549b21

more

06b0c5b

fmt

1828975

more

c9054a3

alexarchambault changed the title ~~Add BOM support~~ Add BOM / dependency management support Nov 21, 2024

alexarchambault added 3 commits November 21, 2024 21:46

Starting to look good 🤔

3dbc225

Merge branch 'main' into pr/bom-support

b260f6f

Merge branch 'main' into pr/bom-support

e13a102

alexarchambault added 2 commits November 22, 2024 11:37

Merge branch 'main' into pr/bom-support

9c75f73

more

97cb113

lefou reviewed Nov 23, 2024

View reviewed changes

lihaoyi reviewed Nov 24, 2024

View reviewed changes

Merge branch 'main' into pr/bom-support

13fdbca

alexarchambault added 5 commits November 25, 2024 19:28

more

8e28493

fmt

7950d4f

Merge branch 'main' into pr/bom-support

4f78fe8

Merge branch 'main' into pr/bom-support

03857e0

more - needs clean-up, only checking if CI passes

745b500

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BOM / dependency management support #3924

Add BOM / dependency management support #3924

alexarchambault commented Nov 8, 2024 •

edited

Loading

himanshumahajan138 commented Nov 9, 2024

alexarchambault commented Nov 9, 2024

himanshumahajan138 commented Nov 9, 2024

lefou left a comment

alexarchambault commented Nov 11, 2024

This comment was marked as outdated.

This comment was marked as outdated.

alexarchambault commented Nov 22, 2024 •

edited

Loading

lefou left a comment

lefou Nov 23, 2024

alexarchambault Nov 25, 2024

lefou Nov 23, 2024

lihaoyi Nov 24, 2024

lefou Nov 23, 2024

lihaoyi Nov 24, 2024

alexarchambault Nov 25, 2024

lefou Nov 25, 2024

lihaoyi Nov 24, 2024 •

edited

Loading

lihaoyi Nov 24, 2024

lihaoyi commented Nov 24, 2024

alexarchambault commented Nov 25, 2024

alexarchambault commented Nov 27, 2024

Add BOM / dependency management support #3924

Are you sure you want to change the base?

Add BOM / dependency management support #3924

Conversation

alexarchambault commented Nov 8, 2024 • edited Loading

himanshumahajan138 commented Nov 9, 2024

alexarchambault commented Nov 9, 2024

himanshumahajan138 commented Nov 9, 2024

lefou left a comment

Choose a reason for hiding this comment

alexarchambault commented Nov 11, 2024

This comment was marked as outdated.

This comment was marked as outdated.

alexarchambault commented Nov 22, 2024 • edited Loading

lefou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lihaoyi Nov 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lihaoyi commented Nov 24, 2024

alexarchambault commented Nov 25, 2024

alexarchambault commented Nov 27, 2024

alexarchambault commented Nov 8, 2024 •

edited

Loading

alexarchambault commented Nov 22, 2024 •

edited

Loading

lihaoyi Nov 24, 2024 •

edited

Loading