WIP: Fetch from Clojars & better SNAPSHOT support #11

martinklepsch · 2018-01-09T11:58:40Z

This is still very much a work in progress but opening a PR for feedback.

Some of the motivation for this has been outlined in https://github.com/boot-clj/boot-bin/issues/6

Besides speed this will also greatly simplify the release process of SNAPSHOTs.

Currently whenever a SNAPSHOT release is being cut we need to create a release on Github. This in itself is a bit of manual labor but fine. The problem is that Github releases are thought of as immutable. You can update files attached to a release but nobody will notice.

This makes it hard for Boot maintainers to ship development builds, making it harder for people to test those etc.pp.

If you believe we shouldn't be using SNAPSHOTs at all chat me up in #boot-dev on Slack. I've thought how we could use proper releases but couldn't come up with an approach that would keep boot -u work as it currently does.

One Caveat/Feature: If you're running a SNAPSHOT it won't update automatically. You'll need to remove the respective files in ~/.boot/cache/bin/$VERSION to trigger a redownload of the updated SNAPSHOT.

martinklepsch · 2018-01-09T13:19:44Z

This isn't properly functioning yet for a few reasons:

boot/base/target/base-$(version)-jar-with-dependencies.jar isn't being uploaded to Clojars (fixed in deploy the boot/base uberjar to Clojars boot#681)
Instead the other boot.jar is being uploaded, similar size but doesn't have a main class specified.

Copying some stuff from Slack:

@micha @alandipert if you have a moment can you clarify the differences between base.jar and base-jar-with-dependencies.jar— from what I can see:

base.jar: built with mvn -q install
-with-depdendencies: built with mvn -q assembly:assembly
both are similar in size

Could we eliminate one of them? do we need both?

Actually it seems that the base.jar after mvn -q install is only 16K — only after the assembly plugin runs the filesize goes to about 10M. Interestingly all the jars for boot/base on Clojars are 8M+ which makes me think, maybe we never uploaded the 16K jar?

Also it seems that boot/base hasn’t been downloaded nearly as much as other boot jars

martinklepsch · 2018-01-09T13:50:18Z

src/boot/bin/BootSnapshotVersionFetcher.java

+            if (Arrays.equals(stack.toArray(), marker) && snapshotVersion == null) {
+                // String versionString = new String(ch, start, length);
+                // System.out.println("version " + versionString);
+                snapshotVersion = new String(ch, start, length); }}};


This doesn't feel terribly robust but at the same time it should work with all the maven metadata files I've seen. Thoughts?

EDIT by this I mean the logic how we decide when we have found a proper version string.

martinklepsch · 2018-01-09T13:53:15Z

src/Boot.java

@@ -182,7 +183,14 @@

    public static String
    downloadUrl(String version, String name) throws Exception {
-        return String.format("https://github.com/boot-clj/boot/releases/download/%s/%s", version, name); }
+        if (0 < version.compareTo("2.8") && name.equals("boot.jar")) {


Hacky version comparison — could this be done better?
Also guarding that boot.sh isn't requested, it should never be as per #9 however.

danielcompton

I don't fully understand all the context here, but Clojars also has an API that you could use to lookup the latest version of Boot, which would mean you wouldn't need to do SNAPSHOT releases, and would probably give you enough info to download the JAR without parsing the maven-metadata.xml file?

danielcompton · 2018-01-09T20:20:37Z

src/Boot.java

@@ -182,7 +183,14 @@

    public static String
    downloadUrl(String version, String name) throws Exception {
-        return String.format("https://github.com/boot-clj/boot/releases/download/%s/%s", version, name); }
+        if (0 < version.compareTo("2.8") && name.equals("boot.jar")) {
+            String urlFormat = "https://clojars.org/repo/boot/base/%s/base-%s-uber.jar";


Use https://repo.clojars.org/boot/base/... to get the CDN version.

martinklepsch · 2018-01-09T20:52:49Z

@danielcompton Thanks for your feedback! The issue with non-snapshot builds is that any invocation of boot -u will look for "RELEASE" builds and thus will download any releases like 2.8.0-alpha1.

An alternative solution to this issue could be modifying boot -u to search for versions without any qualifier. Not sure how easy or hard that would be. Maybe we could also do something like Cojurescript and append the commit number but then I'm not sure how we would differentiate pre-release builds from actual releases.

One benefit of snapshots is also that we could publish pre-releases as part of CI for new commits on master.

Maybe this needs more thought. Would greatly welcome additional input!

martinklepsch · 2018-01-10T13:03:33Z

FYI, I will split this PR into two:

Downloading the jar from clojars
SNAPSHOT stuff (potentially to be revised)

This change shouldn't have any effect until the respective FEATURE FLAG is switched on.

martinklepsch · 2018-01-12T08:48:46Z

I updated this PR with only the changes needed to download boot/base from Clojars.

For now downloading SNAPSHOTs from Clojars is not supported.
There's a feature flag that needs to be switched before we want to release this.
We will need to reconsider ways to ship pre-releases, SNAPSHOTs have their drawbacks as do proper releases.

burn2delete · 2019-01-24T04:21:24Z

This issue has been resolved in the superseding project.
http://github.com/boot-clj/boot-native

martinklepsch force-pushed the fetch-from-clojars branch 2 times, most recently from 315d183 to 778cf34 Compare January 9, 2018 12:09

martinklepsch force-pushed the fetch-from-clojars branch 3 times, most recently from 5dab96c to 244b6e5 Compare January 9, 2018 13:48

martinklepsch commented Jan 9, 2018

View reviewed changes

martinklepsch force-pushed the fetch-from-clojars branch from 244b6e5 to 28caf61 Compare January 9, 2018 13:52

martinklepsch commented Jan 9, 2018

View reviewed changes

martinklepsch mentioned this pull request Jan 9, 2018

deploy the boot/base uberjar to Clojars boot-clj/boot#681

Open

danielcompton reviewed Jan 9, 2018

View reviewed changes

Download boot/base jars from Clojars (disabled)

b5ea8a7

This change shouldn't have any effect until the respective FEATURE FLAG is switched on.

martinklepsch force-pushed the fetch-from-clojars branch from 28caf61 to b5ea8a7 Compare January 12, 2018 08:43

burn2delete closed this Jan 24, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Fetch from Clojars & better SNAPSHOT support #11

WIP: Fetch from Clojars & better SNAPSHOT support #11

martinklepsch commented Jan 9, 2018

martinklepsch commented Jan 9, 2018 •

edited

Loading

martinklepsch Jan 9, 2018 •

edited

Loading

martinklepsch Jan 9, 2018

danielcompton left a comment

danielcompton Jan 9, 2018

martinklepsch commented Jan 9, 2018

martinklepsch commented Jan 10, 2018

martinklepsch commented Jan 12, 2018

burn2delete commented Jan 24, 2019

WIP: Fetch from Clojars & better SNAPSHOT support #11

WIP: Fetch from Clojars & better SNAPSHOT support #11

Conversation

martinklepsch commented Jan 9, 2018

martinklepsch commented Jan 9, 2018 • edited Loading

Copying some stuff from Slack:

martinklepsch Jan 9, 2018 • edited Loading

Choose a reason for hiding this comment

martinklepsch Jan 9, 2018

Choose a reason for hiding this comment

danielcompton left a comment

Choose a reason for hiding this comment

danielcompton Jan 9, 2018

Choose a reason for hiding this comment

martinklepsch commented Jan 9, 2018

martinklepsch commented Jan 10, 2018

martinklepsch commented Jan 12, 2018

burn2delete commented Jan 24, 2019

martinklepsch commented Jan 9, 2018 •

edited

Loading

martinklepsch Jan 9, 2018 •

edited

Loading