Add Elixir.mk - Allows for use of Elixir deps within an Erlang project. #979

artman41 · 2023-06-04T17:21:54Z

This change introduces a new core plugin called elixir.mk.

This change accomplishes the following:

Converts mix.exs projects to use erlang.mk
Compiles .ex files, allowing for a project to use either Erlang, Elixir or a combination of both
Properly configures paths so that Elixir dependencies are added to the $(PROJECT).app
Treats Elixir as a dependency & fetches it as such, rather than relying on a system installation

This change makes use of hex.mk

artman41 · 2023-06-04T17:27:37Z

Currently rebasing the change :)

artman41 · 2023-06-04T17:32:37Z

@essen I've added a test under test/core_elixir.mk though I've been testing locally with a sample project.

essen · 2023-06-05T06:47:28Z

Hello! I've not had a good look yet but I doubt this works with some Elixir NIFs since they often have a Makefile and thus wouldn't enter the elixir autopatch.

artman41 · 2023-06-05T08:03:57Z

Hello! I've not had a good look yet but I doubt this works with some Elixir NIFs since they often have a Makefile and thus wouldn't enter the elixir autopatch.

Damn, that's a good shout - any idea of an elixir lib which uses NIFs?

essen · 2023-06-05T08:48:51Z

There's a few in this search but gotta filter. Oftentimes if the page says mix + make as a build tool, that's an Elixir NIF. But not always.

https://hex.pm/packages?search=nif&sort=total_downloads

artman41 · 2023-06-05T22:54:37Z

There's a few in this search but gotta filter. Oftentimes if the page says mix + make as a build tool, that's an Elixir NIF. But not always.

https://hex.pm/packages?search=nif&sort=total_downloads

I've just commited a change which should now make it support Elixir Nifs (assuming they use elixir_make - I'm not 100% familiar with the Elixir NIF ecosystem so not quite sure if I've accounted for the right things there).

I've added an additional test which will attempt to compile libsalty2, though this depends on libsodium.

If you've any more suggestions @essen, I'm more than happy to add them 👍

artman41 · 2023-06-21T12:55:55Z

Any chance I could poke you to look this over @essen? 😄

essen · 2023-06-26T10:39:27Z

core/elixir.mk

+	$(verbose) $(MAKE) --no-print-directory $(PROJECT).d
+	$(verbose) $(MAKE) --no-print-directory app-build
+
+app-build: ebin/$(PROJECT).app


Should be app-build::.

essen · 2023-06-26T10:40:42Z

Does this work even if using local system installed Elixir?

artman41 · 2023-06-26T17:29:47Z

Does this work even if using local system installed Elixir?

In what sense @essen?

I'm expecting elixir to always be loaded as a dep of the project rather than as an installed version on the system - I can try and account for this though if you'd prefer that functionality to be in there.

I'll edit this for confirmation but the only thing I think that could be problematic with an installed version of elixir is if code:get_path() returned paths to the installed elixir version, though I wouldn't ever expect that to be the case with a standard install.

essen · 2023-06-27T07:17:45Z

I think we should take the dep Elixir if one is available as a dep, and the system Elixir otherwise. Basically the dep can be used as an override.

Most projects don't require a specific version of Elixir, and for example in RabbitMQ we explicitly use the system Elixir (we even build packages for it).

artman41 · 2023-07-02T23:24:41Z

@essen could you have a look over when you have chance?

Based on your feedback, I've updated the MR to default to using the System Install of Elixir if available. This can be manually overriden by a user-set var though (ELIXIR_USE_SYSTEM)

Let me know what you think :)

artman41 · 2023-07-05T10:22:32Z

@essen could you have a look over when you have chance?

Based on your feedback, I've updated the MR to default to using the System Install of Elixir if available. This can be manually overriden by a user-set var though (ELIXIR_USE_SYSTEM)

Let me know what you think :)

just poking you 😄

essen · 2023-07-05T10:33:37Z

I'll get to it no need to remind so often.

artman41 · 2023-09-02T17:11:19Z

@essen sorry to poke on this one again, any idea when you'll be able to get round to it? :)

essen · 2023-09-04T12:42:32Z

test/log/console.log

I don't think these log files should be commited.

essen · 2023-09-04T12:46:31Z

core/elixir.mk

+		|| git describe --dirty --abbrev=7 --tags --always 2>/dev/null || true))
+	$(eval MODULES := $(patsubst %,'%',$(sort $(notdir $(basename \
+		$(filter-out $(ELIXIRC_EXCLUDE_PATHS), $(ERL_FILES) $(CORE_FILES) $(BEAM_FILES)))))))
+	$(eval MODULES := $(call UNIQUE,$(MODULES) $(foreach file, \


Wondering why the unique filter is necessary? I don't think modules should have duplicates.

I think that's a fair point, likely more safety/fear on my part 👍

essen · 2023-09-04T12:48:41Z

core/deps.mk

@@ -148,7 +148,7 @@ endif

 # Core targets.

-ALL_APPS_DIRS_TO_BUILD = $(if $(LOCAL_DEPS_DIRS)$(IS_APP),$(LOCAL_DEPS_DIRS),$(ALL_APPS_DIRS))
+ALL_APPS_DIRS_TO_BUILD = $(if $(filter-out $(ELIXIR_BUILTINS),$(LOCAL_DEPS_DIRS))$(IS_APP),$(filter-out $(ELIXIR_BUILTINS),$(LOCAL_DEPS_DIRS)),$(ALL_APPS_DIRS))


I don't think Elixir builtins should be in LOCAL_DEPS_DIRS to begin with. It's defined as:

LOCAL_DEPS_DIRS = $(foreach a,$(LOCAL_DEPS),$(if $(wildcard $(APPS_DIR)/$(a)),$(APPS_DIR)/$(a)))

It's not meant to contain applications the user didn't write.

likely a misunderstanding on my part - I'd assumed they needed adding to the LOCAL_DEPS_DIRS var since I was adding them to LOCAL_DEPS

essen · 2023-09-04T13:01:39Z

core/deps.mk

@@ -778,14 +780,16 @@ define dep_fetch_ln
 	ln -s $(call dep_repo,$(1)) $(DEPS_DIR)/$(call dep_name,$(1));
 endef

+HEX_REPO = https://repo.hex.pm


Let's not extract that into a variable. We want to use hex_core.

My only pushback against that would be for if someone is using an internal mirror within a corporate setting - it's likely they'd want to instead have the hex side of erlang.mk pointing at whatever their mirror is

looking back at it, I think I'd likely want to change this to be

HEX_REPO ?= https://repo.hex.pm

essen · 2023-09-04T13:06:24Z

core/erlc.mk

@@ -86,7 +86,7 @@ define app_file
 endef
 endif

-app-build: ebin/$(PROJECT).app
+app-build:: ebin/$(PROJECT).app


I'm not sure this is necessary? I don't think the app-build:: ... rule in the elixir.mk file does anything more than we do here. It doesn't need to be defined in two places.

it's a fair point, I've also removed the $(PROJECT).d target & the body of the app:: target above this

essen · 2023-09-04T13:07:33Z

core/elixir.mk

@@ -0,0 +1,334 @@
+ifeq ($(pkg_elixir_commit),master)
+pkg_elixir_commit = main
+endif


Should just change the commit in the package.

essen · 2023-09-04T13:14:23Z

core/elixir.mk

+	$(if $(PROJECT_MOD),{mod$(comma) {'$(PROJECT_MOD)'$(comma) []}}$(comma),)
+	{env, $(subst \,\\,$(PROJECT_ENV))}$(if $(findstring {,$(PROJECT_APP_EXTRA_KEYS)),$(comma)$(newline)$(tab)$(subst \,\\,$(PROJECT_APP_EXTRA_KEYS)),)
+]}.
+endef


Not sure why this was redeclared either.

The reason for this redeclaration was to have .app files generate where the mod may actually be an elixir module

Currently, the mod prop only gets inserted if an erlang file exists.

In efforts to separate erlc.mk from elixir.mk, I opted to redefine the macro - if you'd prefer, I could add a condition to the definition in erlc.mk?

essen · 2023-09-04T13:17:21Z

core/elixir.mk

+						erlang:halt(1)
+				end,
+			maps:get(version, PackageInfo)
+		end,


This should probably be moved to the hex part so we can use it for Erlang deps as well.

Assuming you meant the GetDeps fun, I've moved that to elixir_get_deps.erl within elixir.mk - if you meant the macro itself, let me know and I'll move it over 👍

essen · 2023-09-04T13:21:43Z

Is the thing that transforms paths like Elixir/MyApp/MyMod.ex into module name Elixir.MyApp.MyMod something the compiler does, or am I just blind and I missed it?

artman41 · 2024-10-08T23:42:55Z

@essen had some time recently so will be picking this one back up 👍

artman41 · 2024-10-09T00:04:39Z

Is the thing that transforms paths like Elixir/MyApp/MyMod.ex into module name Elixir.MyApp.MyMod something the compiler does, or am I just blind and I missed it?

Compiler magic 🪄

essen · 2024-11-05T16:30:14Z

I will try to make this work with RabbitMQ CLI which is (currently) in Elixir. Could you rebase to run CI now that I fixed it? Also you will need to edit the workflow to add/change the new suites here https://github.com/ninenines/erlang.mk/blob/master/.github/workflows/ci.yaml#L27

artman41 · 2024-11-16T13:23:37Z

@essen rebased, hopefully ci should work as expected 👍

essen · 2024-11-28T13:47:59Z

I'm looking to merge this now. No need to rebase, I am probably going to cherry-pick parts of the code while I try to make things work.

artman41 force-pushed the elixir.mk branch from 17768ba to 3c51552 Compare June 4, 2023 17:28

artman41 mentioned this pull request Jun 4, 2023

Feature: Compilation of Elixir projects #980

Open

essen reviewed Jun 26, 2023

View reviewed changes

artman41 force-pushed the elixir.mk branch from 3c5a3ec to f5b2b61 Compare July 2, 2023 23:23

artman41 force-pushed the elixir.mk branch from f5b2b61 to 79fd616 Compare July 4, 2023 00:52

artman41 force-pushed the elixir.mk branch from 52f8fd9 to bb8c9e6 Compare July 23, 2023 18:46

essen reviewed Sep 4, 2023

View reviewed changes

essen force-pushed the master branch from 21847bc to df10c43 Compare October 3, 2024 11:08

artman41 force-pushed the elixir.mk branch 2 times, most recently from 74a2fa0 to d141f2c Compare October 8, 2024 23:41

artman41 added 8 commits November 16, 2024 13:21

Implement 'hex' for erlang.mk

5894114

qa: Remove test log files

f43270c

qa: Remove 'unique' on elixir modules

b00d631

qa: Remove 'LOCAL_DEPS_DIRS' changes

d51cd2e

qa: Remove duplicate targets

09e2e85

qa: Remove 'fix' of setting the elixir commit to 'main'

2b6909a

qa: Move 'GetDeps' fun to its own macro within hex.mk

5c24a76

Some cleanup + if crypto isn't installed, fail earlier

701ba5a

artman41 force-pushed the elixir.mk branch from c178a3b to 6a0a340 Compare November 16, 2024 13:23

Add suites to CI

272cd39

artman41 force-pushed the elixir.mk branch from 6a0a340 to 272cd39 Compare November 16, 2024 13:49

This was referenced Nov 21, 2024

Latest erlang.mk gets stuck in loop building amqp_client #1014

Closed

Erlang.mk and Elixir apps #837

Closed

Add Elixir.mk - Allows for use of Elixir deps within an Erlang project. #979

Are you sure you want to change the base?

Add Elixir.mk - Allows for use of Elixir deps within an Erlang project. #979

Conversation

artman41 commented Jun 4, 2023 • edited Loading

artman41 commented Jun 4, 2023

artman41 commented Jun 4, 2023

essen commented Jun 5, 2023

artman41 commented Jun 5, 2023 • edited Loading

essen commented Jun 5, 2023

artman41 commented Jun 5, 2023

artman41 commented Jun 21, 2023

Choose a reason for hiding this comment

essen commented Jun 26, 2023

artman41 commented Jun 26, 2023

essen commented Jun 27, 2023

artman41 commented Jul 2, 2023

artman41 commented Jul 5, 2023

essen commented Jul 5, 2023

artman41 commented Sep 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

essen commented Sep 4, 2023

artman41 commented Oct 8, 2024

artman41 commented Oct 9, 2024

essen commented Nov 5, 2024

artman41 commented Nov 16, 2024

essen commented Nov 28, 2024

artman41 commented Jun 4, 2023 •

edited

Loading

artman41 commented Jun 5, 2023 •

edited

Loading