Add package.el support

[darkfeline]

Add package.el support so Emacs Lisp libraries can be installed as proper Emacs packages.

[phst]

Thanks a lot! JFYI, since this is a rather large change I won't have time to review it until Dec 30 or so.

[darkfeline]

Not sure why the CI is failing, the log says Build completed successfully which is baffling.

[phst]

Not sure why the CI is failing, the log says Build completed successfully which is baffling.

Looks like it's failing the license check in examples/lib-2-pkg.el, probably best/easiest to add a license header to that file (even if the file is trivial).

[darkfeline]

Looks like it's failing the license check in examples/lib-2-pkg.el, probably best/easiest to add a license header to that file (even if the file is trivial).

ty, obvious in hindsight

[darkfeline]

Is there an easy way to run CI locally? A naive "install bazel and make" doesn't seem to work due to some dep/precondition

[phst]

Is there an easy way to run CI locally? A naive "install bazel and make" doesn't seem to work due to some dep/precondition

That should generally work, you might need to install xmllint and texinfo locally though.

[darkfeline]

For reference, additional deps I installed (to be updated until I get it fully working):

• texinfo (not needed?)
• lcov (for genhtml)

mm, I think I'm only getting failures for some coverage stuff now, not sure though. Something about missing coveragerc, which also looks to be hacked around in the code, so I'm inclined to throw it at the CI again.

[phst]

Why? Is there anything special about testonly that requires this?

[darkfeline]

Libraries used only in tests aren't intended to be packaged and from the ones I've seen it's a little annoying to make them so. The point of the package support is to enable packaging, so it doesn't seem useful to enable it for testonly.

Edit: of course we could just manually set enable_package=False for all testonly targets, but since this would be needed for every single one AFAIK it seems easier to just do this.

[phst]

Is it then possible to test package builds at all? For example, I'd like the analysis test in tests/private/defs.bzl to contain a test for the new functionality.

Maybe it would be better to have a tri-state Boolean (e.g. "always", "never", "auto").

[phst]

hmm, not sure whether only using the basename is the right choice. We intentionally don't add the Bazel package directory to the load path by default so that filenames don't have to be globally unique; e.g. runfiles.el needs to be loaded as elisp/runfiles/runfiles, so it seems a bit weird to call the resulting package just runfiles. Maybe the default package name should also include the Bazel package name (i.e. elisp/runfiles/runfiles), or even the repo name?

[darkfeline]

The package name can't have slashes. Note that this is just the default. In most cases, it is accurate to use the basename. In extraordinary cases explicitly supplying the package name is preferable IMO.

What do you mean by repo name?

[phst]

The package name can't have slashes.

Then maybe replace the slashes by dots, hyphens, or some other allowed character?

In most cases, it is accurate to use the basename.

I don't think so. Bazel organizes files in packages so that filenames don't need to be globally unique. It would also be confusing to have an Elisp package name that differs from the feature name in such major ways. For example, it would be hard to explain why the library elisp/proto/proto should have an Elisp package name that's just proto – presumably the library is in a non-root package to improve uniqueness of the library name.

In extraordinary cases explicitly supplying the package name is preferable IMO.

The default should still be unsurprising and somewhat conservative. If users do want a shorter/different Elisp package name, they can override the default.

What do you mean by repo name?

The name of the repository that the library is placed in. (However, that name also isn't part of the feature name, and the repo name can be somewhat unstable if e.g. the repo isn't a module published in the BCR.)

[darkfeline]

Then maybe replace the slashes by dots, hyphens, or some other allowed character?

That does not work for the vast majority of cases. e.g. the magit package should indeed be magit and not blah-something-google-magit.

Bazel organizes files in packages so that filenames don't need to be globally unique.

Emacs packaging basically assumes one global flat namespace for all library files, so they do have to be unique unfortunately.

Specifically, package names must be unique among package names, and library file names must be unique among all library files across all packages/load-path dirs.

While it is possible to refer to and load libraries with slashes, this is not done in Emacs core even though Emacs distributes libraries in nested subdirs (all subdirs are added to load-path so libraries can be loaded by basename) or in any Emacs package manager, 1p or 3p that I know of. This is primarily used when Emacs is used as a batch interpreter, to explicitly load scripts via paths with slashes.

It would also be confusing to have an Elisp package name that differs from the feature name in such major ways

The feature name is expected to be the base name of the library file sans extension (note that a package can contain library files with names different from the package name, although conventionally a package should contain a library file with the same name as the package.).

I did a cursory check and all of our libraries also follow this convention (i.e., a package blah/something.el provides the feature something).

The default should still be unsurprising and somewhat conservative.

This default is unsurprising IMO. It works for almost all of our packages.

[phst]

Then maybe replace the slashes by dots, hyphens, or some other allowed character?

That does not work for the vast majority of cases. e.g. the magit package should indeed be magit and not blah-something-google-magit.

Yes, in such a case the BUILD file should specify the appropriate package name.

Bazel organizes files in packages so that filenames don't need to be globally unique.

Emacs packaging basically assumes one global flat namespace for all library files, so they do have to be unique unfortunately.

Specifically, package names must be unique among package names, and library file names must be unique among all library files across all packages/load-path dirs.

While it is possible to refer to and load libraries with slashes, this is not done in Emacs core even though Emacs distributes libraries in nested subdirs (all subdirs are added to load-path so libraries can be loaded by basename) or in any Emacs package manager, 1p or 3p that I know of. This is primarily used when Emacs is used as a batch interpreter, to explicitly load scripts via paths with slashes.

Emacs core does use library/feature names with slashes, namely in the "cedet" and "term" subdirectories. Granted, it's not very common, but does happen.

It would also be confusing to have an Elisp package name that differs from the feature name in such major ways

The feature name is expected to be the base name of the library file sans extension (note that a package can contain library files with names different from the package name, although conventionally a package should contain a library file with the same name as the package.).

While this is a common convention, it's not universal nor required.

The default should still be unsurprising and somewhat conservative.

This default is unsurprising IMO. It works for almost all of our packages.

It's not so important how the distribution for each approach is because the two choices aren't symmetric.

1. If the basename without directory is chosen as default for the package name, it's likely that unintentionally non-unique package names are created. That's bad because it will result in nasty clashes which might be noticed only when it's too late, e.g. when packages from multiple repositories are merged.
2. On the other hand, if the repository-relative filename is chosen as default, such clashes are less likely because filenames across Bazel repositories are generally assumed to be unique already (even ignoring the repository name). An unintentionally wrong default will then only result in an overly long package name, which is annoying but not harmful.

Given that defaults should be safe, (2) would be the better choice.

---

There are approaches to improve the default, at the expense of a bit more coding complexity. Specifically, the rule author has already stated their intention regarding uniqueness with the load_path attribute: In the common case of load_path = ["."], the package-relative filename is already expected to be globally unique, and the package build code can make use of that expectation. Specifically, I think the following algorithm should give good results in most cases:

1. Normalize the filename so that it's repository-relative (i.e. prepend the Bazel package name to the package-relative filename).
2. Normalize all load path entries so that they are repository-relative as well (see the logic in _compile in elisp/defs.bzl).
3. Relativize the normalized filename against the normalized load path, i.e. strip each load path element in turn from the normalized filename. Fail if that doesn't work, i.e. if the filename isn't inside the load path directory.
4. Take the shortest of the relative filenames. If there's no unique shortest relative filename, fail.
5. Strip the extension to turn the filename into a feature name.
6. Replace slashes in the feature name to obtain a valid Emacs package name.

For example, if a package foo/bar contains a file baz/qux.el with

elisp_library(name = "qux", srcs = ["baz/qux.el"], load_path = ["/foo", "."])

this will result in

1. foo/bar/baz/qux.el
2. ["foo", "foo/bar"]
3. ["bar/baz/qux.el", "baz/qux.el"]
4. baz/qux.el
5. baz/qux
6. baz-qux

Such a file would need to provide at least one of the features baz/qux and bar/baz/qux for require to work, so the generated Emacs package name would match the shortest possible feature name.

[phst]

what happens if these files are in subdirectories? Shouldn't this preserve the directory structure?

[darkfeline]

Packages are flat, so subdirs aren't supported (the packaging framework will only add the package dir to load-path). You can't have libraries with the same name in the load-path namespace anyway (one will override/shadow the other).

[darkfeline]

The Windows build is failing due to a random ASCII 15 character that seems to be getting added to elisp/proto/proto.el. I'm dealing with the pain of trying to set up an environment in Windows... maybe a long shot but if you have any ideas it might save me a lot of time/pain.

Edit: it's 0o15 which is ASCII 13 which is carriage return. So the root cause is pretty obvious (and I feel stupid)...

[darkfeline]

(FYI this is ready to land (or if you have any comments))

[darkfeline]

Friendly ping?

[darkfeline]

Ping?