Skip to content

Commit

Permalink
docs: improve documentation (#109)
Browse files Browse the repository at this point in the history
* chore: ignore MODULE.bazel.lock

27242f4 pushed lock version to `12` (my guess is Bazel 8?) plus other
changes that e.g. don't match running with `.bazelversion`'s `7.3.1`.

Regardless, the `MODULE.bazel.lock` file is still not stable and e.g.
just running it in another operating system causes things to change...
so let's just ignore it like I did for the `e2e` tests in 70c14c0

* docs: README.md

Revamp README adding an improved version of the Bzlmod and `WORKSPACE`
snippets that are available in the release page.

IMHO the release page should now remove those snippets and point to the
README.

* docs: revamp apt.macro doc and create apt/extensions doc

Split the Bzlmod / `WORKSPACE` docs from `apt.install` macro and remove
a bunch of old references. Also mark it all as legacy since the present
and future is all Bzlmod.

Create apt/extensions docs.

Add a whole common section to both docs with (hopefully :)
easy-to-follow documentation inspired by my answer to a Slack question
(https://bazelbuild.slack.com/archives/CA3NW13MH/p1729804678924819).
  • Loading branch information
jjmaestro authored Nov 16, 2024
1 parent 45509b0 commit a555377
Show file tree
Hide file tree
Showing 9 changed files with 543 additions and 146 deletions.
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# ignoring e2e MODULE.bazel.lock for the time being
# ignoring MODULE.bazel.lock for the time being
# see https://github.com/bazelbuild/bazel/issues/20369
MODULE.bazel.lock
e2e/smoke/MODULE.bazel.lock
bazel-*
.bazelrc.user
Expand Down
182 changes: 144 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,62 +1,168 @@
# Bazel rules for fetching Debian packages
# `rules_distroless`

This ruleset designed to replace commands such as `apt-get install`, `passwd`, `groupadd`, `useradd`, `update-ca-certificates`.
Bazel helper rules to aid with some of the steps needed to create a Linux /
Debian installation. These rules are designed to replace commands such as
`apt-get install`, `passwd`, `groupadd`, `useradd`, `update-ca-certificates`.

> [!CAUTION]
> `rules_distroless` is currently in beta and does not yet offer a stable
> Public API. However, many users are already successfully using it in
> production environments. Check [Adopters](#adopters) to see who's already
> using it.
> [!NOTE]
> rules_distroless is an beta software and doesn't have a stable Public API yet, however many are already using it in production.
>
> See [Adopters](#adopters) section to see who's already using it.

# Usage

Our [examples](/examples) demonstrate how to accomplish typical tasks such as <b>create a new user group</b> or <b>create a new home directory</b>.
## Bzlmod (Bazel 6+)

> [!NOTE]
> If you are using Bazel 6 you need to enable Bzlmod by adding `common
> --enable_bzlmod` to `.bazelrc`
> If you are using Bazel 7+ [it's enabled by default].
Add the following to your `MODULE.bazel` file:

```starlark
bazel_dep(name = "rules_distroless", version = "0.3.9")
```

You can find the latest release version in the [Bazel Central Registry].

If you want to use a specific commit (e.g. there are commits in `main` that are
still not part of a release) you can use one of the few mechanisms that Bazel
provides to override repos.

You can use [`git_override`], [`archive_override`], etc (or
[`local_path_override`] if you want to test a local patch):
```starlark
bazel_dep(name = "rules_distroless", version = "0.3.9")

git_override(
module_name = "rules_distroless",
remote = "https://github.com/GoogleContainerTools/rules_distroless.git",
commit = "6ccc0307f618e67a9252bc6ce2112313c2c42b7f",
)
```

## `WORKSPACE` (legacy)

> [!WARNING]
> Bzlmod is replacing the legacy `WORKSPACE` system. The `WORKSPACE` file will
> be disabled by default in Bazel 8 (late 2024) and will be completely removed
> in Bazel 9 (late 2025). Please migrate to Bzlmod following the steps in the
> [Bzlmod migration guide].
Add the following to your `WORKSPACE` file:

```starlark
REPO = "https://github.com/GoogleContainerTools/rules_distroless"

VERSION = "0.3.8"
SHA256 = "6d1d739617e48fc3579781e694d3fabb08fc6c9300510982c01882732c775b8e"
URL = "{repo}/releases/download/v{v}/rules_distroless-v{v}.tar.gz".format(repo=REPO, v=VERSION)

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_distroless",
sha256 = SHA256,
strip_prefix = "rules_distroless-{}".format(VERSION),
url = URL,
)
```

You can find the latest release in the [`rules_distroless` Github releases
page].

If you want to use a specific commit (e.g. there are commits in `main` that are
still not part of a release) you can change the Github URL pointing it to a
Github archive, as follows:

```starlark
REPO = "https://github.com/GoogleContainerTools/rules_distroless"

COMMIT = "6ccc0307f618e67a9252bc6ce2112313c2c42b7f"
SHA256 = ""
URL = "{}/archive/{}.tar.gz".format(REPO, COMMIT)

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_distroless",
sha256 = SHA256,
strip_prefix = "rules_distroless-{}".format(COMMIT),
url = URL,
)
```

Note that the `SHA256` is initially empty. This is the easiest way to get the
correct value because Bazel will print a warning message with the hash so you
can use it to get rid of the warning.

> [!CAUTION]
> GitHub source archives don't have a strong guarantee on the sha256 stability.
> Check Github's [Update on the future stability of source code archives and
> hashes] for more information.

# Examples

The [examples](/examples) demonstrate how to accomplish typical tasks such as
**create a new user group** or **create a new home directory**:

- [groupadd](/examples/group)
- [passwd](/examples/passwd)
- [useradd --home](/examples/home)
- [update-ca-certificates](/examples/cacerts)
- [keytool](/examples/java_keystore)
- [apt-get install](/examples/debian_snapshot) <i>from Debian repositories.</i>
- [apt-get install](/examples/ubuntu_snapshot) <i>from Ubuntu repositories.</i>
- [apt-get install](/examples/debian_snapshot) from Debian repositories.
- [apt-get install](/examples/ubuntu_snapshot) from Ubuntu repositories.

We also we have distroless-specific rules that could be useful
We also have `distroless`-specific rules that could be useful:

- [flatten](/examples/flatten): <i>flatten multiple `tar` archives.</i>
- [os_release](/examples/os_release): <i>create a `/etc/os-release` file</i>
- [locale](/examples/locale): <i>strip `/usr/lib/locale` to be smaller.</i>
- [dpkg_statusd](/examples/statusd): <i>creates a package database at /var/lib/dpkg/status.d for scanners to discover installed packages.</i>
- [flatten](/examples/flatten): flatten multiple `tar` archives.
- [os_release](/examples/os_release): create an `/etc/os-release` file.
- [locale](/examples/locale): strip `/usr/lib/locale` to be smaller.
- [dpkg_statusd](/examples/statusd): creates a `/var/lib/dpkg/status.d`
package database for scanners to discover installed packages.


# Public API Docs

- [apt](/docs/apt.md) Repository rule for fetching/installing Debian/Ubuntu packages.
- [linux](/docs/rules.md) Various rules for creating Linux specific files.
To read more specific documentation for each of the rules in the repo please
check the following docs:

- [apt](/docs/apt.md): repository rule for installing Debian/Ubuntu packages.
- [apt macro](/docs/apt_macro.md): legacy macro for installing Debian/Ubuntu
packages.
- [rules](/docs/rules.md): various helper rules to aid with creating a Linux /
Debian installation from scratch.

## Installation

See the install instructions on the release notes: <https://github.com/GoogleContainerTools/rules_distroless/releases>

To use a commit rather than a release, you can point at any SHA of the repo.

With bzlmod, you can use `archive_override` or `git_override`. For `WORKSPACE`, you modify the `http_archive` call; for example to use commit `abc123` with a `WORKSPACE` file:

1. Replace `url = "https://github.com/GoogleContainerTools/rules_distroless/releases/download/v0.1.0/rules_distroless-v0.1.0.tar.gz"`
with a GitHub-provided source archive like `url = "https://github.com/GoogleContainerTools/rules_distroless/archive/abc123.tar.gz"`
1. Replace `strip_prefix = "rules_distroless-0.1.0"` with `strip_prefix = "rules_distroless-abc123"`
1. Update the `sha256`. The easiest way to do this is to comment out the line, then Bazel will
print a message with the correct value.

> Note that GitHub source archives don't have a strong guarantee on the sha256 stability, see
> <https://github.blog/2023-02-21-update-on-the-future-stability-of-source-code-archives-and-hashes>

# Contributing
This ruleset is primarily funded to support [Google's `distroless` container
images]. We may not work on feature requests that do not support this mission.

This ruleset is primarily funded to support [distroless](github.com/GoogleContainerTools/distroless). We may not work on feature requests that do not support this mission. We will however accept fully tested contributions via pull requests if they align with the project goals (ex. a different compression format) and may reject requests that do not (ex. supporting a non `deb` based packaging format).

# Adopters
We will however accept fully tested contributions via pull requests if they
align with the project goals (e.g. add support for a different compression
format) and may reject requests that do not (e.g. supporting other packaging
formats other than `.deb`).

- distroless: https://github.com/GoogleContainerTools/distroless
- Arize AI: https://www.arize.com

> An adopter? Add your company here by sending us a Pull Request.
# Adopters
- [Google's `distroless` container images]
- [Arize AI](https://www.arize.com)

> [!TIP]
> Are you using `rules_distroless`? Please send us a Pull Request to add your
> project or company name here!

[it's enabled by default]: https://blog.bazel.build/2023/12/11/bazel-7-release.html#bzlmod
[Bazel Central Registry]: https://registry.bazel.build/modules/rules_distroless
[`git_override`]: https://bazel.build/versions/6.0.0/rules/lib/globals#git_override
[`archive_override`]: https://bazel.build/versions/6.0.0/rules/lib/globals#archive_override
[`local_path_override`]: https://bazel.build/versions/6.0.0/rules/lib/globals#local_path_override
[Bzlmod migration guide]: https://bazel.build/external/migration
[`rules_distroless` Github releases page]: https://github.com/GoogleContainerTools/rules_distroless/releases
[Update on the future stability of source code archives and hashes]: https://github.blog/2023-02-21-update-on-the-future-stability-of-source-code-archives-and-hashes
[Google's `distroless` container images]: https://github.com/GoogleContainerTools/distroless
[Arize AI]: https://www.arize.com
1 change: 1 addition & 0 deletions apt/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ load("@bazel_skylib//:bzl_library.bzl", "bzl_library")

exports_files([
"apt.bzl",
"extensions.bzl",
])

bzl_library(
Expand Down
118 changes: 76 additions & 42 deletions apt/apt.bzl
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
"apt-get"
"""
`apt.install` macro
This documentation provides an overview of the convenience `apt.install`
repository macro to create Debian repositories with packages "installed" in
them and available to use in Bazel.
"""

load("//apt/private:index.bzl", _deb_package_index = "deb_package_index")
load("//apt/private:resolve.bzl", _deb_resolve = "deb_resolve")
Expand All @@ -10,80 +16,108 @@ def _apt_install(
nolock = False,
package_template = None,
resolve_transitive = True):
"""A convience repository macro for apt rules.
"""Repository macro to create Debian repositories.
This documentation provides an overview of the convenience repository macro around `package_index` and `resolve` repository rules.
> [!WARNING]
> THIS IS A LEGACY MACRO. Use it only if you are still using `WORKSPACE`.
> Otherwise please use the [`apt` module extension](apt.md).
### WORKSPACE Example
Here's an example to create a Debian repo with `apt.install`:
```starlark
load("@rules_distroless//apt:index.bzl", "deb_index")
# WORKSPACE
load("@rules_distroless//apt:apt.bzl", "apt")
deb_index(
apt.install(
name = "bullseye",
# For the initial setup, the lockfile attribute can be omitted and
# generated by running
# bazel run @bullseye//:lock
# This will generate the lock.json file next to the manifest file by
# replacing `.yaml` with `.lock.json`.
# If you explicitly want to run without a lock, set nolock to True to
# avoid the DEBUG messages
lock = "//examples/apt:bullseye.lock.json",
# lock = "//examples/apt:bullseye.lock.json",
manifest = "//examples/apt:bullseye.yaml",
)
load("@bullseye//:packages.bzl", "bullseye_packages")
bullseye_packages()
```
### BZLMOD Example
Note that, for the initial setup (or if we want to run without a lock) the
lockfile attribute can be omitted. All you need is a YAML
[manifest](/examples/debian_snapshot/bullseye.yaml):
```yaml
version: 1
```starlark
apt = use_extension("@rules_distroless//apt:extensions.bzl", "apt")
apt.install(
name = "bullseye",
lock = "//examples/apt:bullseye.lock.json",
manifest = "//examples/apt:bullseye.yaml",
)
use_repo(apt, "bullseye")
sources:
- channel: bullseye main
url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z
archs:
- amd64
packages:
- perl
```
### Macro Expansion
`apt.install` will parse the manifest and will fetch and install the
packages for the given architectures in the Bazel repo `@<NAME>`.
Each `<PACKAGE>/<ARCH>` has two targets that match the usual structure of a
Debian package: `data` and `control`.
You can use the package like so: `@<REPO>//<PACKAGE>/<ARCH>:<TARGET>`.
This macro will expand to two repositories: `#name` and `#name#_resolve`.
E.g. for the previous example, you could use `@bullseye//perl/amd64:data`.
### Typical Workflow
### Lockfiles
A typical workflow for `deb_index` involves the generation of a lockfile (`deb_resolve`) and the consumption of the lockfile by `deb_package_index` for generating a DAG.
As mentioned, the macro can be used without a lock because the lock will be
generated internally on-demand. However, this comes with the cost of
performing a new package resolution on repository cache misses.
#### Lockfile Generation
The lockfile can be generated by running `bazel run @bullseye//:lock`. This
will generate a `.lock.json` file of the same name and in the same path as
the YAML `manifest` file.
The lockfile generation can be on-demand by omitting the `lock` attribute. However, this comes with the cost of performing a new package resolution on repository cache misses.
If you explicitly want to run without a lock and avoid the warning messages
set the `nolock` argument to `True`.
While we strongly encourage users to check in the generated lockfile, it's not always possible because Debian repositories are rolling by default. Therefore, a lockfile generated today might not work later if the upstream repository removes or publishes a new version of a package.
### Best Practice: use snapshot archive URLs
If you explicitly want to run without a lock, set the `nolock` attribute to True to avoid the DEBUG messages.
While we strongly encourage users to check in the generated lockfile, it's
not always possible because Debian repositories are rolling by default.
Therefore, a lockfile generated today might not work later if the upstream
repository removes or publishes a new version of a package.
#### Snapshot repositories
To avoid this problems and increase the reproducibility it's recommended to
avoid using normal Debian mirrors and use snapshot archives instead.
##### Debian
Users can still use a `snapshot` repository and check in the generated lockfiles. This is possible because `snapshot.debian.org` is an immutable point-in-time snapshot of the upstream repositories, meaning packages never get deleted or updated in a specific snapshot.
Snapshot archives provide a way to access Debian package mirrors at a point
in time. Basically, it's a "wayback machine" that allows access to (almost)
all past and current packages based on dates and version numbers.
An example of this can be found at [/examples/debian_snapshot](/examples/debian_snapshot).
Debian has had snapshot archives for [10+
years](https://lists.debian.org/debian-announce/2010/msg00002.html). Ubuntu
began providing a similar service recently and has packages available since
March 1st 2023.
##### Ubuntu
Ubuntu also has a similar point-in-time snapshot service hosted at `snapshot.ubuntu.com`, which can be used similarly.
To use this services simply use a snapshot URL in the manifest. Here's two
examples showing how to do this for Debian and Ubuntu:
* [/examples/debian_snapshot](/examples/debian_snapshot)
* [/examples/ubuntu_snapshot](/examples/ubuntu_snapshot)
An example of this can be found at [/examples/ubuntu_snapshot](/examples/ubuntu_snapshot).
For more infomation, please check https://snapshot.debian.org and/or
https://snapshot.ubuntu.com.
Args:
name: name of the repository
manifest: label to a `manifest.yaml`
lock: label to a `lock.json`
nolock: bool, set to True if you explicitly want to run without a lock and avoid the DEBUG messages.
package_template: (EXPERIMENTAL!) a template file for generated BUILD files.
Available template replacement keys are: `{target_name}`, `{deps}`, `{urls}`, `{name}`, `{arch}`, `{sha256}`, `{repo_name}`
resolve_transitive: whether dependencies of dependencies should be resolved and added to the lockfile.
nolock: bool, set to True if you explicitly want to run without a lock
and avoid the DEBUG messages.
package_template: (EXPERIMENTAL!) a template file for generated BUILD
files. Available template replacement keys are:
`{target_name}`, `{deps}`, `{urls}`, `{name}`,
`{arch}`, `{sha256}`, `{repo_name}`
resolve_transitive: whether dependencies of dependencies should be
resolved and added to the lockfile.
"""
_deb_resolve(
name = name + "_resolve",
Expand Down
Loading

0 comments on commit a555377

Please sign in to comment.