diff --git a/dev/search/search_index.json b/dev/search/search_index.json
index d58e0501..a0c03551 100644
--- a/dev/search/search_index.json
+++ b/dev/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"#rattler-build-a-fast-conda-package-builder","title":"rattler-build
: A Fast Conda Package Builder","text":"The rattler-build
tooling and library creates cross-platform relocatable binaries / packages from a simple recipe format. The recipe format is heavily inspired by conda-build
and boa
, and the output of a regular rattler-build
run is a package that can be installed using mamba
, rattler
or conda
.
rattler-build
does not have any dependencies on conda-build
or Python and works as a standalone binary.
"},{"location":"#installation","title":"Installation","text":"The recommended way of installing rattler-build
, being a conda-package builder, is through a conda package manager. Next to rattler-build
we are also building pixi
.
With pixi
you can install rattler-build
globally:
pixi global install rattler-build\n
Other options are:
CondaHomebrewArch LinuxBinary conda install rattler-build -c conda-forge\n\nmamba install rattler-build -c conda-forge\nmicromamba install rattler-build -c conda-forge\n\npixi global install rattler-build\npixi add rattler-build # To a pixi project\n
brew install rattler-build\n
pacman -S rattler-build\n
# Download the latest release from the GitHub releases page, for example the linux x86 version with curl:\ncurl -SL --progress-bar https://github.com/prefix-dev/rattler-build/releases/latest/download/rattler-build-x86_64-unknown-linux-musl\n
You can grab version of rattler-build
from the Github Releases."},{"location":"#completion","title":"Completion","text":"When installing rattler-build
you might want to enable shell completion. Afterwards, restart the shell or source the shell config file.
"},{"location":"#bash-default-on-most-linux-systems","title":"Bash (default on most Linux systems)","text":"echo 'eval \"$(rattler-build completion --shell bash)\"' >> ~/.bashrc\n
"},{"location":"#zsh-default-on-macos","title":"Zsh (default on macOS)","text":"echo 'eval \"$(rattler-build completion --shell zsh)\"' >> ~/.zshrc\n
"},{"location":"#powershell-pre-installed-on-all-windows-systems","title":"PowerShell (pre-installed on all Windows systems)","text":"Add-Content -Path $PROFILE -Value '(& rattler-build completion --shell powershell) | Out-String | Invoke-Expression'\n
Failure because no profile file exists
Make sure your profile file exists, otherwise create it with:
New-Item -Path $PROFILE -ItemType File -Force\n
"},{"location":"#fish","title":"Fish","text":"echo 'rattler-build completion --shell fish | source' >> ~/.config/fish/config.fish\n
"},{"location":"#nushell","title":"Nushell","text":"Add the following to the end of your Nushell env file (find it by running $nu.env-path
in Nushell):
mkdir ~/.cache/rattler-build\nrattler-build completion --shell nushell | save -f ~/.cache/rattler-build/completions.nu\n
And add the following to the end of your Nushell configuration (find it by running $nu.config-path
):
use ~/.cache/rattler-build/completions.nu *\n
"},{"location":"#elvish","title":"Elvish","text":"echo 'eval (rattler-build completion --shell elvish | slurp)' >> ~/.elvish/rc.elv\n
"},{"location":"#dependencies","title":"Dependencies","text":"Currently rattler-build
needs some dependencies on the host system which are executed as subprocess. We plan to reduce the number of external dependencies over time by writing what we need in Rust to make rattler-build
fully self-contained.
tar
to unpack tarballs downloaded from the internet in a variety of formats. .gz
, .bz2
and .xz
are widely used and one might have to install the compression packages as well (e.g. gzip
, bzip2
, ...) patch
to patch source code after downloading install_name_tool
is necessary on macOS to rewrite the rpath
of shared libraries and executables to make it relative patchelf
is required on Linux to rewrite the rpath
and runpath
of shared libraries and executables git
to checkout Git repositories (not implemented yet, but will require git
in the future) msvc
on Windows because we cannot ship the MSVC compiler on conda-forge (needs to be installed on the host machine) On Windows, to obtain these dependencies from conda-forge, one can install m2-patch
, m2-bzip2
, m2-gzip
, m2-tar
.
"},{"location":"#github-action","title":"GitHub Action","text":"There is a GitHub Action for rattler-build
. It can be used to install rattler-build
in CI/CD workflows and run a build command. Please check out the GitHub Action documentation for more information.
"},{"location":"#the-recipe-format","title":"The recipe format","text":"Note You can find all examples below in the examples
folder in the codebase and run them with rattler-build
.
A simple example recipe for the xtensor
header-only C++ library:
# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json\n\ncontext:\n name: xtensor\n version: 0.24.6\n\npackage:\n name: ${{ name|lower }}\n version: ${{ version }}\n\nsource:\n url: https://github.com/xtensor-stack/xtensor/archive/${{ version }}.tar.gz\n sha256: f87259b51aabafdd1183947747edfff4cff75d55375334f2e81cee6dc68ef655\n\nbuild:\n number: 0\n script:\n - if: win\n then: |\n cmake -G \"NMake Makefiles\" -D BUILD_TESTS=OFF -D CMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% %SRC_DIR%\n nmake\n nmake install\n else: |\n cmake ${CMAKE_ARGS} -DBUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$PREFIX $SRC_DIR -DCMAKE_INSTALL_LIBDIR=lib\n make install\n\nrequirements:\n build:\n - ${{ compiler('cxx') }}\n - cmake\n - if: unix\n then: make\n host:\n - xtl >=0.7,<0.8\n run:\n - xtl >=0.7,<0.8\n run_constraints:\n - xsimd >=8.0.3,<10\n\ntests:\n - script:\n - if: unix or emscripten\n then:\n - test -d ${PREFIX}/include/xtensor\n - test -f ${PREFIX}/include/xtensor/xarray.hpp\n - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfig.cmake\n - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfigVersion.cmake\n - if: win\n then:\n - if not exist %LIBRARY_PREFIX%\\include\\xtensor\\xarray.hpp (exit 1)\n - if not exist %LIBRARY_PREFIX%\\share\\cmake\\xtensor\\xtensorConfig.cmake (exit 1)\n - if not exist %LIBRARY_PREFIX%\\share\\cmake\\xtensor\\xtensorConfigVersion.cmake (exit 1)\n\nabout:\n homepage: https://github.com/xtensor-stack/xtensor\n license: BSD-3-Clause\n license_file: LICENSE\n summary: The C++ tensor algebra library\n description: Multi dimensional arrays with broadcasting and lazy computing\n documentation: https://xtensor.readthedocs.io\n repository: https://github.com/xtensor-stack/xtensor\n\nextra:\n recipe-maintainers:\n - some-maintainer\n
A recipe for the rich
Python package (using noarch
):
context:\n version: \"13.4.2\"\n\npackage:\n name: \"rich\"\n version: ${{ version }}\n\nsource:\n - url: https://pypi.io/packages/source/r/rich/rich-${{ version }}.tar.gz\n sha256: d653d6bccede5844304c605d5aac802c7cf9621efd700b46c7ec2b51ea914898\n\nbuild:\n # Thanks to `noarch: python` this package works on all platforms\n noarch: python\n script:\n - python -m pip install . -vv\n\nrequirements:\n host:\n - pip\n - poetry-core >=1.0.0\n - python 3.10\n run:\n # sync with normalized deps from poetry-generated setup.py\n - markdown-it-py >=2.2.0\n - pygments >=2.13.0,<3.0.0\n - python 3.10\n - typing_extensions >=4.0.0,<5.0.0\n\ntests:\n - python:\n imports:\n - rich\n pip_check: true\n\nabout:\n homepage: https://github.com/Textualize/rich\n license: MIT\n license_file: LICENSE\n summary: Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal\n description: |\n Rich is a Python library for rich text and beautiful formatting in the terminal.\n\n The Rich API makes it easy to add color and style to terminal output. Rich\n can also render pretty tables, progress bars, markdown, syntax highlighted\n source code, tracebacks, and more \u2014 out of the box.\n documentation: https://rich.readthedocs.io\n repository: https://github.com/Textualize/rich\n
A recipe for the curl
library:
context:\n version: \"8.0.1\"\n\npackage:\n name: curl\n version: ${{ version }}\n\nsource:\n url: http://curl.haxx.se/download/curl-${{ version }}.tar.bz2\n sha256: 9b6b1e96b748d04b968786b6bdf407aa5c75ab53a3d37c1c8c81cdb736555ccf\n\nbuild:\n number: 0\n\nrequirements:\n build:\n - ${{ compiler('c') }}\n - if: win\n then:\n - cmake\n - ninja\n - if: unix\n then:\n - make\n - perl\n - pkg-config\n - libtool\n host:\n - if: linux\n then:\n - openssl\n\nabout:\n homepage: http://curl.haxx.se/\n license: MIT/X derivate (http://curl.haxx.se/docs/copyright.html)\n license_file: COPYING\n summary: tool and library for transferring data with URL syntax\n description: |\n Curl is an open source command line tool and library for transferring data\n with URL syntax. It is used in command lines or scripts to transfer data.\n documentation: https://curl.haxx.se/docs/\n repository: https://github.com/curl/curl\n
For the curl
library recipe, two additional script files (build.sh
and build.bat
) are needed.
build.sh
#!/bin/bash\n\n# Get an updated config.sub and config.guess\ncp $BUILD_PREFIX/share/libtool/build-aux/config.* .\n\nif [[ $target_platform =~ linux.* ]]; then\n USESSL=\"--with-openssl=${PREFIX}\"\nelse\n USESSL=\"--with-secure-transport\"\nfi;\n\n./configure \\\n --prefix=${PREFIX} \\\n --host=${HOST} \\\n ${USESSL} \\\n --with-ca-bundle=${PREFIX}/ssl/cacert.pem \\\n --disable-static --enable-shared\n\nmake -j${CPU_COUNT} ${VERBOSE_AT}\nmake install\n\n# Includes man pages and other miscellaneous.\nrm -rf \"${PREFIX}/share\"\n
build.bat
mkdir build\n\ncmake -GNinja ^\n -DCMAKE_BUILD_TYPE=Release ^\n -DBUILD_SHARED_LIBS=ON ^\n -DCMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% ^\n -DCMAKE_PREFIX_PATH=%LIBRARY_PREFIX% ^\n -DCURL_USE_SCHANNEL=ON ^\n -DCURL_USE_LIBSSH2=OFF ^\n -DUSE_ZLIB=ON ^\n -DENABLE_UNICODE=ON ^\n %SRC_DIR%\n\nIF %ERRORLEVEL% NEQ 0 exit 1\n\nninja install --verbose\n
"},{"location":"authentication_and_upload/","title":"Server authentication","text":""},{"location":"authentication_and_upload/#authenticating-with-a-server","title":"Authenticating with a server","text":"You may want to use private channels for which you need to be authenticated. To do this ephemerally you can use the RATTLER_AUTH_FILE
environment variable to point to a JSON file with the following structure:
{\n \"*.prefix.dev\": {\n \"BearerToken\": \"your_token\"\n },\n \"otherhost.com\": {\n \"BasicHTTP\": {\n \"username\": \"your_username\",\n \"password\": \"your_password\"\n }\n },\n \"anaconda.org\": {\n \"CondaToken\": \"your_token\"\n }\n}\n
The keys are the host names. You can use wildcard specifiers here (e.g. *.prefix.dev
to match all subdomains of prefix.dev
, such as repo.prefix.dev
). This will allow you to also obtain packages from any private channels that you have access to.
The following known authentication methods are supported:
BearerToken
: prefix.dev CondaToken
: anaconda.org, quetz BasicHTTP
: artifactory "},{"location":"authentication_and_upload/#uploading-packages","title":"Uploading packages","text":"If you want to upload packages, then rattler-build comes with a built-in upload
command. There are 4 options:
prefix.dev
: you can create public or private channels on the prefix.dev hosted server anaconda.org
: you can upload packages to the free anaconda.org server quetz
: you can host your own quetz server and upload packages to it artifactory
: you can upload packages to a JFrog Artifactory server The command is:
rattler-build upload <server> <package_files>\n
Note: you can also use the RATTLER_AUTH_FILE
environment variable to authenticate with the server.
"},{"location":"authentication_and_upload/#prefixdev","title":"prefix.dev","text":""},{"location":"authentication_and_upload/#trusted-publishing-via-oidc","title":"Trusted publishing via OIDC","text":"rattler-build
supports authentication with https://prefix.dev through OIDC with GitHub Actions. An API key is no longer required, rattler-build can manage the complete authentication workflow for you. You only have to set up a specific repository and workflow under \"Trusted Publishers\" on prefix.dev.
Here you can find an example GitHub Actions workflow
.github/workflows/build.ymlpermissions:\n contents: read\n id-token: write\n\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - name: Build conda package\n uses: prefix-dev/rattler-build-action@v0.2.19\n\n - name: Upload all packages\n shell: bash\n run: |\n shopt -s nullglob\n EXIT_CODE=0\n for pkg in $(find output -type f \\( -name \"*.conda\" -o -name \"*.tar.bz2\" \\) ); do\n if ! rattler-build upload prefix -c my-channel \"${pkg}\"; then\n EXIT_CODE=1\n fi\n done\n exit $EXIT_CODE\n
"},{"location":"authentication_and_upload/#token","title":"Token","text":"To upload to prefix.dev, you need to have an account. You can then create a token in the settings of your account. The token is used to authenticate the upload.
export PREFIX_API_KEY=<your_token>\nrattler-build upload prefix -c <channel> <package_files>\n
You can also use the --api-key=$PREFIX_API_KEY
option to pass the token directly to the command. Note that you need to have created the channel on the prefix.dev website before you can upload to it.
"},{"location":"authentication_and_upload/#quetz","title":"Quetz","text":"You need to pass a token and API key to upload to a channel on your own Quetz server. The token is used to authenticate the upload.
export QUETZ_API_KEY=<your_token>\nrattler-build upload quetz -u <url> -c <channel> <package_files>\n
"},{"location":"authentication_and_upload/#artifactory","title":"Artifactory","text":"To upload to an Artifactory server, you need to pass a username and password. The username and password are used to authenticate the upload.
export ARTIFACTORY_USERNAME=<your_username>\nexport ARTIFACTORY_PASSWORD=<your_password>\nrattler-build upload artifactory -u <url> -c <channel> <package_files>\n
"},{"location":"authentication_and_upload/#anacondaorg","title":"anaconda.org","text":"To upload to anaconda.org, you need to specify the owner and API key. The API key is used to authenticate the upload.
The owner is the owner of the distribution, for example, your user name or organization.
One can also specify a label such as dev
for release candidates using the -c
flag. The default value is main
.
You can also add the --force
argument to forcibly upload a new package (and overwrite any existing ones).
export ANACONDA_API_KEY=<your_token>\nrattler-build upload anaconda -o <your_username> -c <label> <package_files>\n
"},{"location":"automatic_linting/","title":"Enabling Automatic Linting in VSCode","text":"Our new recipe format adheres to a strict JSON schema, which you can access here.
This schema is implemented using pydantic
and can be rendered into a JSON schema file. The YAML language server extension in VSCode is capable of recognizing this schema, providing useful hints during the editing process.
To enable automatic linting with the YAML language server, you need to add the following line at the beginning of your recipe file:
# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json\n
Alternatively, if you prefer not to add this line to your file, you can install the JSON Schema Store Catalog extension. This extension will also enable automatic linting for your recipe files.
"},{"location":"build_options/","title":"Advanced build options","text":"There are some specialized build options to control various features:
prefix replacement variant configuration encoded file type These are all found under the build
key in the recipe.yaml
.
"},{"location":"build_options/#include-only-certain-files-in-the-package","title":"Include only certain files in the package","text":"Sometimes you may want to include only a subset of the files installed by the build process in your package. For this, the files
key can be used. Only new files are considered for inclusion (ie. files that were not in the host environment beforehand).
recipe.yamlbuild:\n # select files to be included in the package\n # this can be used to remove files from the package, even if they are installed in the\n # environment\n files: list of globs\n
For example, to only include the header files in a package, you could use:
recipe.yamlbuild:\n files:\n - include/**/*.h\n
Glob patterns throughout the recipe file can also use a flexible include
/ exclude
pair, such as:
recipe.yamlbuild:\n files:\n include:\n - include/**/*.h\n exclude:\n - include/**/private.h\n
"},{"location":"build_options/#glob-evaluation","title":"Glob evaluation","text":"Glob patterns are used throughout the build options to specify files. The patterns are matched against the relative path of the file in the build directory. Patterns can contain *
to match any number of characters, ?
to match a single character, and **
to match any number of directories.
For example:
*.txt
matches all files ending in .txt
**/*.txt
matches all files ending in .txt
in any directory **/test_*.txt
matches all files starting with test_
and ending in .txt
in any directory foo/
matches all files under the foo
directory The globs are always evaluated relative to the prefix directory. If you have no include
globs, but an exclude
glob, then all files are included except those that match the exclude
glob. This is equivalent to include: ['**']
.
"},{"location":"build_options/#always-include-and-always-copy-files","title":"Always include and always copy files","text":"There are some options that control the inclusion of files in the final package.
The always_include_files
option can be used to include files even if they are already in the environment as part of some other host dependency. This is normally \"clobbering\" and should be used with caution (since packages should not have any overlapping files).
The always_copy_files
option can be used to copy files instead of linking them. This is useful for files that might be modified inside the environment (e.g. configuration files). Normally, files are linked from a central cache into the environment to save space \u2013 that means that files modified in one environment will be modified in all environments. This is not always desirable, and in that case you can use the always_copy_files
option.
??? note \"How always_copy_files
works\" The always_copy_files
option works by setting the no_link
option in the info/paths.json
to true
for the files in question. This means that the files are copied instead of linked when the package is installed.
recipe.yamlbuild:\n # include files even if they are already in the environment\n # as part of some other host dependency\n always_include_files: list of globs\n\n # do not soft- or hard-link these files, but always copy them was `no_link`\n always_copy_files: list of globs\n
"},{"location":"build_options/#merge-build-and-host-environments","title":"Merge build and host environments","text":"In very rare cases you might want to merge the build and host environments to obtain the \"legacy\" behavior of conda-build.
recipe.yamlbuild:\n # merge the build and host environments (used in many R packages on Windows)\n merge_build_and_host_envs: bool (defaults to false)\n
"},{"location":"build_options/#prefix-detection-replacement-options","title":"Prefix detection / replacement options","text":"During installation time the \"install\"-prefix is injected into text and binary files. Sometimes this is not desired, and sometimes the user might want closer control over the automatic text/binary detection.
The main difference between prefix replacement for text and binary files is that for binary files, the prefix string is padded with null bytes to match the length of the original prefix. The original prefix is the very long placeholder string that you might have seen in the build process.
On Windows, binary prefix replacement is never performed.
recipe.yamlpackage:\n name: mypackage\n version: 1.0\n\nbuild:\n # settings concerning the prefix detection in files\n prefix_detection:\n # force the file type of the given files to be TEXT or BINARY\n # for prefix replacement\n force_file_type:\n # force TEXT file type (list of globs)\n text: list of globs\n # force binary file type (list of globs)\n binary: list of globs\n\n # ignore all or specific files for prefix replacement`\n ignore: bool | [path] (defaults to false)\n\n # whether to detect binary files with prefix or not\n # defaults to true on Unix and (always) false on Windows\n ignore_binary_files: bool\n
"},{"location":"build_options/#variant-configuration","title":"Variant configuration","text":"To control the variant precisely you can use the \"variant configuration\" options.
A variant package has the same version number, but different \"hash\" and potentially different dependencies or build options. Variant keys are extracted from the variant_config.yaml
file and usually any used Jinja variables or dependencies without version specifier are used as variant keys.
Variant keys can also be forcibly set or ignored with the use_keys
and ignore_keys
options.
In order to decide which of the variant packages to prefer and install by default, the down_prioritize_variant
option can be used. The higher the value, the less preferred the variant is.
More about variants can be found in the variant documentation.
The following options are available in the build
section to control the variant configuration:
recipe.yamlbuild:\n # settings for the variant\n variant:\n # Keys to forcibly use for the variant computation\n # even if they are not in the dependencies\n use_keys: list of strings\n\n # Keys to forcibly ignore for the variant computation\n # even if they are in the dependencies\n ignore_keys: list of strings\n\n # used to prefer this variant less\n down_prioritize_variant: integer (defaults to 0, higher is less preferred)\n
"},{"location":"build_options/#dynamic-linking-configuration","title":"Dynamic linking configuration","text":"After the package is built, rattler-build performs some \"post-processing\" on the binaries and libraries.
This entails making the shared libraries relocatable and checking that all linked libraries are present in the run requirements. The following settings control this behavior.
With the rpath
option you can forcibly set the rpath
of the shared libraries. The path is relative to the install prefix. Any rpath
setting is ignored on Windows.
The rpath_allowlist
option can be used to allow the rpath
to point to locations outside of the environment. This is useful if you want to link against libraries that are not part of the conda environment (e.g. proprietary software).
If you want to stop rattler-build
from relocating the binaries, you can set binary_relocation
to false
. If you want to only relocate some binaries, you can select the relevant ones with a glob pattern.
To read more about rpath
s and how rattler-build creates relocatable binary packages, see the internals docs.
If you link against some libraries (possibly even outside of the prefix, in a system location), then you can use the missing_dso_allowlist
to allow linking against these and suppress any warnings. This list is pre-populated with a list of known system libraries on the different operating systems.
As part of the post-processing, rattler-build
checks for overlinking and overdepending. \"Overlinking\" is when a binary links against a library that is not specified in the run requirements. This is usually a mistake because the library would not be present in the environment when the package is installed.
Conversely, \"overdepending\" is when a library is part of the run requirements, but is not actually used by any of the binaries/libraries in the package.
recipe.yamlbuild:\n # settings for shared libraries and executables\n dynamic_linking:\n # linux only, list of rpaths relative to the installation prefix\n rpaths: list of paths (defaults to ['lib/'])\n\n # Allow runpath / rpath to point to these locations\n # outside of the environment\n rpath_allowlist: list of globs\n\n # whether to relocate binaries or not. If this is a list of paths, then\n # only the listed paths are relocated\n binary_relocation: bool (defaults to true) | list of globs\n\n # Allow linking against libraries that are not in the run requirements\n missing_dso_allowlist: list of globs\n\n # what to do when detecting overdepending\n overdepending_behavior: \"ignore\" or \"error\" # (defaults to \"error\")\n\n # what to do when detecting overlinking\n overlinking_behavior: \"ignore\" or \"error\" # (defaults to \"error\")\n
"},{"location":"build_script/","title":"Build scripts","text":"The build.sh
file is the build script for Linux and macOS and build.bat
is the build script for Windows. These scripts contain the logic that carries out your build steps. Anything that your build script copies into the $PREFIX
or %PREFIX%
folder will be included in your output package.
For example, this build.sh
:
build.shmkdir -p $PREFIX/bin\ncp $RECIPE_DIR/my_script_with_recipe.sh $PREFIX/bin/super-cool-script.sh\n
There are many environment variables defined for you to use in build.sh and build.bat. Please see environment variables for more information.
build.sh
and build.bat
are optional. You can instead use the build/script
key in your recipe.yaml
, with each value being either a string command or a list of string commands. Any commands you put there must be able to run on every platform for which you build. For example, you can't use the cp
command because cmd.exe
won't understand it on Windows.
build.sh
is run with bash
and build.bat
is run with cmd.exe
.
recipe.yamlbuild:\n script:\n - if: unix\n then:\n - mkdir -p $PREFIX/bin\n - cp $RECIPE_DIR/my_script_with_recipe.sh $PREFIX/bin/super-cool-script.sh\n - if: win\n then:\n - mkdir %PREFIX%\\bin\n - copy %RECIPE_DIR%\\my_script_with_recipe.bat %PREFIX%\\bin\\super-cool-script.bat\n
"},{"location":"build_script/#environment-variables","title":"Environment variables","text":"There are many environment variables that are automatically set during the build process.
However, you can also set your own environment variables easily in the script
section of your recipe:
recipe.yamlbuild:\n script:\n # Either use `content` or `file` to specify the script\n # Note: this script only works on Unix :)\n content: |\n echo $FOO\n echo $BAR\n echo \"Secret value: $BAZ\"\n env:\n # hard coded value for `FOO`\n FOO: \"foo\"\n # Forward a value from the \"outer\" environment\n # Without `default=...`, the build process will error if `BAR` is not set\n BAR: ${{ env.get(\"BAR\", default=\"NOBAR\") }}\n secrets:\n # This value is a secret and will be masked in the logs and not stored in the rendered recipe\n # The value needs to be available as an environment variable in the outer environment\n - BAZ\n
"},{"location":"build_script/#alternative-script-interpreters","title":"Alternative script interpreters","text":"With rattler-build
and the new recipe syntax you can select an interpreter
for your script.
So far, the following interpreters are supported:
bash
(default on Unix) cmd.exe
(default on Windows) nushell
python
Note
Using alternative interpreters is less battle-tested than using bash
or cmd.exe
. If you encounter any issues, please open an issue.
"},{"location":"build_script/#using-nushell","title":"Using nushell
","text":"In order to use nushell
you can select the interpreter: nu
or have a build.nu
file in your recipe directory. Nushell works on Windows, macOS and Linux with the same syntax.
recipe.yamlbuild:\n script:\n interpreter: nu\n content: |\n echo \"Hello from nushell!\"\n\n# Note: it's required to have `nushell` in the `build` section of your recipe!\nrequirements:\n build:\n - nushell\n
"},{"location":"build_script/#using-python","title":"Using python
","text":"In order to use python
you can select the interpreter: python
or have a build.py
file in your recipe directory and python
in the requirements/build
section.
recipe.yamlbuild:\n script:\n interpreter: python\n content: |\n print(\"Hello from Python!\")\n\n# Note: it's required to have `python` in the `build` section of your recipe!\nrequirements:\n build:\n - python\n
"},{"location":"build_script/#default-environment-variables-set-during-the-build-process","title":"Default environment variables set during the build process","text":"During the build process, the following environment variables are set, on Windows with build.bat
and on macOS and Linux with build.sh
. By default, these are the only variables available to your build script. Unless otherwise noted, no variables are inherited from the shell environment in which you invoke conda-build
. To override this behavior, see :ref:inherited-env-vars
.
ARCH
Either 32
or 64
, to specify whether the build is 32-bit or 64-bit. The value depends on the ARCH environment variable and defaults to the architecture the interpreter running conda was compiled with.
CMAKE_GENERATOR
The CMake generator string for the current build environment. On Linux systems, this is always Unix Makefiles
. On Windows, it is generated according to the Visual Studio version activated at build time, for example, Visual Studio 9 2008 Win64
.
CONDA_BUILD=1
Always set to indicate that the conda-build process is running.
CPU_COUNT
Represents the number of CPUs on the system.
SHLIB_EXT
Denotes the shared library extension specific to the operating system (e.g. .so
for Linux, .dylib
for macOS, and .dll
for Windows).
HTTP_PROXY
, HTTPS_PROXY
Inherited from the user's shell environment, specifying the HTTP and HTTPS proxy settings.
LANG
Inherited from the user's shell environment, defining the system language and locale settings.
MAKEFLAGS
Inherited from the user's shell environment. This can be used to set additional arguments for the make command, such as -j2 to utilize 2 CPU cores for building the recipe.
PY_VER
Specifies the Python version against which the build is occurring. This can be modified with a variant_config.yaml
file.
PATH
Inherited from the user's shell environment and augmented with the activated host and build prefixes.
PREFIX
The build prefix to which the build script should install the software.
PKG_BUILDNUM
Indicates the build number of the package currently being built.
PKG_NAME
The name of the package that is being built.
PKG_VERSION
The version of the package currently under construction.
PKG_BUILD_STRING
The complete build string of the package being built, including the hash (e.g. py311h21422ab_0).
PKG_HASH
Represents the hash of the package being built, excluding the leading 'h' (e.g. 21422ab). This is applicable from conda-build 3.0 onwards.
PYTHON
The path to the Python executable in the host prefix. Python is installed in the host prefix only when it is listed as a host requirement.
R
The path to the R executable in the build prefix. R is installed in the build prefix only when it is listed as a build requirement.
RECIPE_DIR
The directory where the recipe is located.
SP_DIR
The location of Python's site-packages, where Python libraries are installed.
SRC_DIR
The path to where the source code is unpacked or cloned. If the source file is not a recognized archive format, this directory contains a copy of the source file.
STDLIB_DIR
The location of Python's standard library.
build_platform
Represents the native subdirectory of the conda executable, indicating the platform for which the build is occurring.
Removed from conda-build
are: - NPY_VER
- PY3K
"},{"location":"build_script/#windows","title":"Windows","text":"Unix-style packages on Windows are built in a special Library
directory under the build prefix. The environment variables listed in the following table are defined only on Windows.
Variable Description LIBRARY_BIN
<build prefix>\\Library\\bin
. LIBRARY_INC
<build prefix>\\Library\\include
. LIBRARY_LIB
<build prefix>\\Library\\lib
. LIBRARY_PREFIX
<build prefix>\\Library
. SCRIPTS
<build prefix>\\Scripts
. Not yet supported in rattler-build
:
CYGWIN_PREFIX
VS_MAJOR
VS_VERSION
VS_YEAR
Additionally, the following variables are forwarded from the environment:
ALLUSERSPROFILE
APPDATA
CommonProgramFiles
CommonProgramFiles(x86)
CommonProgramW6432
COMPUTERNAME
ComSpec
HOMEDRIVE
HOMEPATH
LOCALAPPDATA
LOGONSERVER
NUMBER_OF_PROCESSORS
PATHEXT
ProgramData
ProgramFiles
ProgramFiles(x86)
ProgramW6432
PROMPT
PSModulePath
PUBLIC
SystemDrive
SystemRoot
TEMP
TMP
USERDOMAIN
USERNAME
USERPROFILE
windir
PROCESSOR_ARCHITEW6432
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
"},{"location":"build_script/#unix","title":"Unix","text":"The environment variables listed in the following table are defined only on macOS and Linux.
Variable Description HOME
Standard $HOME environment variable. PKG_CONFIG_PATH
Path to pkgconfig
directory, defaults to `$PREFIX/lib/pkgconfig SSL_CERT_FILE
Path to SSL_CERT_FILE
file. CFLAGS
Empty, can be forwarded from env to set additional arguments to C compiler. CXXFLAGS
Same as CFLAGS for C++ compiler. LDFLAGS
Empty, additional flags to be passed to the linker when linking object files into an executable or shared object."},{"location":"build_script/#macos","title":"macOS","text":"The environment variables listed in the following table are defined only on macOS.
Variable Description MACOSX_DEPLOYMENT_TARGET
Same as the Anaconda Python macOS deployment target. Currently 10.9
for intel 32- and 64bit macOS, and 11.0 for arm64. OSX_ARCH
i386
or x86_64
or arm64
, depending on the target platform"},{"location":"build_script/#linux","title":"Linux","text":"The environment variable listed in the following table is defined only on Linux.
Variable Description LD_RUN_PATH
Defaults to <build prefix>/lib
. QEMU_LD_PREFIX
The prefix used by QEMU's user mode emulation for library paths. QEMU_UNAME
Set qemu uname release string to 'uname'. DEJAGNU
The path to the dejagnu testing framework used by the GCC test suite. DISPLAY
The X11 display to use for graphical applications. BUILD
Target triple ({build_arch}-conda_{build_distro}-linux-gnu
) where build_distro is one of cos6
or cos7
, for Centos 6 or 7"},{"location":"compilers/","title":"Compilers and cross-compilation","text":"To use a compiler in your project, it's best to use the ${{ compiler('lang') }}
template function. The compiler function works by taking a language, determining the configured compiler for that language, and adding some information about the target platform to the selected compiler. To configure a compiler for a specific language, the variant_config.yaml
file can be used.
For example, in a recipe that uses a C-compiler, you can use the following code:
requirements:\n build:\n - ${{ compiler('c') }}\n
To set the compiler that you want to use, create a variant config that looks like the following:
c_compiler:\n - gcc\n\n# optionally you can specify a version\nc_compiler_version:\n - 9.3.0\n
When the template function is evaluated, it will look something like: gcc_linux-64 9.3.0
. You can define your own compilers. For example, for Rust you can use ${{ compiler('rust') }}
and rust_compiler_{version}
in your variant config.
"},{"location":"compilers/#cross-compilation","title":"Cross-compilation","text":"Cross-compilation is supported by rattler-build
and the compiler template function is part of what makes it possible. When you want to cross-compile from linux-64
to linux-aarch64
(i.e. intel to ARM), you can pass --target-platform linux-aarch64
to the rattler-build
command. This will cause the compiler template function to select a compiler that is configured for linux-aarch64
. The above example would resolve to gcc_linux-aarch64 9.3.0
. Provided that the package is available for linux-64
(your build platform), the compilation should succeed.
The distinction between the build
and host
sections begins to make sense when thinking about cross-compilation. The build
environment is resolved to packages that need to run at compilation time. For example, cmake
, gcc
, and autotools
are all tools that need to be executed. Therefore, the build
environment resolves to packages for the linux-64
architecture (in our example). On the other hand, the host
packages resolve to linux-aarch64
- those are packages that we want to link against.
# packages that need to run at build time (cmake, gcc, autotools, etc.)\n# in the platform that rattler-build is executed on (the build_platform)\nbuild:\n - cmake\n - ${{ compiler('c') }}\n# packages that we want to link against in the architecture we are\n# cross-compiling to the target_platform\nhost:\n - libcurl\n - openssl\n
"},{"location":"converting_from_conda_build/","title":"Converting a recipe from conda-build","text":"The recipe format of rattler-build
differs in some aspects from conda-build
. This document aims to help you convert a recipe from conda-build
to rattler-build
.
"},{"location":"converting_from_conda_build/#automatic-conversion","title":"Automatic conversion","text":"To convert a recipe from meta.yaml
to recipe.yaml
you can use the automatic conversion utility.
To install conda-recipe-manager
, run
pixi global install conda-recipe-manager\n# or\nconda install -c conda-forge conda-recipe-manager\n
Then, run the conversion utility:
conda-recipe-manager convert my-recipe/meta.yaml\n
This will print the converted recipe to the console. You can save it to a file by redirecting the output:
conda-recipe-manager convert my-recipe/meta.yaml > recipe.yaml\n
To learn more about the tool, or contribute, find the repository here.
"},{"location":"converting_from_conda_build/#converting-jinja-and-selectors","title":"Converting Jinja and selectors","text":"To use jinja
in the new recipes, you need to keep in mind two conversions. The {% set version = \"1.2.3\" %}
syntax is replaced by the context
section in the new recipe format.
{% set version = \"1.2.3\" %}\n
becomes
context:\n version: \"1.2.3\"\n
To use the values or other Jinja expressions (e.g. from the variant config) you can use the ${{ version }}
syntax. Note the $
sign before the curly braces - it makes Jinja fully compatible with the YAML format.
meta.yaml# instead of\npackage:\n version: \"{{ version }}\"\nsource:\n url: https://example.com/foo-{{ version }}.tar.gz\n
becomes
recipe.yamlpackage:\n version: ${{ version }}\nsource:\n url: https://example.com/foo-${{ version }}.tar.gz\n
"},{"location":"converting_from_conda_build/#converting-selectors","title":"Converting selectors","text":"conda-build
has a line based \"selector\" system, to e.g. disable certain fields on Windows vs. Unix.
In rattler-build we\u00a0use two different syntaxes: an if/else/then
map or a inline jinja expression.
A typical selector in conda-build
looks something like this:
meta.yamlrequirements:\n host:\n - pywin32 # [win]\n
To convert this to rattler-build
syntax, you can use one of the following two syntaxes:
recipe.yamlrequirements:\n host:\n - ${{ \"pywin32\" if win }} # empty strings are automatically filtered\n # or\n - if: win\n then:\n - pywin32 # this list extends the outer list\n
"},{"location":"converting_from_conda_build/#converting-the-recipe-script","title":"Converting the recipe script","text":"We still support the build.sh
script, but the bld.bat
script was renamed to build.bat
in order to be more consistent with the build.sh
script.
You can also choose a different name for your script:
build:\n # note: if there is no extension, we will try to find .sh on unix and .bat on windows\n script: my_build_script\n
There are also new ways of writing scripts, for example with nushell
or python
Variant keys in build scripts
conda-build
tries to analyze the build scripts for any usage of variant keys. We do not attempt that. If you want to use variant keys in your build script that are not used anywhere else you need to manually add them to your script environment, e.g.
recipe.yamlbuild:\n script:\n content: echo $MY_VARIANT\n env:\n MY_VARIANT: ${{ my_variant }}\n
"},{"location":"converting_from_conda_build/#converting-the-recipe-structure","title":"Converting the recipe structure","text":"There are a few differences in the recipe structure. However, the schema will tell you quite easily what is expected and you should see red squiggly lines in your editor (e.g. VSCode) if you make a mistake.
Here are a few differences:
"},{"location":"converting_from_conda_build/#converting-the-test-section","title":"Converting the test section","text":"The test
section is renamed to tests
and is a list of independent tests. Each test runs in its own environment.
Let's have a look at converting an existing test section:
meta.yamltest:\n imports:\n - mypackage\n commands:\n - mypackage --version\n
This would now be split into two tests:
recipe.yamltests:\n - script:\n - mypackage --version\n - python:\n imports:\n - mypackage\n # by default we perform a `pip check` in the python test but\n # it can be disabled by setting this to false\n pip_check: false\n
The script
tests also take a requirements
section with run
and build
requirements. The build
requirements can be used to install emulators and similar tools that need to run to execute tests in a cross-compilation environment.
"},{"location":"experimental_features/","title":"Experimental features","text":"Warning
These are experimental features of rattler-build
and may change or go away completely.
Currently only the build
and rebuild
commands support the following experimental features.
To enable them, use the --experimental
flag with the command. Or, use the environment variable, RATTLER_BUILD_EXPERIMENTAL=true
.
"},{"location":"experimental_features/#jinja-functions","title":"Jinja functions","text":""},{"location":"experimental_features/#load_from_filefile_path","title":"load_from_file(<file_path>)
","text":"The Jinja function load_from_file
allows loading from files; specifically, it allows loading from toml
, json
, and yaml
file types to an object to allow it to fetch things directly from the file. It loads all other files as strings.
"},{"location":"experimental_features/#usage","title":"Usage","text":"load_from_file
is useful when there is a project description in a well-defined project file such as Cargo.toml
, package.json
, pyproject.toml
, package.yaml
, or stack.yaml
. It enables the recipe to be preserved in as simple a state as possible, especially when there is no need to keep the changes in sync; some example use cases for this are with CI/CD infrastructure or when there is a well-defined output format.
Below is an example loading a Cargo.toml
inside of the rattler-build
GitHub repository:
recipe.yamlcontext:\n name: ${{ load_from_file(\"Cargo.toml\").package.name }}\n version: ${{ load_from_file(\"Cargo.toml\").package.version }}\n source_url: ${{ load_from_file(\"Cargo.toml\").package.homepage }}\n rust_toolchain: ${{ load_from_file(\"rust-toolchains\") }}\n\npackage:\n name: ${{ name }}\n version: ${{ version }}\n\nsource:\n git: ${{ source_url }}\n tag: ${{ source_tag }}}}\n\nrequirements:\n build:\n - rust ==${{ rust_toolchain }}\n\nbuild:\n script: cargo build --release -p ${{ name }}\n\ntest:\n - script: cargo test -p ${{ name }}\n - script: cargo test -p rust-test -- --test-threads=1\n\nabout:\n home: ${{ source_url }}\n repository: ${{ source_url }}\n documentation: ${{ load_from_file(\"Cargo.toml\").package.documentation }}\n summary: ${{ load_from_file(\"Cargo.toml\").package.description }}\n license: ${{ load_from_file(\"Cargo.toml\").package.license }}\n
"},{"location":"experimental_features/#git-functions","title":"git
functions","text":"git
functions are useful for getting the latest tag and commit hash. These can be used in the context
section of the recipe, to fetch version information from a repository.
Examples # latest tag in the repo\ngit.latest_tag(<git_repo_url>)\n\n# latest tag revision(aka, hash of tag commit) in the repo\ngit.latest_tag_rev(<git_repo_url>)\n\n# latest commit revision(aka, hash of head commit) in the repo\ngit.head_rev(<git_repo_url>)\n
"},{"location":"experimental_features/#usage_1","title":"Usage","text":"These can be useful for automating minor things inside of the recipe itself, such as if the current version is the latest version or if the current hash is the latest hash, etc.
recipe.yamlcontext:\n git_repo_url: \"https://github.com/prefix-dev/rattler-build\"\n latest_tag: ${{ git.latest_tag( git_repo_url ) }}\n\npackage:\n name: \"rattler-build\"\n version: ${{ latest_tag }}\n\nsource:\n git: ${{ git_repo_url }}\n tag: ${{ latest_tag }}\n
There is currently no guarantee of caching for repo fetches when using git
functions. This may lead to some performance issues.
"},{"location":"highlevel/","title":"What is rattler-build
?","text":"rattler-build
is a tool to build and package software so that it can be installed on any operating system \u2013 with any compatible package manager such as mamba
, conda
, or rattler
. We are also intending for rattler-build
to be used as a library to drive builds of packages from any other recipe format in the future.
"},{"location":"highlevel/#how-does-rattler-build-work","title":"How does rattler-build
work?","text":"Building of packages consists of several steps. It all begins with a recipe.yaml
file that specifies how the package is to be built and what the dependencies are. From the recipe file, rattler-build
executes several steps:
Rendering: Parse the recipe file and evaluate conditionals, Jinja expressions, and variables, and variants.
Fetch source: Retrieve specified source files, such as .tar.gz
files, git
repositories, local paths. Additionally, this step will apply patches that can be specified alongside the source file.
Install build environments: Download and install dependencies into temporary \"host\" and \"build\" workspaces. Any dependencies that are needed at build time are installed in this step.
Build source: Execute the build script to build/compile the source code and install it into the host environment.
Prepare package files: Collect all files that are new in the \"host\" environment and apply some transformations if necessary; specifically, we edit the rpath
on Linux
and macOS
to make binaries relocatable.
Package: Bundle all the files in a package and write out any additional metadata into the info/index.json
, info/about.json
, and info/paths.json
files. This also creates the test files that are bundled with the package.
Test: Run any tests specified in the recipe. The package is considered done if it passes all the tests, otherwise its moved to broken/
in the output directory.
After this process, a package is created. This package can be uploaded to somewhere like a custom prefix.dev private or public channel.
"},{"location":"highlevel/#how-to-run-rattler-build","title":"How to run rattler-build
","text":"Running rattler-build
is straightforward. It can be done on the command line:
rattler-build build --recipe myrecipe/recipe.yaml\n
A custom channel that is not conda-forge (the default) can be specified like so:
rattler-build build -c robostack --recipe myrecipe/recipe.yaml\n
You can also use the --recipe-dir
argument if you want to build all the packages in a directory:
rattler-build build --recipe-dir myrecipes/\n
"},{"location":"highlevel/#overview-of-a-recipeyaml","title":"Overview of a recipe.yaml
","text":"A recipe.yaml
file is separated into multiple sections and can conditionally include or exclude sections. Recipe files also support a limited amount of string interpolation with Jinja (specifically minijinja
in our case).
A simple example of a recipe file for the zlib
package would look as follows:
recipe.yaml# variables from the context section can be used in the rest of the recipe\n# in jinja expressions\ncontext:\n version: 1.2.13\n\npackage:\n name: zlib\n version: ${{ version }}\n\nsource:\n url: http://zlib.net/zlib-${{ version }}.tar.gz\n sha256: b3a24de97a8fdbc835b9833169501030b8977031bcb54b3b3ac13740f846ab30\n\nbuild:\n # build numbers can be set arbitrarily\n number: 0\n script:\n # build script to install the package into the $PREFIX (host prefix)\n - if: unix\n then:\n - ./configure --prefix=$PREFIX\n - make -j$CPU_COUNT\n - if: win\n then:\n - cmake -G \"Ninja\" -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=%LIBRARY_PREFIX%\n - ninja install\n\nrequirements:\n build:\n # compiler is a special function.\n - ${{ compiler(\"c\") }}\n # The following two dependencies are only needed on Windows,\n # and thus conditionally selected\n - if: win\n then:\n - cmake\n - ninja\n - if: unix\n then:\n - make\n
The sections of a recipe are:
sections description context
Defines variables that can be used in the Jinja context later in the recipe (e.g. name and version are commonly interpolated in strings) package
This section defines the name and version of the package you are currently building and will be the name of the final output source
Defines where the source code is going to be downloaded from and checksums build
Settings for the build and the build script requirements
Allows the definition of build, host, run and run-constrained dependencies"},{"location":"internals/","title":"What does rattler-build
do to build a package?","text":"rattler-build
creates conda packages which are relocatable packages. These packages are built up with some rules and conventions in mind.
"},{"location":"internals/#what-goes-into-a-package","title":"What goes into a package?","text":"Generally speaking, any new files that are copied into the $PREFIX
directory at build time are part of the new package. However, there is some filtering going on to exclude unwanted files, and noarch: python
packages have special handling as well. The rules are as follows:
"},{"location":"internals/#filtering","title":"Filtering","text":""},{"location":"internals/#general-file-filtering","title":"General File Filtering","text":"Certain files are filtered out to prevent them from being included in the package. These include:
.pyo files: Optimized Python files are not included because they are considered harmful. .la files: Libtool archive files that are not needed at runtime. .DS_Store files: macOS-specific files that are irrelevant to the package. .git files and directories: Version control files, including .gitignore
and the .git
directory, which are not needed in the package. share/info/dir This file is ignored because it would be written from multiple packages. "},{"location":"internals/#special-handling-for-noarch-python-packages","title":"Special Handling for noarch: python
Packages","text":"For packages marked as noarch: python
, special transformations are applied to ensure compatibility across different platforms:
Stripping Python Library Prefix: The \"lib/pythonX.X\" prefix is removed, retaining only the \"site-packages\" part of the path. Skipping __pycache__
Directories and .pyc
Files: These are excluded and recreated during installation (they are specific to the Python version). Replacing bin
and Scripts
Directories: On Unix systems, the bin
directory is replaced with python-scripts
. On Windows systems, the Scripts
directory is replaced with python-scripts
. Remove explicitly mentioned entrypoints: For noarch: python
packages, entry points registered in the package are also taken into account. Files in the bin
or Scripts
directories that match entry points are excluded to avoid duplications. "},{"location":"internals/#symlink-handling","title":"Symlink Handling","text":"Symlinks are carefully managed to ensure they are relative rather than absolute, which aids in making the package relocatable:
Absolute symlinks pointing within the $PREFIX
are converted to relative symlinks. On Unix systems, this conversion is handled directly by creating new relative symlinks. On Windows, a warning is issued since symlink creation requires administrator privileges. "},{"location":"internals/#making-packages-relocatable-with-rattler-build","title":"Making Packages Relocatable with rattler-build
","text":"Often, the most challenging aspect of building a package using rattler-build
is making it relocatable. A relocatable package can be installed into any prefix, allowing it to be used outside the environment in which it was built. This is in contrast to a non-relocatable package, which can only be utilized within its original build environment.
rattler-build
automatically performs the following actions to make packages relocatable:
Binary object file conversion: Binary object files are converted to use relative paths using install_name_tool
on macOS and patchelf
on Linux. This uses $ORIGIN
for elf files on Linux and @loader_path
for Mach-O files on macOS to make the rpath
relative to the executable / shared library. Text file prefix registration: Any text file without NULL
bytes containing the placeholder prefix have the registered prefix replaced with the install prefix. Binary file prefix detection and registration: Binary files containing the build prefix can be automatically registered. The registered files will have their build prefix replaced with the install prefix at install time. This works by padding the install prefix with null terminators, such that the length of the binary file remains the same. The build prefix must be long enough to accommodate any reasonable installation prefix. On macOS and Linux, rattler-build
pads the build prefix to 255 characters by appending _placehold
to the end of the build directory name. "},{"location":"multiple_output_cache/","title":"The cache for multiple outputs","text":"Note
The \"multi-output\" cache is a little bit different from a compilation cache. If you look for tips and tricks on how to use sccache
or ccache
with rattler-build
, please refer to the tips and tricks section.
Sometimes you build a package and want to split the contents into multiple sub-packages. For example, when building a C/C++ package, you might want to create multiple packages for the runtime requirements (library), and the development time requirements such as header files.
The \"cache\" output makes this easy. It allows you to specify a single top-level cache that can produce arbitrary files, that can then be used in other packages.
Let's take a look at an example:
recipe.yamlrecipe:\n name: mypackage\n version: '0.1.0'\n\ncache:\n requirements:\n build:\n - ${{ compiler('c') }}\n build:\n script:\n - mkdir -p $PREFIX/lib\n - mkdir -p $PREFIX/include\n - echo \"This is the library\" > lib/library.txt\n - echo \"This is the header\" > include/header.txt\n\noutputs:\n - package:\n name: mypackage-library\n build:\n files:\n - lib/*\n\n - package:\n name: mypackage-headers\n build:\n files:\n - include/*\n
Note
Since this is an experimental feature, you need to pass the --experimental
flag to enable parsing of the cache
top-level section.
In this example, we have a single package called mypackage
that creates two outputs: mypackage-library
and mypackage-headers
. The cache output will run like a regular output, but after the build is finished, the files will be copied to a \"cache\" directory (in your output folder, under output/build_cache
).
The files in the cache folder are then copied into the $PREFIX
of each output package. Since they are \"new\" files in the prefix, they will be included in the output package. The easiest way to select a subset of the files in the prefix is by using the files
field in the output definition. You can use a list of globs to select only the files that you want.
For something more complicated you can also use include
and exclude
fields in the files
selector. Please refer to the the build options documentation.
"},{"location":"multiple_output_cache/#run-exports-from-the-cache","title":"Run exports from the cache","text":"Since the cache output also has build- and host requirements we need to additionally take care of eventual \"run-exports\" from the cache output. Run exports from the cache-dependencies are handled very similar to the run exports from a given output. We append any run exports to the outputs.
If the cache has an \"ignore run exports\" section, than we apply those filters at the cache level. If the output ignores any run exports, then we also ignore the run-exports if they would come from the cache.
"},{"location":"multiple_output_cache/#caching-in-the-src_dir","title":"Caching in the $SRC_DIR","text":"If you used conda-build
a lot, you might have noticed that a top-level build is also caching the changes in the $SRC_DIR
. This is not the case for rattler-build
yet.
You could try to work around by e.g. copying files into the $PREFIX
and restoring them in each output.
"},{"location":"package_spec/","title":"Package specification","text":"rattler-build
produces \"conda\" packages. These packages work with the mamba
and conda
package managers, and they work cross-platform on Windows, Linux and macOS.
By default, a conda package is a tar.bz2
archive which contains:
Metadata under the info/
directory A collection of files that are installed directly into an install prefix The format is identical across platforms and operating systems. During the install process, all files are extracted into the install prefix, except the ones in info/
. Installing a conda package into an environment is similar to executing the following commands:
cd <environment prefix>\ntar xjf mypkg-1.0.0-h2134.tar.bz2\n
Only files, including symbolic links, are part of a conda package. Directories are not included. Directories are created and removed as needed, but you cannot create an empty directory from the tar archive directly.
There is also a newer archive type, suffixed with .conda
. This archive type consists of an outer \"zip\" archive that is not compressed, and two inner archives that are compressed with zstd
, which is very fast for decompression.
The inner archives are split into info
and pkg
files, which makes it possible to extract only the info
part of the archive (only the metadata), which is often smaller in size.
"},{"location":"package_spec/#package-filename","title":"Package filename","text":"A conda package conforms to the following filename:
<name>-<version>-<hash>.tar.bz2 OR <name>-<version>-<hash>.conda\n
"},{"location":"package_spec/#special-files-in-packages","title":"Special files in packages","text":"There are some special files in a package:
activation and deactivation scripts that are executed when the environment is activated or deactivated post-link and pre-unlink scripts that are executed when the package is installed or uninstalled You can read more about these files in the activation scripts and other special files section.
"},{"location":"package_spec/#package-metadata","title":"Package metadata","text":"The info/
directory contains all metadata about a package. Files in this location are not installed under the install prefix. Although you are free to add any file to this directory, conda only inspects the content of the files discussed below:
"},{"location":"package_spec/#infoindexjson","title":"info/index.json
","text":"This file contains basic information about the package, such as name, version, build string, and dependencies. The content of this file is stored in repodata.json
, which is the repository index file, hence the name index.json
. The JSON object is a dictionary containing the keys shown below.
name: string
The lowercase name of the package. May contain lowercase characters, underscores, and dashes.
version: string
The package version. May not contain \"-
\". Acknowledges PEP 440.
build: string
The build string. May not contain \"-
\". Differentiates builds of packages with otherwise identical names and versions, such as:
A build with other dependencies, such as Python 3.4 instead of Python 2.7. A bug fix in the build process. Some different optional dependencies, such as MKL versus ATLAS linkage. Nothing in conda actually inspects the build string. Strings such as np18py34_1
are designed only for human readability and conda never parses them. build_number: integer
A non-negative integer representing the build number of the package. Unlike the build string, the build_number
is inspected by conda. Conda uses it to sort packages that have otherwise identical names and versions to determine the latest one. This is important because new builds that contain bug fixes for the way a package is built may be added to a repository.
depends: list of match specs
A list of dependency specifications, where each element is a string. These come from the run
section of the recipe or any run exports of dependencies.
constrains: list of match specs
A list of optional dependency constraints. The packages listed under constrains
are not installed by default, but if they are installed they have to respect the constraints.
subdir: string
The subdir (like linux-64
) of this package.
arch: string
Optional. The architecture the package is built for. EXAMPLE: x86_64
. This key is generally not used (duplicate information from sudir
).
platform: string
Optional. The OS that the package is built for, e.g. osx
. This key is generally not used (duplicate information from sudir
).
"},{"location":"package_spec/#infopathsjson","title":"info/paths.json
","text":"The paths.json
file lists all files that are installed into the environment.
It consists of a list of path entries, each with the following keys:
_path: string
The relative path of the file
path_type: optional, string
The type of linking, can be hardlink
, softlink
, or directory
. Default is hardlink
.
file_mode: - optional, string
The file mode can be binary
or text
. This is only relevant for prefix replacement.
prefix_placeholder: optional, string
The prefix placeholder string that is encoded in the text or binary file, which is replaced at installation time. Note that this prefix placeholder uses /
even on Windows.
no_link: bool, optional
Determines whether this file should be linked or not when installing the package (linking the file from the cache into the environment). Defaults to false
.
sha256: string
The SHA256
hash of the file. For symbolic links it contains the SHA256
hash of the file pointed to.
size_in_bytes: number
The size, in bytes, of the file. For symbolic links, it contains the file size of the file pointed to.
Due to the way the binary replacement works, the placeholder prefix must be longer than the install prefix.
"},{"location":"package_spec/#infolicense","title":"info/license/<...>
","text":"All licenses mentioned in the recipe are copied to this folder.
"},{"location":"package_spec/#infoaboutjson","title":"info/about.json
","text":"Optional file. Contains the entries of the \"about\" section of the recipe of the recipe.yaml
file. The following keys are added to info/about.json
if present in the build recipe:
Renamed fields
The new recipe spec renamed a few fields (from conda-build's original implementation). This means that some fields in the about.json
file still have the old names (for backwards compatibility), while you would generally use different names in the recipe.
home: url (from about.homepage)
The URL of the homepage of the package.
dev_url: url (from about.repository)
The URL of the development repository of the package.
doc_url: url (from about.documentation)
The URL of the documentation of the package.
license_url: url
The URL of the license of the package.
license: string (from about.license)
The SPDX license identifier of the package.
summary: string
A short summary of the package.
description: string
A longer description of the package.
license_family: string
(this field is not used anymore as we rely on SPDX license identifiers)
"},{"location":"package_spec/#inforecipe","title":"info/recipe/<...>
","text":"A directory containing the full contents of the build recipe. This folder also contains a rendered version of the recipe (rendered_recipe.yaml
). This rendered version is used for the rebuild
command. However, note that currently this format is still in flux and can change at any time.
You can also use --no-include-recipe
to disable the inclusion of the recipe in the package.
"},{"location":"rebuild/","title":"Rebuilding a package","text":"The rebuild
command allows you to rebuild a package from an existing package. The main use case is to examine if a package can be rebuilt in a reproducible manner. You can read more about reproducible builds here.
"},{"location":"rebuild/#usage","title":"Usage","text":"rattler-build rebuild ./mypkg-0.1.0-h60d57d3_0.tar.bz2\n
"},{"location":"rebuild/#how-it-works","title":"How it works","text":"The recipe is \"rendered\" and stored into the package. The way the recipe is rendered is subject to change. For the moment, the rendered recipe is stored as info/recipe/rendered_recipe.yaml
. It includes the exact package versions that were used at build time. When rebuilding, we use the package resolutions from the rendered recipe, and execute the same build script as the original package.
We also take great care to sort files in a deterministic manner as well as erasing any time stamps. The SOURCE_DATE_EPOCH
environment variable is set to the same timestamp as the original build for additional determinism (some build tools use this variable to set timestamps).
"},{"location":"rebuild/#how-to-check-the-reproducibility-of-a-package","title":"How to check the reproducibility of a package","text":"There is an excellent tool called diffoscope
that allows you to compare two packages and see the differences. You can install it with pixi
:
pixi global install diffoscope\n
To compare two packages, you can use the following command:
rattler-build rebuild ./build0.tar.bz2\ndiffoscope ./build0.tar.bz2 ./mypkg-0.1.0-h60d57d3_0.tar.bz2\n
"},{"location":"recipe_generation/","title":"Generating recipes for different ecosystems","text":"Rattler-build has some builtin functionality to generate recipes for different (existing) ecosystems.
Currently we support the following ecosystems:
pypi
(Python) - generates a recipe for a Python package cran
(R) - generates a recipe for an R package To generate a recipe for a Python package, you can use the following command:
rattler-build generate-recipe pypi jinja2\n
This will generate a recipe for the jinja2
package from PyPI and print it to the console. To turn it into a recipe, you can either pipe the stdout to a file or use the -w
flag. The -w
flag will create a new folder with the recipe in it.
The generated recipe for jinja2
will look something like:
recipe.yamlpackage:\n name: jinja2\n version: 3.1.4\n\nsource:\n- url: https://files.pythonhosted.org/packages/ed/55/39036716d19cab0747a5020fc7e907f362fbf48c984b14e62127f7e68e5d/jinja2-3.1.4.tar.gz\n sha256: 4a3aee7acbbe7303aede8e9648d13b8bf88a429282aa6122a993f0ac800cb369\n\nbuild:\n script: python -m pip install .\n\nrequirements:\n host:\n - flit_core <4\n - python >=3.7\n - pip\n run:\n - python >=3.7\n - markupsafe >=2.0\n # - babel >=2.7 # extra == 'i18n'\n\ntests: []\n\nabout:\n summary: A very fast and expressive template engine.\n documentation: https://jinja.palletsprojects.com/\n
"},{"location":"recipe_generation/#generating-recipes-for-r-packages","title":"Generating recipes for R packages","text":"To generate a recipe for an R package, you can use the following command:
rattler-build generate-recipe cran dplyr\n
The R
recipe generation supports some additional flags:
-u/--universe
select an R universe to use (e.g. bioconductor
) -t/--tree
generate multiple recipes, for every dependency as well R packages will be prefixed with r-
to avoid name conflicts with Python packages. The generated recipe for dplyr
will look something like:
recipe.yamlpackage:\n name: r-dplyr\n version: 1.1.4\n\nsource:\n- url: https://cran.r-project.org/src/contrib/dplyr_1.1.4.tar.gz\n md5: e3066ea859b26e0d3b992c476ea3af2e\n\nbuild:\n script: R CMD INSTALL --build .\n python: {}\n\nrequirements:\n host:\n - r-base >=3.5.0\n run:\n - r-cli >=3.4.0\n - r-generics\n - r-glue >=1.3.2\n - r-lifecycle >=1.0.3\n - r-magrittr >=1.5\n - r-methods\n - r-pillar >=1.9.0\n - r-r6\n - r-rlang >=1.1.0\n - r-tibble >=3.2.0\n - r-tidyselect >=1.2.0\n - r-utils\n - r-vctrs >=0.6.4\n # - r-bench # suggested\n # - r-broom # suggested\n # - r-callr # suggested\n # - r-covr # suggested\n # - r-dbi # suggested\n # - r-dbplyr >=2.2.1 # suggested\n # - r-ggplot2 # suggested\n # - r-knitr # suggested\n # - r-lahman # suggested\n # - r-lobstr # suggested\n # - r-microbenchmark # suggested\n # - r-nycflights13 # suggested\n # - r-purrr # suggested\n # - r-rmarkdown # suggested\n # - r-rmysql # suggested\n # - r-rpostgresql # suggested\n # - r-rsqlite # suggested\n # - r-stringi >=1.7.6 # suggested\n # - r-testthat >=3.1.5 # suggested\n # - r-tidyr >=1.3.0 # suggested\n # - r-withr # suggested\n\nabout:\n homepage: https://dplyr.tidyverse.org, https://github.com/tidyverse/dplyr\n summary: A Grammar of Data Manipulation\n description: |-\n A fast, consistent tool for working with data frame like\n objects, both in memory and out of memory.\n license: MIT\n license_file: LICENSE\n repository: https://github.com/cran/dplyr\n
Tip
You can use the generated recipes to build your own \"forge\" with rattler-build
. Read more about it in the Building your own forge section.
"},{"location":"selectors/","title":"Selectors in recipes","text":"Recipe and variant configuration files can utilize selectors to conditionally add, remove, or modify dependencies, configuration options, or even skip recipe execution based on specific conditions.
Selectors are implemented using an if / then / else
map, which is a valid YAML dictionary. The condition is evaluated using minijinja
and follows the same syntax as a Python expression.
During rendering, several variables are set based on the platform and variant being built. For example, the unix
variable is true for macOS and Linux, while win
is true for Windows. Consider the following recipe executed on Linux:
requirements:\n host:\n - if: unix\n then: unix-tool\n - if: win\n then: win-tool\n
This will be evaluated as:
requirements:\n host:\n - unix-tool\n
The line containing the Windows-specific configuration is removed. Multiple items can also be selected, such as:
host:\n - if: linux\n then:\n - linux-tool-1\n - linux-tool-2\n - linux-tool-3\n
For Linux, this will result in:
host:\n - linux-tool-1\n - linux-tool-2\n - linux-tool-3\n
Other examples often found in the wild:
if: build_platform != target_platform ... # true if cross-platform build\nif: osx and arm64 ... # true for apple silicon (osx-arm64)\nif: linux and (aarch64 or ppc64le)) ... # true for linux ppc64le or linux-aarch64\n
"},{"location":"selectors/#available-variables","title":"Available variables","text":"The following variables are available during rendering of the recipe:
Variable Description target_platform
the configured target_platform
for the build build_platform
the configured build_platform
for the build linux
\"true\" if target_platform
is Linux osx
\"true\" if target_platform
is OSX / macOS win
\"true\" if target_platform
is Windows unix
\"true\" if target_platform
is a Unix (macOS or Linux) x86
, x86_64
x86 32/64-bit Architecture aarch64
, arm64
64-bit Arm (these are the same but are both supported for legacy) armV6l
, armV7l
32-bit Arm ppc64
, s390x
, Big endian ppc64le
Little endian riscv32
, riscv64
The RISC-V Architecture wasm32
The WebAssembly Architecture"},{"location":"selectors/#variant-selectors","title":"Variant selectors","text":"To select based on variant configuration you can use the names in the selectors as well. For example, if the build uses python: 3.8
as a variant, we can use if: python == \"3.8\"
to enable a dependency for only when the Python version is 3.8.
String comparison
The comparison is a string comparison done by minijinja
, so it is important to use the correct string representation of the variant. Use the match
function to compare versions.
variants.yamlpython:\n - 3.8\n - 3.9\n
recipe.yamlrequirements:\n host:\n - if: python == \"3.8\" # (1)!\n then: mydep\n else: otherdep\n
This will only add mydep
when the Python version is 3.8. This comparison is a string comparison, so it is important to use the correct string representation of the variant. "},{"location":"selectors/#the-match-function","title":"The match
function","text":"!!! note \"Rename from cmp
to match
\" The cmp
function has been renamed to match
to better reflect its purpose.
Inside selectors, one can use a special match
function to test if the selected variant version has a matching version. For example, having the following variants file, we could use the these tests:
variants.yamlpython:\n - 3.8\n - 3.9\n
recipe.yaml- if: match(python, \"3.8\") # true, false\n then: mydep\n- if: match(python, \">=3.8\") # true, true\n then: mydep\n- if: match(python, \"<3.8\") # false, false (1)\n then: mydep\n
else:
would also have worked here. This function eliminates the need to implement any Python-specific conda-build
selectors (such as py3k
, py38
, etc.) or the py
and npy
integers.
Please note that during the initial phase of rendering we do not know the variant, and thus the match
condition always evaluates to true
.
"},{"location":"selectors/#selector-evaluation","title":"Selector evaluation","text":"Except for the rattler-build specific selectors, the selectors are evaluated using the minijinja
engine. This means that the selectors are evaluated by minijinja
thus Python like expressions. Some notable options are:
- if: python == \"3.8\" # equal\n- if: python != \"3.8\" # not equal\n- if: python and linux # true if python variant is set and the target_platform is linux\n- if: python and not linux # true if python variant is set and the target_platform is not linux\n- if: python and (linux or osx) # true if python variant is set and the target_platform is linux or osx\n
"},{"location":"special_files/","title":"Activation scripts and other special files","text":"A conda
package can contain \"special\" files in the prefix. These files are scripts that are executed during activation, installation, or uninstallation process.
If possible, they should be avoided since they execute arbitrary code at installation time and slow down the installation and activation process.
"},{"location":"special_files/#activation-scripts","title":"Activation scripts","text":"The activation scripts are executed when the environment containing the package is activated (e.g. when doing micromamba activate myenv
or pixi run ...
).
The scripts are located in special folders:
etc/conda/activate.d/{script.sh/bat}
- scripts in this folder are executed before the environment is activated etc/conda/deactivate.d/{script.sh/bat}
- scripts in this folder are executed when the environment is deactivated The scripts are executed in lexicographical order, so you can prefix them with numbers to control the order of execution.
To add a script to the package, just make sure that you install the file in this folder. For example, on Linux:
mkdir -p $PREFIX/etc/conda/activate.d\ncp activate-mypkg.sh $PREFIX/etc/conda/activate.d/10-activate-mypkg.sh\n\nmkdir -p $PREFIX/etc/conda/deactivate.d\ncp deactivate-mypkg.sh $PREFIX/etc/conda/deactivate.d/10-deactivate-mypkg.sh\n
"},{"location":"special_files/#post-link-and-pre-unlink-scripts","title":"Post-link and pre-unlink scripts","text":"The post-link
and pre-unlink
scripts are executed when the package is installed or uninstalled. They are both heavily discouraged but implemented for compatibility with conda in rattler-build
since version 0.17.
For a post-link
script to be executed when a package is installed, the built package needs to have a .<package_name>-post-link.{sh/bat}
in its bin/
folder. The same is applicable for pre-unlink
scripts, just with the name .<package_name>-pre-unlink.{sh/bat}
(note the leading period). For example, for a package mypkg
, you would need to have a .mypkg-post-link.sh
in its bin/
folder.
To make sure the scripts are included in the correct location, use your recipe's build script or build/script
key. For example, assuming you have a post-link.sh
script in your source, alongside the recipe in the recipe's folder, the following configuration will copy it correctly:
build:\n ...\n script:\n - ...\n - mkdir -p $PREFIX/bin\n - cp $RECIPE_DIR/post-link.sh $PREFIX/bin/.mypkg-post-link.sh\n - chmod +x $PREFIX/bin/.mypkg-post-link.sh\n
The $PREFIX
and $RECIPE_DIR
environment variables will be set during the build process to help you specify the correct paths.
"},{"location":"testing/","title":"Testing packages","text":"When you are developing a package, you should write tests for it. The tests are automatically executed right after the package build has finished.
The tests from the test section are actually packaged into your package and can also be executed straight from the existing package.
The idea behind adding the tests into the package is that you can execute the tests independently from building the package. That is also why we are shipping a test
subcommand that takes as input an existing package and executes the tests:
rattler-build test --package-file ./xtensor-0.24.6-h60d57d3_0.tar.bz2\n
Running the above command will extract the package and create a clean environment where the package and dependencies are installed. Then the tests are executed in this newly-created environment.
If you inspect the package contents, you would find the test files under info/test/*
.
"},{"location":"testing/#how-tests-are-translated","title":"How tests are translated","text":"The tests
section allows you to specify the following things:
tests:\n - script:\n # commands to run to test the package. If any of the commands\n # returns with an error code, the test is considered failed.\n - echo \"Hello world\"\n - pytest ./tests\n\n # additional requirements at test time\n requirements:\n run:\n - pytest\n\n files:\n # Extra files to be copied to the test directory from the \"work directory\"\n source:\n - tests/\n - test.py\n - *.sh\n recipe:\n - more_tests/*.py\n\n # This test section tries to import the Python modules and errors if it can't\n - python:\n imports:\n - mypkg\n - mypkg.subpkg\n
When you are writing a test for your package, additional files are created and added to your package. These files are placed under the info/tests/{index}/
folder for each test.
For a script test:
All the files are copied straight into the test folder (under info/tests/{index}/
) The script is turned into a run_test.sh
or run_test.bat
file The extra requirements are stored as a JSON file called test_time_dependencies.json
For a Python import test:
A JSON file is created that is called python_test.json
and stores the imports to be tested and whether to execute pip check
or not. This file is placed under info/tests/{index}/
For a downstream test:
A JSON file is created that is called downstream_test.json
and stores the downstream tests to be executed. This file is placed under info/tests/{index}/
"},{"location":"testing/#legacy-tests","title":"Legacy tests","text":"Legacy tests (from conda-build
) are still supported for execution. These tests are stored as files under the info/test/
folder.
The files are:
run_test.sh
(Unix) run_test.bat
(Windows) run_test.py
(for the Python import tests) test_time_dependencies.json
(for additional dependencies at test time) Additionally, the info/test/
folder contains all the files specified in the test section as source_files
and files
. The tests are executed pointing to this directory as the current working directory.
"},{"location":"tips_and_tricks/","title":"Tips and tricks for rattler-build","text":"This section contains some tips and tricks for using rattler-build
.
"},{"location":"tips_and_tricks/#using-sccache-or-ccache-with-rattler-build","title":"Using sccache or ccache with rattler-build
","text":"When debugging a recipe it can help a lot to use sccache
or ccache
. You can install both tools e.g. with pixi global install sccache
.
To use them with a CMake project, you can use the following variables:
export CMAKE_C_COMPILER_LAUNCHER=sccache\nexport CMAKE_CXX_COMPILER_LAUNCHER=sccache\n\n# or more generally\n\nexport C=\"sccache $C\"\nexport CXX=\"sccache $CXX\"\n
However, both ccache
and sccache
are sensitive to changes in the build location. Since rattler-build
, by default, always creates a new build directory with the timestamp, you need to use the --no-build-id
flag. This will disable the time stamp in the build directory and allow ccache
and sccache
to cache the build.
rattler-build build --no-build-id --recipe ./path/to/recipe.yaml\n
"},{"location":"tips_and_tricks/#building-your-own-forge","title":"Building your own \"forge\"","text":"You might want to publish your own software packages to a channel you control. These might be packages that are not available in the main conda-forge channel, or proprietary packages, or packages that you have modified in some way.
Doing so is pretty straightforward with rattler-build
and a CI provider of your choice. We have a number of example repositories for \"custom\" forges:
rust-forge: This repository builds a number of Rust packages for Windows, macOS and Linux on top of Github Actions. r-forge: The same idea, but for R
packages "},{"location":"tips_and_tricks/#directory-structure","title":"Directory structure","text":"To create your own forge, you should create a number of sub-directories where each sub-directory should contain at most one recipe. With the --recipe-dir
flag of rattler-build, the program will go and collect all recipes it finds in the given directory or sub-directories.
We can combine this with the --skip-existing=all
flag which will skip all packages that are already built locally or in the channel (if you upload them). Using all
will also look at the repodata.json
file in the channel to see if the package is already there. Packages are skipped based on their complete name, including the version and build string.
To note: the build string changes if the variant configuration changes! So if you update a package in the variant configuration, the packages that need rebuilding should be rebuilt.
Note
You can generate recipes for different ecosystems with the rattler-build generate-recipe
command. Read more about it in the Generating recipes section.
"},{"location":"tips_and_tricks/#ci-setup","title":"CI setup","text":"As an example, the following is the CI setup for rust-forge
. The workflow uses rattler-build
to build and upload packages to a custom channel on https://prefix.dev \u2013 but you can also use rattler-build
to upload to your own quetz
instance, or a channel on anaconda.org
.
Example CI setup for rust-forge
The following is an example of a Github Actions workflow for rust-forge
:
.github/workflows/forge.ymlname: Build all packages\n\non:\n push:\n branches:\n - main\n workflow_dispatch:\n pull_request:\n branches:\n - main\n\njobs:\n build:\n strategy:\n matrix:\n include:\n - { target: linux-64, os: ubuntu-20.04 }\n - { target: win-64, os: windows-latest }\n # force older macos-13 to get x86_64 runners\n - { target: osx-64, os: macos-13 }\n - { target: osx-arm64, os: macos-14 }\n fail-fast: false\n\n runs-on: ${{ matrix.os }}\n steps:\n - uses: actions/checkout@v4\n with:\n fetch-depth: 2\n - uses: prefix-dev/setup-pixi@v0.5.1\n with:\n pixi-version: v0.24.2\n cache: true\n\n - name: Run code in changed subdirectories\n shell: bash\n env:\n TARGET_PLATFORM: ${{ matrix.target }}\n\n run: |\n pixi run rattler-build build --recipe-dir . \\\n --skip-existing=all --target-platform=$TARGET_PLATFORM \\\n -c conda-forge -c https://prefix.dev/rust-forge\n\n - name: Upload all packages\n shell: bash\n # do not upload on PR\n if: github.event_name == 'push'\n env:\n PREFIX_API_KEY: ${{ secrets.PREFIX_API_KEY }}\n run: |\n # ignore errors because we want to ignore duplicate packages\n for file in output/**/*.conda; do\n pixi run rattler-build upload prefix -c rust-forge \"$file\" || true\n done\n
"},{"location":"tui/","title":"Terminal User Interface","text":"rattler-build
offers a terminal user interface for building multiple packages and viewing the logs.
To launch the TUI, run the build
command with the --tui
flag as shown below:
$ rattler-build build -r recipe.yaml --tui\n
Note
rattler-build-tui
is gated behind the tui
feature flag to avoid extra dependencies. Build the project with --features tui
arguments to enable the TUI functionality.
"},{"location":"tui/#key-bindings","title":"Key Bindings","text":"Key Action \u23ce Build a Build all j/k Next/previous package up/down/left/right Scroll logs e Edit recipe (via $EDITOR
) c, : Open command prompt (available commands: edit
) q, ctrl-c, esc, Quit"},{"location":"variants/","title":"Variant configuration","text":"rattler-build
can automatically build multiple variants of a given package. For example, a Python package might need multiple variants per Python version (especially if it is a binary package such as numpy
).
For this use case, one can specify variant configuration files. A variant configuration file has 2 special entries and a list of packages with variants. For example:
variants.yaml# special entry #1, the zip keys\nzip_keys:\n- [python, numpy]\n\n# special entry #2, the pin_run_as_build key\npin_run_as_build:\n numpy:\n max_pin: 'x.x'\n\n# entries per package version that users are interested in\npython:\n# Note that versions are _strings_ (not numbers)\n- \"3.8\"\n- \"3.9\"\n- \"3.10\"\n\nnumpy:\n- \"1.12\"\n- \"1.12\"\n- \"1.20\"\n
If we have a recipe, that has a build
, host
or run
dependency on python
we will build multiple variants of this package, one for each configured python
version (\"3.8\", \"3.9\" and \"3.10\").
For example:
# ...\nrequirements:\n host:\n - python\n
... will be rendered as (for the first variant):
# ...\nrequirements:\n host:\n- python 3.8*\n
Note that variants are only applied if the requirement doesn't specify any constraints. If the requirement would be python >3.8,<3.10
then the variant entry would be ignored.
"},{"location":"variants/#automatic-variantsyaml-discovery","title":"Automatic variants.yaml
discovery","text":"rattler-build
automatically includes the variant configuration from a variants.yaml
file next to a recipe. Use the --ignore-recipe-variants
option to disable automatic discovery of variants.yaml
files next to the recipes.
To include a variant config file from another location or include multiple configuration files use the --variant-config
option:
rattler-build build --variant-config ~/user_variants.yaml --variant-config /opt/rattler-build/global_variants.yaml --recipe myrecipe.yaml\n
"},{"location":"variants/#package-hash-from-variant","title":"Package hash from variant","text":"You might have wondered what the role of the build string is. The build string is (if not explicitly set) computed from the variant configuration. It serves as a mechanism to discern different build configurations that produce a package with the same name and version.
The hash is computed by dumping all of the variant configuration values that are used by a given recipe into a JSON file, and then hashing that JSON file.
For example, in our python
example, we would get a variant configuration file that looks something like:
{\n \"python\": \"3.8\"\n}\n
This JSON string is then hashed with the MD5 hash algorithm, and produces the hash. For certain packages (such as Python packages) special rules exists, and the py<Major.Minor>
version is prepended to the hash, so that the final hash would look something like py38h123123
.
"},{"location":"variants/#zip-keys","title":"Zip keys","text":"Zip keys modify how variants are combined. Usually, each variant key that has multiple entries is expanded to a build matrix. For example, if we have:
python: [\"3.8\", \"3.9\"]\nnumpy: [\"1.12\", \"1.14\"]\n
...then we obtain 4 variants for a recipe that uses both numpy
and python
:
- python 3.8, numpy 1.12\n- python 3.8, numpy 1.14\n- python 3.9, numpy 1.12\n- python 3.9, numpy 1.14\n
However, if we use the zip_keys
and specify:
zip_keys: [\"python\", \"numpy\"]\npython: [\"3.8\", \"3.9\"]\nnumpy: [\"1.12\", \"1.14\"]\n
...then the versions are \"zipped up\" and we only get 2 variants. Note that both python
and numpy
need to specify the exact same number of versions to make this work.
The resulting variants with the zip applied are:
- python 3.8, numpy 1.12\n- python 3.9, numpy 1.14\n
"},{"location":"variants/#pin-run-as-build","title":"Pin run as build","text":"The pin_run_as_build
key allows the user to inject additional pins. Usually, the run_exports
mechanism is used to specify constraints for runtime dependencies from build time dependencies, but pin_run_as_build
offers a mechanism to override that if the package does not contain a run exports file.
For example:
pin_run_as_build:\n libcurl:\n min_pin: 'x'\n max_pin: 'x'\n
If we now have a recipe that uses libcurl
in the host
and run
dependencies like:
requirements:\n host:\n - libcurl\n run:\n - libcurl\n
During resolution, libcurl
might be evaluated to libcurl 8.0.1 h13284
. Our new runtime dependency then looks like:
requirements:\n host:\n - libcurl 8.0.1 h13284\n run:\n - libcurl >=8,<9\n
"},{"location":"variants/#prioritizing-variants","title":"Prioritizing variants","text":"You might produce multiple variants for a package, but want to define a priority for a given variant. The variant with the highest priority would be the default package that is selected by the resolver.
There are two mechanisms to make this possible: mutex
packages and the down_prioritize_variant
option in the recipe.
"},{"location":"variants/#the-down_prioritize_variant-option","title":"The down_prioritize_variant
option","text":"Note
It is not always necessary to use the down_prioritize_variant
option - only if the solver has no other way to prefer a given variant. For example, if you have a package that has multiple variants for different Python versions, the solver will automatically prefer the variant with the highest Python version.
The down_prioritize_variant
option allows you to specify a variant that should be down-prioritized. For example:
recipe.yamlbuild:\n variant:\n use_keys:\n # use cuda from the variant config, e.g. to build multiple CUDA variants\n - cuda\n # this will down-prioritize the cuda variant versus other variants of the package\n down_prioritize_variant: ${{ 1 if cuda else 0 }}\n
"},{"location":"variants/#mutex-packages","title":"Mutex packages","text":"Another way to make sure the right variants are selected are \"mutex\" packages. A mutex package is a package that is mutually exclusive. We use the fact that only one package of a given name can be installed at a time (the solver has to choose).
A mutex package might be useful to make sure that all packages that depend on BLAS are compiled against the same BLAS implementation. The mutex package will serve the purpose that \"openblas
\" and \"mkl
\" can never be installed at the same time.
We could define a BLAS mutex package like this:
variant_config.yamlblas_variant:\n - \"openblas\"\n - \"mkl\"\n
And then the recipe.yaml
for the mutex
package could look like this:
recipe.yamlpackage:\n name: blas_mutex\n version: 1.0\n\nbuild:\n string: ${{ blas_variant }}${{ hash }}_${{ build_number }}\n variant_config:\n # make sure that `openblas` is preferred over `mkl`\n down_prioritize_variant: ${{ 1 if blas_variant == \"mkl\" else 0 }}\n
This will create two package: blas_mutex-1.0-openblas
and blas_mutex-1.0-mkl
. Only one of these packages can be installed at a time because they share the same name. The solver will then only select one of these two packages.
The blas
package in turn should have a run_export
for the blas_mutex
package, so that any package that links against blas
also has a dependency on the correct blas_mutex
package:
recipe.yamlpackage:\n name: openblas\n version: 1.0\n\nrequirements:\n # any package depending on openblas should also depend on the correct blas_mutex package\n run_export:\n # Add a run export on _any_ version of the blas_mutex package whose build string starts with \"openblas\"\n - blas_mutex * openblas*\n
Then the recipe of a package that wants to build two variants, one for openblas
and one for mkl
could look like this:
recipe.yamlpackage:\n name: fastnumerics\n version: 1.0\n\nrequirements:\n host:\n # build against both openblas and mkl\n - ${{ blas_variant }}\n run:\n # implicitly adds the correct blas_mutex package through run exports\n # - blas_mutex * ${{ blas_variant }}*\n
"},{"location":"reference/cli/","title":"Command-Line Help for rattler-build
","text":"This document contains the help content for the rattler-build
command-line program.
"},{"location":"reference/cli/#rattler-build","title":"rattler-build
","text":"Usage: rattler-build [OPTIONS] [COMMAND]
"},{"location":"reference/cli/#subcommands","title":"Subcommands:","text":" build
\u2014 Build a package from a recipe test
\u2014 Run a test for a single package rebuild
\u2014 Rebuild a package from a package file instead of a recipe upload
\u2014 Upload a package completion
\u2014 Generate shell completion script generate-recipe
\u2014 Generate a recipe from PyPI or CRAN auth
\u2014 Handle authentication to external channels "},{"location":"reference/cli/#options","title":"Options:","text":" -v
, --verbose
Increase logging verbosity
-q
, --quiet
Decrease logging verbosity
--log-style <LOG_STYLE>
Logging style
Default value: fancy
Possible values: fancy
: Use fancy logging output json
: Use JSON logging output plain
: Use plain logging output --wrap-log-lines <WRAP_LOG_LINES>
Wrap log lines at the terminal width. This is automatically disabled on CI (by detecting the CI
environment variable)
Possible values: true
, false
--color <COLOR>
Enable or disable colored output from rattler-build. Also honors the CLICOLOR
and CLICOLOR_FORCE
environment variable
Default value: auto
Possible values: always
: Always use colors never
: Never use colors auto
: Use colors when the output is a terminal "},{"location":"reference/cli/#build","title":"build
","text":"Build a package from a recipe
Usage: rattler-build build [OPTIONS]
"},{"location":"reference/cli/#options_1","title":"Options:","text":" -r
, --recipe <RECIPE>
The recipe file or directory containing recipe.yaml
. Defaults to the current directory
--recipe-dir <RECIPE_DIR>
The directory that contains recipes
--up-to <UP_TO>
Build recipes up to the specified package
--build-platform <BUILD_PLATFORM>
The build platform to use for the build (e.g. for building with emulation, or rendering)
Default value: current platform --target-platform <TARGET_PLATFORM>
The target platform for the build
--host-platform <HOST_PLATFORM>
The host platform for the build. If set, it will be used to determine also the target_platform (as long as it is not noarch)
Default value: current platform -c
, --channel <CHANNEL>
Add a channel to search for dependencies in
Default value: conda-forge
-m
, --variant-config <VARIANT_CONFIG>
Variant configuration files for the build
--ignore-recipe-variants
Do not read the variants.yaml
file next to a recipe
Possible values: true
, false
--render-only
Render the recipe files without executing the build
Possible values: true
, false
--with-solve
Render the recipe files with solving dependencies
Possible values: true
, false
--keep-build
Keep intermediate build artifacts after the build
Possible values: true
, false
--no-build-id
Don't use build id(timestamp) when creating build directory name
Possible values: true
, false
--compression-threads <COMPRESSION_THREADS>
The number of threads to use for compression (only relevant when also using --package-format conda
)
--use-zstd
Enable support for repodata.json.zst
Default value: true
Possible values: true
, false
--use-bz2
Enable support for repodata.json.bz2
Default value: true
Possible values: true
, false
--experimental
Enable experimental features
Possible values: true
, false
--auth-file <AUTH_FILE>
Path to an auth-file to read authentication information from
--tui
Launch the terminal user interface
Default value: false
Possible values: true
, false
--extra-meta <EXTRA_META>
Extra metadata to include in about.json
"},{"location":"reference/cli/#modifying-result","title":"Modifying result","text":" --package-format <PACKAGE_FORMAT>
The package format to use for the build. Can be one of tar-bz2
or conda
. You can also add a compression level to the package format, e.g. tar-bz2:<number>
(from 1 to 9) or conda:<number>
(from -7 to 22).
--no-include-recipe
Don't store the recipe in the final package
Possible values: true
, false
--no-test
Do not run tests after building (deprecated, use --test=skip
instead)
Default value: false
Possible values: true
, false
--test <TEST>
The strategy to use for running tests
Default value: native-and-emulated
Possible values: skip
: Skip the tests native
: Run the tests only if the build platform is the same as the host platform. Otherwise, skip the tests. If the target platform is noarch, the tests are always executed native-and-emulated
: Always run the tests --color-build-log
Don't force colors in the output of the build script
Default value: true
Possible values: true
, false
--output-dir <OUTPUT_DIR>
Output directory for build artifacts.
--skip-existing <SKIP_EXISTING>
Whether to skip packages that already exist in any channel If set to none
, do not skip any packages, default when not specified. If set to local
, only skip packages that already exist locally, default when using --skip-existing. If set to
all`, skip packages that already exist in any channel
Default value: none
Possible values: none
: Do not skip any packages local
: Skip packages that already exist locally all
: Skip packages that already exist in any channel --noarch-build-platform <NOARCH_BUILD_PLATFORM>
Define a \"noarch platform\" for which the noarch packages will be built for. The noarch builds will be skipped on the other platforms
"},{"location":"reference/cli/#test","title":"test
","text":"Run a test for a single package
This creates a temporary directory, copies the package file into it, and then runs the indexing. It then creates a test environment that installs the package and any extra dependencies specified in the package test dependencies file.
With the activated test environment, the packaged test files are run:
info/test/run_test.sh
or info/test/run_test.bat
on Windows * info/test/run_test.py
These test files are written at \"package creation time\" and are part of the package.
Usage: rattler-build test [OPTIONS] --package-file <PACKAGE_FILE>
"},{"location":"reference/cli/#options_2","title":"Options:","text":" -c
, --channel <CHANNEL>
Channels to use when testing
-p
, --package-file <PACKAGE_FILE>
The package file to test
--compression-threads <COMPRESSION_THREADS>
The number of threads to use for compression
--use-zstd
Enable support for repodata.json.zst
Default value: true
Possible values: true
, false
--use-bz2
Enable support for repodata.json.bz2
Default value: true
Possible values: true
, false
--experimental
Enable experimental features
Possible values: true
, false
--auth-file <AUTH_FILE>
Path to an auth-file to read authentication information from
"},{"location":"reference/cli/#modifying-result_1","title":"Modifying result","text":""},{"location":"reference/cli/#rebuild","title":"rebuild
","text":"Rebuild a package from a package file instead of a recipe
Usage: rattler-build rebuild [OPTIONS] --package-file <PACKAGE_FILE> --test <TEST>
"},{"location":"reference/cli/#options_3","title":"Options:","text":" -p
, --package-file <PACKAGE_FILE>
The package file to rebuild
--no-test
Do not run tests after building (deprecated, use --test=skip
instead)
Default value: false
Possible values: true
, false
--compression-threads <COMPRESSION_THREADS>
The number of threads to use for compression
--use-zstd
Enable support for repodata.json.zst
Default value: true
Possible values: true
, false
--use-bz2
Enable support for repodata.json.bz2
Default value: true
Possible values: true
, false
--experimental
Enable experimental features
Possible values: true
, false
--auth-file <AUTH_FILE>
Path to an auth-file to read authentication information from
"},{"location":"reference/cli/#modifying-result_2","title":"Modifying result","text":" --test <TEST>
The strategy to use for running tests
Possible values: skip
: Skip the tests native
: Run the tests only if the build platform is the same as the host platform. Otherwise, skip the tests. If the target platform is noarch, the tests are always executed native-and-emulated
: Always run the tests --output-dir <OUTPUT_DIR>
Output directory for build artifacts.
"},{"location":"reference/cli/#upload","title":"upload
","text":"Upload a package
Usage: rattler-build upload [OPTIONS] [PACKAGE_FILES]... <COMMAND>
"},{"location":"reference/cli/#subcommands_1","title":"Subcommands:","text":" quetz
\u2014 Upload to aQuetz server. Authentication is used from the keychain / auth-file artifactory
\u2014 Options for uploading to a Artifactory channel. Authentication is used from the keychain / auth-file prefix
\u2014 Options for uploading to a prefix.dev server. Authentication is used from the keychain / auth-file anaconda
\u2014 Options for uploading to a Anaconda.org server "},{"location":"reference/cli/#arguments","title":"Arguments:","text":""},{"location":"reference/cli/#options_4","title":"Options:","text":" --use-zstd
Enable support for repodata.json.zst
Default value: true
Possible values: true
, false
--use-bz2
Enable support for repodata.json.bz2
Default value: true
Possible values: true
, false
--experimental
Enable experimental features
Possible values: true
, false
--auth-file <AUTH_FILE>
Path to an auth-file to read authentication information from
"},{"location":"reference/cli/#modifying-result_3","title":"Modifying result","text":""},{"location":"reference/cli/#quetz","title":"quetz
","text":"Upload to aQuetz server. Authentication is used from the keychain / auth-file
Usage: rattler-build upload quetz [OPTIONS] --url <URL> --channel <CHANNEL>
"},{"location":"reference/cli/#options_5","title":"Options:","text":" -u
, --url <URL>
The URL to your Quetz server
-c
, --channel <CHANNEL>
The URL to your channel
-a
, --api-key <API_KEY>
The Quetz API key, if none is provided, the token is read from the keychain / auth-file
"},{"location":"reference/cli/#artifactory","title":"artifactory
","text":"Options for uploading to a Artifactory channel. Authentication is used from the keychain / auth-file
Usage: rattler-build upload artifactory [OPTIONS] --url <URL> --channel <CHANNEL>
"},{"location":"reference/cli/#options_6","title":"Options:","text":" -u
, --url <URL>
The URL to your Artifactory server
-c
, --channel <CHANNEL>
The URL to your channel
-r
, --username <USERNAME>
Your Artifactory username
-p
, --password <PASSWORD>
Your Artifactory password
"},{"location":"reference/cli/#prefix","title":"prefix
","text":"Options for uploading to a prefix.dev server. Authentication is used from the keychain / auth-file
Usage: rattler-build upload prefix [OPTIONS] --channel <CHANNEL>
"},{"location":"reference/cli/#options_7","title":"Options:","text":" -u
, --url <URL>
The URL to the prefix.dev server (only necessary for self-hosted instances)
Default value: https://prefix.dev
-c
, --channel <CHANNEL>
The channel to upload the package to
-a
, --api-key <API_KEY>
The prefix.dev API key, if none is provided, the token is read from the keychain / auth-file
"},{"location":"reference/cli/#anaconda","title":"anaconda
","text":"Options for uploading to a Anaconda.org server
Usage: rattler-build upload anaconda [OPTIONS] --owner <OWNER>
"},{"location":"reference/cli/#options_8","title":"Options:","text":" -o
, --owner <OWNER>
The owner of the distribution (e.g. conda-forge or your username)
-c
, --channel <CHANNEL>
The channel / label to upload the package to (e.g. main / rc)
-a
, --api-key <API_KEY>
The Anaconda API key, if none is provided, the token is read from the keychain / auth-file
-u
, --url <URL>
The URL to the Anaconda server
Default value: https://api.anaconda.org
-f
, --force
Replace files on conflict
Default value: false
Possible values: true
, false
"},{"location":"reference/cli/#completion","title":"completion
","text":"Generate shell completion script
Usage: rattler-build completion --shell <SHELL>
"},{"location":"reference/cli/#options_9","title":"Options:","text":""},{"location":"reference/cli/#generate-recipe","title":"generate-recipe
","text":"Generate a recipe from PyPI or CRAN
Usage: rattler-build generate-recipe <COMMAND>
"},{"location":"reference/cli/#subcommands_2","title":"Subcommands:","text":" pypi
\u2014 Generate a recipe for a Python package from PyPI cran
\u2014 Generate a recipe for an R package from CRAN "},{"location":"reference/cli/#pypi","title":"pypi
","text":"Generate a recipe for a Python package from PyPI
Usage: rattler-build generate-recipe pypi [OPTIONS] <PACKAGE>
"},{"location":"reference/cli/#arguments_1","title":"Arguments:","text":""},{"location":"reference/cli/#options_10","title":"Options:","text":" -w
, --write
Whether to write the recipe to a folder
Possible values: true
, false
-u
, --use-mapping
Whether to use the conda-forge PyPI name mapping
Default value: true
Possible values: true
, false
-t
, --tree
Whether to generate recipes for all dependencies
Possible values: true
, false
"},{"location":"reference/cli/#cran","title":"cran
","text":"Generate a recipe for an R package from CRAN
Usage: rattler-build generate-recipe cran [OPTIONS] <PACKAGE>
"},{"location":"reference/cli/#arguments_2","title":"Arguments:","text":""},{"location":"reference/cli/#options_11","title":"Options:","text":" -u
, --universe <UNIVERSE>
The R Universe to fetch the package from (defaults to cran
)
-t
, --tree
Whether to create recipes for the whole dependency tree or not
Possible values: true
, false
-w
, --write
Whether to write the recipe to a folder
Possible values: true
, false
"},{"location":"reference/cli/#auth","title":"auth
","text":"Handle authentication to external channels
Usage: rattler-build auth <COMMAND>
"},{"location":"reference/cli/#subcommands_3","title":"Subcommands:","text":" login
\u2014 Store authentication information for a given host logout
\u2014 Remove authentication information for a given host "},{"location":"reference/cli/#login","title":"login
","text":"Store authentication information for a given host
Usage: rattler-build auth login [OPTIONS] <HOST>
"},{"location":"reference/cli/#arguments_3","title":"Arguments:","text":""},{"location":"reference/cli/#options_12","title":"Options:","text":" --token <TOKEN>
The token to use (for authentication with prefix.dev)
--username <USERNAME>
The username to use (for basic HTTP authentication)
--password <PASSWORD>
The password to use (for basic HTTP authentication)
--conda-token <CONDA_TOKEN>
The token to use on anaconda.org / quetz authentication
"},{"location":"reference/cli/#logout","title":"logout
","text":"Remove authentication information for a given host
Usage: rattler-build auth logout <HOST>
"},{"location":"reference/cli/#arguments_4","title":"Arguments:","text":" This document was generated automatically by clap-markdown
.
"},{"location":"reference/jinja/","title":"Jinja","text":"rattler-build
comes with a couple of useful Jinja functions and filters that can be used in the recipe.
"},{"location":"reference/jinja/#functions","title":"Functions","text":""},{"location":"reference/jinja/#the-compiler-function","title":"The compiler function","text":"The compiler function can be used to put together a compiler that works for the current platform and the compilation \"target_platform
\". The syntax looks like: ${{ compiler('c') }}
where 'c'
signifies the programming language that is used.
This function evaluates to <compiler>_<target_platform> <compiler_version>
. For example, when compiling on linux
and to linux-64
, this function evaluates to gcc_linux-64
.
The values can be influenced by the variant_configuration
. The <lang>_compiler
and <lang>_compiler_version
variables are the keys with influence. See below for an example:
"},{"location":"reference/jinja/#usage-in-a-recipe","title":"Usage in a recipe","text":"recipe.yamlrequirements:\n build:\n - ${{ compiler('c') }}\n
With a corresponding variant_configuration:
variant_configuration.yamlc_compiler:\n- clang\nc_compiler_version:\n- 9.0\n
The variables shown above would select the clang
compiler in version 9.0
. Note that the final output will still contain the target_platform
, so that the full compiler will read clang_linux-64 9.0
when compiling with --target-platform linux-64
.
rattler-build
defines some default compilers for the following languages (inherited from conda-build
):
c
: gcc
on Linux, clang
on osx
and vs2017
on Windows cxx
: gxx
on Linux, clangxx
on osx
and vs2017
on Windows fortran
: gfortran
on Linux, gfortran
on osx
and vs2017
on Windows rust
: rust
"},{"location":"reference/jinja/#the-stdlib-function","title":"The stdlib
function","text":"The stdlib
function closely mirrors the compiler function. It can be used to put together a standard library that works for the current platform and the compilation \"target_platform
\".
Usage: ${{ stdlib('c') }}
Results in <stdlib>_<target_platform> <stdlib_version>
. And uses the variant variables <lang>_stdlib
and <lang>_stdlib_version
to influence the output.
"},{"location":"reference/jinja/#usage-in-a-recipe_1","title":"Usage in a recipe:","text":"recipe.yamlrequirements:\n build:\n # these are usually paired!\n - ${{ compiler('c') }}\n - ${{ stdlib('c') }}\n
With a corresponding variant_configuration:
variant_configuration.yaml# these are the values `conda-forge` uses in their pinning file\n# found at https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml\nc_stdlib:\n- sysroot\nc_stdlib_version:\n- 2.17\n
"},{"location":"reference/jinja/#the-pin-functions","title":"The pin
functions","text":"A pin is created based on the version input (from a subpackage or a package resolution).
The pin functions take the following three arguments:
lower_bound
(default: \"x.x.x.x.x.x\"
): The lower bound pin expression to be used. When set to None
, no lower bound is set. upper_bound
(default: \"x\"
): The maximum pin to be used. When set to None
, no upper bound is set. The lower bound and upper bound can either be a \"pin expression\" (only x
and .
are allowed) or a hard-coded version string.
A \"pin expression\" is applied to the version input to create the lower and upper bounds. For example, if the version is 3.10.5
with a lower_bound=\"x.x\", upper_bound=\"x.x.x\"
, the lower bound will be 3.10
and the upper bound will be 3.10.6.0a0
. A pin expression for the upper_bound
will increment the last selected segment of the version by 1
, and append .0a0
to the end to prevent any alpha versions from being selected.
If the last segment of the version contains a letter (e.g. 9e
or 1.1.1j
), then incrementing the version will set that letter to a
, e.g. 9e
will become 10a
, and 1.1.1j
will become 1.1.2a
. In this case, also no 0a0
is appended to the end.
Sometimes you want to strongly connect your outputs. This can be achieved with the following input:
exact=True
(default: False
): This will pin the version exactly to the version of the output, incl. the build string. To override the lower or upper bound with a hard-coded value, you can use the following input:
lower_bound
(default: None
): This will override the lower bound with the given value. upper_bound
(default: None
): This will override the upper bound with the given value. Both lower_bound
and upper_bound
expect a valid version string (e.g. 1.2.3
).
To add an build-string matching expression, you can use the build
argument:
build
(default: None
): This will add a build string matching expression to the pin. The build string matching expression is a string that is used to match the build string with the match spec. For example, if the build string is py38_0
, the build string matching expression could be py38*
or to match exactly py38_0
. The build
and exact
options are mutually exclusive. "},{"location":"reference/jinja/#the-pin_subpackage-function","title":"The pin_subpackage
function","text":" ${{ pin_subpackage(\"mypkg\", lower_bound=\"x.x\", upper_bound=\"x.x\") }}
creates a pin to another output in the recipe. With an input of 3.1.5
, this would create a pin of mypkg >=3.1,<3.2.0a0
. ${{ pin_subpackage(\"other_output\", exact=True) }}
creates a pin to another output in the recipe with an exact version. ${{ pin_subpackage(\"other_output\", lower_bound=\"1.2.3\", upper_bound=\"1.2.4\") }}
creates a pin to another output in the recipe with a lower bound of 1.2.3
and an upper bound of 1.2.4
. This is equivalent to writing other_output >=1.2.3,<1.2.4
. ${{ pin_subpackage(\"foo\", build=\"py38*\") }}
creates a matchspec like foo >=3.1,<3.2.0a0 py38*
. "},{"location":"reference/jinja/#the-pin_compatible-function","title":"The pin_compatible
function","text":"The pin compatible function works exactly as the pin_subpackage
function, but it pins the package in the run requirements based on the resolved package of the host
or build
section.
pin_compatible
pins a package in the run requirements based on the resolved package of the host
or build
section. "},{"location":"reference/jinja/#the-cdt-function","title":"The cdt
function","text":" ${{ cdt(\"mypkg\") }}
creates a cross-dependency to another output in the recipe. This function helps add Core Dependency Tree packages as dependencies by converting packages as required according to hard-coded logic. See below for an example of how this function can be used:
# on x86_64 system\ncdt('package-name') # outputs: package-name-cos6-x86_64\n# on aarch64 system\ncdt('package-name') # outputs: package-name-cos6-aarch64\n
"},{"location":"reference/jinja/#the-hash-variable","title":"The hash
variable","text":" ${{ hash }}
is the variant hash and is useful in the build string computation. "},{"location":"reference/jinja/#the-version_to_buildstring-function","title":"The version_to_buildstring
function","text":" ${{ python | version_to_buildstring }}
converts a version from the variant to a build string (it removes the .
character and takes only the first two elements of the version). "},{"location":"reference/jinja/#the-env-object","title":"The env
object","text":"You can use the env
object to retrieve environment variables and forward them to your build script. ${{ env.get(\"MY_ENV_VAR\") }}
will return the value of the environment variable MY_ENV_VAR
or throw an error if it is not set.
To supply a default value when the environment variable is not set, you can use ${{ env.get(\"MY_ENV_VAR\", default=\"default_value\") }}
. In this case, if MY_ENV_VAR
is not set, the value default_value
will be returned (and no error is thrown).
You can also check for the existence of an environment variable:
${{ env.exists(\"MY_ENV_VAR\") }}
will return true
if the environment variable MY_ENV_VAR
is set and false
otherwise. "},{"location":"reference/jinja/#filters","title":"Filters","text":"A feature of jinja
is called \"filters\". Filters are functions that can be applied to variables in a template expression.
The syntax for a filter is {{ variable | filter_name }}
. A filter can also take arguments, such as ... | replace('foo', 'bar')
.
The following Jinja filters are available, taken from the upstream minijinja
library:
replace
: replace a string with another string (e.g. \"{{ 'foo' | replace('oo', 'aa') }}\"
will return \"faa\"
) lower
: convert a string to lowercase (e.g. \"{{ 'FOO' | lower }}\"
will return \"foo\"
) upper
: convert a string to uppercase (e.g. \"{{ 'foo' | upper }}\"
will return \"FOO\"
) - int
: convert a string to an integer (e.g. \"{{ '42' | int }}\"
will return 42
) abs
: return the absolute value of a number (e.g. \"{{ -42 | abs }}\"
will return 42
) bool
: convert a value to a boolean (e.g. \"{{ 'foo' | bool }}\"
will return true
) default
: return a default value if the value is falsy (e.g. \"{{ '' | default('foo') }}\"
will return \"foo\"
) first
: return the first element of a list (e.g. \"{{ [1, 2, 3] | first }}\"
will return 1
) - last
: return the last element of a list (e.g. \"{{ [1, 2, 3] | last }}\"
will return 3
) length
: return the length of a list (e.g. \"{{ [1, 2, 3] | length }}\"
will return 3
) list
: convert a string to a list (e.g. \"{{ 'foo' | list }}\"
will return ['f', 'o', 'o']
) join
: join a list with a separator (e.g. \"{{ [1, 2, 3] | join('.') }}\"
will return \"1.2.3\"
) min
: return the minimum value of a list (e.g. \"{{ [1, 2, 3] | min }}\"
will return 1
) max
: return the maximum value of a list (e.g. \"{{ [1, 2, 3] | max }}\"
will return 3
) reverse
: reverse a list (e.g. \"{{ [1, 2, 3] | reverse }}\"
will return [3, 2, 1]
) slice
: slice a list (e.g. \"{{ [1, 2, 3] | slice(1, 2) }}\"
will return [2]
) batch
: This filter works pretty much like slice
just the other way round. It returns a list of lists with the given number of items. If you provide a second parameter this is used to fill up missing items. sort
: sort a list (e.g. \"{{ [3, 1, 2] | sort }}\"
will return [1, 2, 3]
) trim
: remove leading and trailing whitespace from a string (e.g. \"{{ ' foo ' | trim }}\"
will return \"foo\"
) unique
: remove duplicates from a list (e.g. \"{{ [1, 2, 1, 3] | unique }}\"
will return [1, 2, 3]
) split
: split a string into a list (e.g. \"{{ '1.2.3' | split('.') | list }}\"
will return ['1', '2', '3']
). By default, splits on whitespace. Removed filters The following filters are removed from the builtins:
attr
indent
select
selectattr
dictsort
reject
rejectattr
round
map
title
capitalize
urlencode
escape
pprint
safe
items
float
tojson
"},{"location":"reference/jinja/#extra-filters-for-recipes","title":"Extra filters for recipes","text":""},{"location":"reference/jinja/#the-version_to_buildstring-filter","title":"The version_to_buildstring
filter","text":" ${{ python | version_to_buildstring }}
converts a version from the variant to a build string (it removes the .
character and takes only the first two elements of the version). For example the following:
context:\n cuda: \"11.2.0\"\n\nbuild:\n string: ${{ hash }}_cuda${{ cuda_version | version_to_buildstring }}\n
Would evaluate to a abc123_cuda112
(assuming the hash was abc123
).
"},{"location":"reference/jinja/#various-remarks","title":"Various remarks","text":""},{"location":"reference/jinja/#inline-conditionals-with-jinja","title":"Inline conditionals with Jinja","text":"The new recipe format allows for inline conditionals with Jinja. If they are falsey, and no else
branch exists, they will render to an empty string (which is, for example in a list or dictionary, equivalent to a YAML null
).
When a recipe is rendered, all values that are null
must be filtered from the resulting YAML.
requirements:\n host:\n - ${{ \"numpy\" if cuda == \"yes\" }}\n
If cuda
is not equal to yes, the first item of the host requirements will be empty (null) and thus filtered from the final list.
This must also work for dictionary values. For example:
build:\n number: ${{ 100 if cuda == \"yes\" }}\n # or an `else` branch can be used, of course\n number: ${{ 100 if cuda == \"yes\" else 0 }}\n
"},{"location":"reference/recipe_file/","title":"The recipe spec","text":"rattler-build
implements a new recipe spec, different from the traditional \"meta.yaml
\" file used in conda-build
. A recipe has to be stored as a recipe.yaml
file.
"},{"location":"reference/recipe_file/#history","title":"History","text":"A discussion was started on what a new recipe spec could or should look like. The fragments of this discussion can be found here.
The reason for a new spec are:
make it easier to parse (i.e. \"pure YAML\"); conda-build
uses a mix of comments and Jinja to achieve a great deal of flexibility, but it's hard to parse the recipe with a computer iron out some inconsistencies around multiple outputs (build
vs. build/script
and more) remove any need for recursive parsing & solving finally, the initial implementation in boa
relied on conda-build
; rattler-build
removes any dependency on Python or conda-build
and reimplements everything in Rust "},{"location":"reference/recipe_file/#major-differences-from-conda-build","title":"Major differences from conda-build
","text":""},{"location":"reference/recipe_file/#spec","title":"Spec","text":"The recipe spec has the following parts:
context
: to set up variables that can later be used in Jinja string interpolation package
: defines name, version etc. of the top-level package source
: points to the sources that need to be downloaded in order to build the recipe build
: defines how to build the recipe and what build number to use requirements
: defines requirements of the top-level package test
: defines tests for the top-level package outputs
: a recipe can have multiple outputs. Each output can and should have a package
, requirements
and test
section "},{"location":"reference/recipe_file/#spec-reference","title":"Spec reference","text":"The spec is also made available through a JSON Schema (which is used for validation). The schema (and pydantic
source file) can be found in this repository: recipe-format
To use with VSCode(yaml-plugin) and other IDEs: Either start the document with the following line:
# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json\n
Or, using yaml.schemas
, yaml.schemas: {\n \"https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json\": \"**/recipe.yaml\",\n}\n
Read more about this here. See more in the automatic linting chapter.
"},{"location":"reference/recipe_file/#examples","title":"Examples","text":"recipe.yaml# this sets up \"context variables\" (in this case name and version) that\n# can later be used in Jinja expressions\ncontext:\n version: 1.1.0\n name: imagesize\n\n# top level package information (name and version)\npackage:\n name: ${{ name }}\n version: ${{ version }}\n\n# location to get the source from\nsource:\n url: https://pypi.io/packages/source/${{ name[0] }}/${{ name }}/${{ name }}-${{ version }}.tar.gz\n sha256: f3832918bc3c66617f92e35f5d70729187676313caa60c187eb0f28b8fe5e3b5\n\n# build number (should be incremented if a new build is made, but version is not incrementing)\nbuild:\n number: 1\n script: python -m pip install .\n\n# the requirements at build and runtime\nrequirements:\n host:\n - python\n - pip\n run:\n - python\n\n# tests to validate that the package works as expected\ntests:\n - python:\n imports:\n - imagesize\n\n# information about the package\nabout:\n homepage: https://github.com/shibukawa/imagesize_py\n license: MIT\n summary: 'Getting image size from png/jpeg/jpeg2000/gif file'\n description: |\n This module analyzes jpeg/jpeg2000/png/gif image header and\n return image size.\n repository: https://github.com/shibukawa/imagesize_py\n documentation: https://pypi.python.org/pypi/imagesize\n\n# the below is conda-forge specific!\nextra:\n recipe-maintainers:\n - somemaintainer\n
"},{"location":"reference/recipe_file/#package-section","title":"Package section","text":"Specifies package information.
package:\n name: bsdiff4\n version: \"2.1.4\"\n
name: The lower case name of the package. It may contain \"-
\", but no spaces. version: The version number of the package. Use the PEP-386 verlib conventions. Cannot contain \"-
\". YAML interprets version numbers such as 1.0 as floats, meaning that 0.10 will be the same as 0.1. To avoid this, put the version number in quotes so that it is interpreted as a string. "},{"location":"reference/recipe_file/#source-section","title":"Source section","text":"Specifies where the source code of the package is coming from. The source may come from a tarball file, git
, hg
, or svn
. It may be a local path and it may contain patches.
"},{"location":"reference/recipe_file/#source-from-tarball-or-zip-archive","title":"Source from tarball or zip
archive","text":"source:\n url: https://pypi.python.org/packages/source/b/bsdiff4/bsdiff4-1.1.4.tar.gz\n md5: 29f6089290505fc1a852e176bd276c43\n sha1: f0a2c9a30073449cfb7d171c57552f3109d93894\n sha256: 5a022ff4c1d1de87232b1c70bde50afbb98212fd246be4a867d8737173cf1f8f\n
If an extracted archive contains only 1 folder at its top level, its contents will be moved 1 level up, so that the extracted package contents sit in the root of the work folder.
"},{"location":"reference/recipe_file/#specifying-a-file-name","title":"Specifying a file name","text":"For URL and local paths you can specify a file name. If the source is an archive and a file name is set, automatic extraction is disabled.
source:\n url: https://pypi.python.org/packages/source/b/bsdiff4/bsdiff4-1.1.4.tar.gz\n # will put the file in the work directory as `bsdiff4-1.1.4.tar.gz`\n file_name: bsdiff4-1.1.4.tar.gz\n
"},{"location":"reference/recipe_file/#source-from-git","title":"Source from git
","text":"source:\n git: https://github.com/ilanschnell/bsdiff4.git\n # branch: master # note: defaults to fetching the repo's default branch\n
You can use rev
to pin the commit version directly:
source:\n git: https://github.com/ilanschnell/bsdiff4.git\n rev: \"50a1f7ed6c168eb0815d424cba2df62790f168f0\"\n
Or you can use the tag
:
source:\n git: https://github.com/ilanschnell/bsdiff4.git\n tag: \"1.1.4\"\n
git
can also be a relative path to the recipe directory:
source:\n git: ../../bsdiff4/.git\n tag: \"1.1.4\"\n
Furthermore, if you want to fetch just the current \"HEAD
\" (this may result in non-deterministic builds), then you can use depth
.
source:\n git: https://github.com/ilanschnell/bsdiff4.git\n depth: 1 # note: the behaviour defaults to -1\n
Note: tag
or rev
may not be available within commit depth range, hence we don't allow using rev
or the tag
and depth
of them together if not set to -1
.
source:\n git: https://github.com/ilanschnell/bsdiff4.git\n tag: \"1.1.4\"\n depth: 1 # error: use of `depth` with `rev` is invalid, they are mutually exclusive\n
When you want to use git-lfs
, you need to set lfs: true
. This will also pull the lfs
files from the repository.
source:\n git: ../../bsdiff4/.git\n tag: \"1.1.4\"\n lfs: true # note: defaults to false\n
"},{"location":"reference/recipe_file/#source-from-a-local-path","title":"Source from a local path","text":"If the path is relative, it is taken relative to the recipe directory. The source is copied to the work directory before building.
source:\n path: ../src\n use_gitignore: false # note: defaults to true\n
By default, all files in the local path that are ignored by git
are also ignored by rattler-build
. You can disable this behavior by setting use_gitignore
to false
.
"},{"location":"reference/recipe_file/#patches","title":"Patches","text":"Patches may optionally be applied to the source.
source:\n #[source information here]\n patches:\n - my.patch # the patch file is expected to be found in the recipe\n
"},{"location":"reference/recipe_file/#destination-path","title":"Destination path","text":"Within rattler-build
's work directory, you may specify a particular folder to place the source into. rattler-build
will always drop you into the same folder ([build folder]/work
), but it's up to you whether you want your source extracted into that folder, or nested deeper. This feature is particularly useful when dealing with multiple sources, but can apply to recipes with single sources as well.
source:\n #[source information here]\n target_directory: my-destination/folder\n
"},{"location":"reference/recipe_file/#source-from-multiple-sources","title":"Source from multiple sources","text":"Some software is most easily built by aggregating several pieces.
The syntax is a list of source dictionaries. Each member of this list follows the same rules as the single source. All features for each member are supported.
Example:
source:\n - url: https://package1.com/a.tar.bz2\n target_directory: stuff\n - url: https://package1.com/b.tar.bz2\n target_directory: stuff\n - git: https://github.com/mamba-org/boa\n target_directory: boa\n
Here, the two URL tarballs will go into one folder, and the git
repo is checked out into its own space. git
will not clone into a non-empty folder.
"},{"location":"reference/recipe_file/#build-section","title":"Build section","text":"Specifies build information.
Each field that expects a path can also handle a glob pattern. The matching is performed from the top of the build environment, so to match files inside your project you can use a pattern similar to the following one: \"**/myproject/**/*.txt\"
. This pattern will match any .txt
file found in your project. Quotation marks (\"\"
) are required for patterns that start with a *
.
Recursive globbing using **
is also supported.
"},{"location":"reference/recipe_file/#build-number-and-string","title":"Build number and string","text":"The build number should be incremented for new builds of the same version. The number defaults to 0
. The build string cannot contain \"-
\". The string defaults to the default rattler-build
build string plus the build number.
build:\n number: 1\n string: abc\n
"},{"location":"reference/recipe_file/#dynamic-linking","title":"Dynamic linking","text":"This section contains settings for the shared libraries and executables.
build:\n dynamic_linking:\n rpath_allowlist: [\"/usr/lib/**\"]\n
"},{"location":"reference/recipe_file/#python-entry-points","title":"Python entry points","text":"The following example creates a Python entry point named \"bsdiff4
\" that calls bsdiff4.cli.main_bsdiff4()
.
build:\n python:\n entry_points:\n - bsdiff4 = bsdiff4.cli:main_bsdiff4\n - bspatch4 = bsdiff4.cli:main_bspatch4\n
"},{"location":"reference/recipe_file/#script","title":"Script","text":"By default, rattler-build
uses a build.sh
file on Unix (macOS and Linux) and a build.bat
file on Windows, if they exist in the same folder as the recipe.yaml
file. With the script parameter you can either supply a different filename or write out short build scripts. You may need to use selectors to use different scripts for different platforms.
build:\n # A very simple build script\n script: pip install .\n\n # The build script can also be a list\n script:\n - pip install .\n - echo \"hello world\"\n - if: unix\n then:\n - echo \"unix\"\n
"},{"location":"reference/recipe_file/#skipping-builds","title":"Skipping builds","text":"Lists conditions under which rattler-build
should skip the build of this recipe. Particularly useful for defining recipes that are platform-specific. By default, a build is never skipped.
build:\n skip:\n - win\n ...\n
"},{"location":"reference/recipe_file/#architecture-independent-packages","title":"Architecture-independent packages","text":"Allows you to specify \"no architecture\" when building a package, thus making it compatible with all platforms and architectures. Architecture-independent packages can be installed on any platform.
Assigning the noarch
key as generic
tells conda
to not try any manipulation of the contents.
build:\n noarch: generic\n
noarch: generic
is most useful for packages such as static JavaScript assets and source archives. For pure Python packages that can run on any Python version, you can use the noarch: python
value instead:
build:\n noarch: python\n
Note
At the time of this writing, noarch
packages should not make use of preprocess-selectors: noarch
packages are built with the directives which evaluate to true
in the platform it is built on, which probably will result in incorrect/incomplete installation in other platforms.
"},{"location":"reference/recipe_file/#include-build-recipe","title":"Include build recipe","text":"The recipe and rendered recipe.yaml
file are included in the package_metadata
by default. You can disable this by passing --no-include-recipe
on the command line.
Note
There are many more options in the build section. These additional options control how variants are computed, prefix replacements, and more. See the full build options for more information.
"},{"location":"reference/recipe_file/#requirements-section","title":"Requirements section","text":"Specifies the build and runtime requirements. Dependencies of these requirements are included automatically.
Versions for requirements must follow the conda
/mamba
match specification. See build-version-spec
.
"},{"location":"reference/recipe_file/#build","title":"Build","text":"Tools required to build the package.
These packages are run on the build system and include things such as version control systems (git
, svn
) make tools (GNU make, Autotool, CMake) and compilers (real cross, pseudo-cross, or native when not cross-compiling), and any source pre-processors.
Packages which provide \"sysroot
\" files, like the CDT
packages (see below), also belong in the build
section.
requirements:\n build:\n - git\n - cmake\n
"},{"location":"reference/recipe_file/#host","title":"Host","text":"Represents packages that need to be specific to the target platform when the target platform is not necessarily the same as the native build platform. For example, in order for a recipe to be \"cross-capable\", shared libraries requirements must be listed in the host
section, rather than the build
section, so that the shared libraries that get linked are ones for the target platform, rather than the native build platform. You should also include the base interpreter for packages that need one. In other words, a Python package would list python
here and an R package would list mro-base
or r-base
.
requirements:\n build:\n - ${{ compiler('c') }}\n - if: linux\n then:\n - ${{ cdt('xorg-x11-proto-devel') }}\n host:\n - python\n
Note
When both \"build
\" and \"host
\" sections are defined, the build
section can be thought of as \"build tools\" - things that run on the native platform, but output results for the target platform (e.g. a cross-compiler that runs on linux-64
, but targets linux-armv7
).
The PREFIX
environment variable points to the host prefix. With respect to activation during builds, both the host and build environments are activated. The build prefix is activated before the host prefix so that the host prefix has priority over the build prefix. Executables that don't exist in the host prefix should be found in the build prefix.
The build
and host
prefixes are always separate when both are defined, or when ${{ compiler() }}
Jinja2 functions are used. The only time that build
and host
are merged is when the host
section is absent, and no ${{ compiler() }}
Jinja2 functions are used in meta.yaml
.
"},{"location":"reference/recipe_file/#run","title":"Run","text":"Packages required to run the package.
These are the dependencies that are installed automatically whenever the package is installed. Package names should follow the package match specifications.
requirements:\n run:\n - python\n - six >=1.8.0\n
To build a recipe against different versions of NumPy and ensure that each version is part of the package dependencies, list numpy
as a requirement in recipe.yaml
and use a conda_build_config.yaml
file with multiple NumPy versions.
"},{"location":"reference/recipe_file/#run-constraints","title":"Run constraints","text":"Packages that are optional at runtime but must obey the supplied additional constraint if they are installed.
Package names should follow the package match specifications.
requirements:\n run_constraints:\n - optional-subpackage ==${{ version }}\n
For example, let's say we have an environment that has package \"a\" installed at version 1.0. If we install package \"b\" that has a run_constraints
entry of \"a >1.0
\", then mamba
would need to upgrade \"a\" in the environment in order to install \"b\".
This is especially useful in the context of virtual packages, where the run_constraints
dependency is not a package that mamba
manages, but rather a virtual package that represents a system property that mamba
can't change. For example, a package on Linux may impose a run_constraints
dependency on __glibc >=2.12
. This is the version bound consistent with CentOS 6. Software built against glibc 2.12 will be compatible with CentOS 6. This run_constraints
dependency helps mamba
, conda
or pixi
tell the user that a given package can't be installed if their system glibc version is too old.
"},{"location":"reference/recipe_file/#run-exports","title":"Run exports","text":"Packages may have runtime requirements such as shared libraries (e.g. zlib
), which are required for linking at build time, and for resolving the link at run time. With run_exports
packages runtime requirements can be implicitly added. run_exports
are weak by default, these two requirements for the zlib
package are therefore equivalent:
recipe.yaml for zlib requirements:\n run_exports:\n - ${{ pin_subpackage('libzlib', exact=True) }}\n
recipe.yaml for zlib requirements:\n run_exports:\n weak:\n - ${{ pin_subpackage('libzlib', exact=True) }}\n
The alternative to weak
is strong
. For gcc
this would look like this:
recipe.yaml for gcc requirements:\n run_exports:\n strong:\n - ${{ pin_subpackage('libgcc', exact=True) }}\n
weak
exports will only be implicitly added as runtime requirement, if the package is a host dependency. strong
exports will be added for both build and host dependencies. In the following example you can see the implicitly added runtime dependencies.
recipe.yaml of some package using gcc and zlib requirements:\n build:\n - gcc # has a strong run export\n host:\n - zlib # has a (weak) run export\n # - libgcc <-- implicitly added by gcc\n run:\n # - libgcc <-- implicitly added by gcc\n # - libzlib <-- implicitly added by libzlib\n
"},{"location":"reference/recipe_file/#ignore-run-exports","title":"Ignore run exports","text":"There maybe cases where an upstream package has a problematic run_exports
constraint. You can ignore it in your recipe by listing the upstream package name in the ignore_run_exports
section in requirements
.
You can ignore them by package name, or by naming the runtime dependency directly.
requirements:\n ignore_run_exports:\n from_package:\n - zlib\n
Using a runtime dependency name:
requirements:\n ignore_run_exports:\n by_name:\n - libzlib\n
Note
ignore_run_exports
only applies to runtime dependencies coming from an upstream package.
"},{"location":"reference/recipe_file/#tests-section","title":"Tests section","text":"rattler-build
supports four different types of tests. The \"script test\" installs the package and runs a list of commands. The \"Python test\" attempts to import a list of Python modules and runs pip check
. The \"downstream test\" runs the tests of a downstream package that reverse depends on the package being built. And lastly, the \"package content test\" checks if the built package contains the mentioned items.
The tests
section is a list of these items:
tests:\n - script:\n - echo \"hello world\"\n requirements:\n run:\n - pytest\n files:\n source:\n - test-data.txt\n\n - python:\n imports:\n - bsdiff4\n pip_check: true # this is the default\n - downstream: numpy\n
"},{"location":"reference/recipe_file/#script-test","title":"Script test","text":"The script test has 3 top-level keys: script
, files
and requirements
. Only the script
key is required.
"},{"location":"reference/recipe_file/#test-commands","title":"Test commands","text":"Commands that are run as part of the test.
tests:\n - script:\n - echo \"hello world\"\n - bsdiff4 -h\n - bspatch4 -h\n
"},{"location":"reference/recipe_file/#extra-test-files","title":"Extra test files","text":"Test files that are copied from the source work directory into the temporary test directory and are needed during testing (note that the source work directory is otherwise not available at all during testing).
You can also include files that come from the recipe
folder. They are copied into the test directory as well.
At test execution time, the test directory is the current working directory.
tests:\n - script:\n - ls\n files:\n source:\n - myfile.txt\n - tests/\n - some/directory/pattern*.sh\n recipe:\n - extra-file.txt\n
"},{"location":"reference/recipe_file/#test-requirements","title":"Test requirements","text":"In addition to the runtime requirements, you can specify requirements needed during testing. The runtime requirements that you specified in the \"run
\" section described above are automatically included during testing (because the built package is installed as it regularly would be).
In the build
section you can specify additional requirements that are only needed on the build system for cross-compilation (e.g. emulators or compilers).
tests:\n - script:\n - echo \"hello world\"\n requirements:\n build:\n - myemulator\n run:\n - nose\n
"},{"location":"reference/recipe_file/#python-tests","title":"Python tests","text":"For this test type you can list a set of Python modules that need to be importable. The test will fail if any of the modules cannot be imported.
The test will also automatically run pip check
to check for any broken dependencies. This can be disabled by setting pip_check: false
in the YAML.
tests:\n - python:\n imports:\n - bsdiff4\n - bspatch4\n pip_check: true # can be left out because this is the default\n
Internally this will write a small Python script that imports the modules:
import bsdiff4\nimport bspatch4\n
"},{"location":"reference/recipe_file/#check-for-package-contents","title":"Check for package contents","text":"Checks if the built package contains the mentioned items. These checks are executed directly at the end of the build process to make sure that all expected files are present in the package.
tests:\n - package_contents:\n # checks for the existence of files inside $PREFIX or %PREFIX%\n # or, checks that there is at least one file matching the specified `glob`\n # pattern inside the prefix\n files:\n - etc/libmamba/test.txt\n - etc/libmamba\n - etc/libmamba/*.mamba.txt\n\n # checks for the existence of `mamba/api/__init__.py` inside of the\n # Python site-packages directory (note: also see Python import checks)\n site_packages:\n - mamba.api\n\n\n # looks in $PREFIX/bin/mamba for unix and %PREFIX%\\Library\\bin\\mamba.exe on Windows\n # note: also check the `commands` and execute something like `mamba --help` to make\n # sure things work fine\n bin:\n - mamba\n\n # searches for `$PREFIX/lib/libmamba.so` or `$PREFIX/lib/libmamba.dylib` on Linux or macOS,\n # on Windows for %PREFIX%\\Library\\lib\\mamba.dll & %PREFIX%\\Library\\bin\\mamba.bin\n lib:\n - mamba\n\n # searches for `$PREFIX/include/libmamba/mamba.hpp` on unix, and\n # on Windows for `%PREFIX%\\Library\\include\\libmamba\\mamba.hpp`\n include:\n - libmamba/mamba.hpp\n
"},{"location":"reference/recipe_file/#downstream-tests","title":"Downstream tests","text":"Warning
Downstream tests are not yet implemented in rattler-build
.
A downstream test can mention a single package that has a dependency on the package being built. The test will install the package and run the tests of the downstream package with our current package as a dependency.
Sometimes downstream packages do not resolve. In this case, the test is ignored.
tests:\n - downstream: numpy\n
"},{"location":"reference/recipe_file/#outputs-section","title":"Outputs section","text":"Explicitly specifies packaging steps. This section supports multiple outputs, as well as different package output types. The format is a list of mappings.
When using multiple outputs, certain top-level keys are \"forbidden\": package
and requirements
. Instead of package
, a top-level recipe
key can be defined. The recipe.name
is ignored but the recipe.version
key is used as default version for each output. Other \"top-level\" keys are merged into each output (e.g. the about
section) to avoid repetition. Each output is a complete recipe, and can have its own build
, requirements
, and test
sections.
recipe:\n # the recipe name is ignored\n name: some\n version: 1.0\n\noutputs:\n - package:\n # version is taken from recipe.version (1.0)\n name: some-subpackage\n\n - package:\n name: some-other-subpackage\n version: 2.0\n
Each output acts like an independent recipe and can have their own script
, build_number
, and so on.
outputs:\n - package:\n name: subpackage-name\n build:\n script: install-subpackage\n
If script
lacks a file extension, the appropriate extension for the platform will be appended, e.g. the above will run install-subpackage.sh
in bash
on most platforms and install-subpackage.bat
in cmd.exe
on Windows.
Each output is built independently. You should take care of not packaging the same files twice.
"},{"location":"reference/recipe_file/#subpackage-requirements","title":"Subpackage requirements","text":"Like a top-level recipe, a subpackage may have zero or more dependencies listed as build, host or run requirements.
The dependencies listed as subpackage build requirements are available only during the packaging phase of that subpackage.
outputs:\n - package:\n name: subpackage-name\n requirements:\n build:\n - some-dep\n run:\n - some-dep\n
You can also use the pin_subpackage
function to pin another output from the same recipe.
outputs:\n - package:\n name: libtest\n - package:\n name: test\n requirements:\n build:\n - ${{ pin_subpackage('libtest', max_pin='x.x') }}\n
The outputs are topologically sorted by the dependency graph which is taking the pin_subpackage
invocations into account. When using pin_subpackage(name, exact=True)
a special behavior is used where the name
package is injected as a \"variant\" and the variant matrix is expanded appropriately. For example, when you have the following situation, with a variant_config.yaml
file that contains openssl: [1, 3]
:
outputs:\n - package:\n name: libtest\n requirements:\n host:\n - openssl\n - package:\n name: test\n requirements:\n build:\n - ${{ pin_subpackage('libtest', exact=True) }}\n
Due to the variant config file, this will build two versions of libtest
. We will also build two versions of test
, one that depends on libtest (openssl 1)
and one that depends on libtest (openssl 3)
.
"},{"location":"reference/recipe_file/#about-section","title":"About section","text":"Specifies identifying information about the package. The information displays in the package server.
about:\n homepage: https://example.com/bsdiff4\n license: BSD-3-Clause # (1)!\n license_file: LICENSE\n summary: binary diff and patch using the BSDIFF4-format\n description: |\n Long description of bsdiff4 ...\n repository: https://github.com/ilanschnell/bsdiff4\n documentation: https://docs.com\n
Only the SPDX specifiers are allowed, more info here: SPDX If you want another license type LicenseRef-<YOUR-LICENSE>
can be used, e.g. license: LicenseRef-Proprietary
"},{"location":"reference/recipe_file/#license-file","title":"License file","text":"Adds a file containing the software license to the package metadata. Many licenses require the license statement to be distributed with the package. The filename is relative to the source or recipe directory. The value can be a single filename or a YAML list for multiple license files. Values can also point to directories with license information. Directory entries must end with a /
suffix (this is to lessen unintentional inclusion of non-license files; all the directory's contents will be unconditionally and recursively added).
about:\n license_file:\n - LICENSE\n - vendor-licenses/\n
"},{"location":"reference/recipe_file/#extra-section","title":"Extra section","text":"A schema-free area for storing non-conda
-specific metadata in standard YAML form.
Example: To store recipe maintainers information extra:\n maintainers:\n - name of maintainer\n
"},{"location":"reference/recipe_file/#templating-with-jinja","title":"Templating with Jinja","text":"rattler-build
supports limited Jinja templating in the recipe.yaml
file.
You can set up Jinja variables in the context
section:
context:\n name: \"test\"\n version: \"5.1.2\"\n # later keys can reference previous keys\n # and use jinja functions to compute new values\n major_version: ${{ version.split('.')[0] }}\n
Later in your recipe.yaml
you can use these values in string interpolation with Jinja:
source:\n url: https://github.com/mamba-org/${{ name }}/v${{ version }}.tar.gz\n
Jinja has built-in support for some common string manipulations.
In rattler-build, complex Jinja is completely disallowed as we try to produce YAML that is valid at all times. So you should not use any {% if ... %}
or similar Jinja constructs that produce invalid YAML. Furthermore, instead of plain double curly brackets Jinja statements need to be prefixed by $
, e.g. ${{ ... }}
:
package:\n name: {{ name }} # WRONG: invalid yaml\n name: ${{ name }} # correct\n
For more information, see the Jinja template documentation and the list of available environment variables env-vars
.
Jinja templates are evaluated during the build process.
"},{"location":"reference/recipe_file/#additional-jinja2-functionality-in-rattler-build","title":"Additional Jinja2 functionality in rattler-build","text":"Besides the default Jinja2 functionality, additional Jinja functions are available during the rattler-build
process: pin_compatible
, pin_subpackage
, and compiler
.
The compiler function takes c
, cxx
, fortran
and other values as argument and automatically selects the right (cross-)compiler for the target platform.
build:\n - ${{ compiler('c') }}\n
The pin_subpackage
function pins another package produced by the recipe with the supplied parameters.
Similarly, the pin_compatible
function will pin a package according to the specified rules.
"},{"location":"reference/recipe_file/#pin-expressions","title":"Pin expressions","text":"rattler-build
knows pin expressions. A pin expression can have a min_pin
, max_pin
and exact
value. A max_pin
and min_pin
are specified with a string containing only x
and .
, e.g. max_pin=\"x.x.x\"
would signify to pin the given package to <1.2.3
(if the package version is 1.2.2
, for example).
A pin with min_pin=\"x.x\",max_pin=\"x.x\"
for a package of version 1.2.2
would evaluate to >=1.2,<1.3.0a0
.
If exact=true
, then the hash
is included, and the package is pinned exactly, e.g. ==1.2.2 h1234
. This is a unique package variant that cannot exist more than once, and thus is \"exactly\" pinned.
"},{"location":"reference/recipe_file/#pin-subpackage","title":"Pin subpackage","text":"Pin subpackage refers to another package from the same recipe file. It is commonly used in the build/run_exports
section to export a run export from the package, or with multiple outputs to refer to a previous build.
It looks something like:
package:\n name: mypkg\n version: \"1.2.3\"\n\nrequirements:\n run_exports:\n # this will evaluate to `mypkg <1.3`\n - ${{ pin_subpackage(name, max_pin='x.x') }}\n
"},{"location":"reference/recipe_file/#pin-compatible","title":"Pin compatible","text":"Pin compatible lets you pin a package based on the version retrieved from the variant file (if the pinning from the variant file needs customization).
For example, if the variant specifies a pin for numpy: 1.11
, one can use pin_compatible
to relax it:
requirements:\n host:\n # this will select nupy 1.11\n - numpy\n run:\n # this will export `numpy >=1.11,<2`, instead of the stricter `1.11` pin\n - ${{ pin_compatible('numpy', min_pin='x.x', max_pin='x') }}\n
"},{"location":"reference/recipe_file/#the-env-jinja-functions","title":"The env Jinja functions","text":"You can access the current environment variables using the env
object in Jinja.
There are three functions:
env.get(\"ENV_VAR\")
will insert the value of \"ENV_VAR\" into the recipe. env.get(\"ENV_VAR\", default=\"undefined\")
will insert the value of ENV_VAR
into the recipe or, if ENV_VAR
is not defined, the specified default value (in this case \"undefined\") env.exists(\"ENV_VAR\")
returns a boolean true of false if the env var is set to any value This can be used for some light templating, for example:
build:\n string: ${{ env.get(\"GIT_BUILD_STRING\") }}_${{ PKG_HASH }}\n
"},{"location":"reference/recipe_file/#match-function","title":"match
function","text":"This function matches the first argument (the package version) against the second argument (the version spec) and returns the resulting boolean. This only works for packages defined in the \"variant_config.yaml\" file.
recipe.yamlmatch(python, '>=3.4')\n
For example, you could require a certain dependency only for builds against python 3.4 and above:
recipe.yamlrequirements:\n build:\n - if: match(python, '>=3.4')\n then:\n - some-dep\n
With a corresponding variant config that looks like the following:
variant_config.yamlpython: [\"3.2\", \"3.4\", \"3.6\"]\n
Example: match
usage example
"},{"location":"reference/recipe_file/#cdt-function","title":"cdt
function","text":"This function helps add Core Dependency Tree packages as dependencies by converting packages as required according to hard-coded logic.
# on x86_64 system\ncdt('package-name') # outputs: package-name-cos6-x86_64\n# on aarch64 system\ncdt('package-name') # outputs: package-name-cos6-aarch64\n
Example: cdt
usage example
"},{"location":"reference/recipe_file/#preprocessing-selectors","title":"Preprocessing selectors","text":"You can add selectors to any item, and the selector is evaluated in a preprocessing stage. If a selector evaluates to true
, the item is flattened into the parent element. If a selector evaluates to false
, the item is removed.
Selectors can use if ... then ... else
as follows:
source:\n - if: not win\n then:\n - url: http://path/to/unix/source\n else:\n - url: http://path/to/windows/source\n\n# or the equivalent with two if conditions:\n\nsource:\n - if: unix\n then:\n - url: http://path/to/unix/source\n - if: win\n then:\n - url: http://path/to/windows/source\n
A selector is a valid Python statement that is executed. You can read more about them in the \"Selectors in recipes\" chapter.
The use of the Python version selectors, py27
, py34
, etc. is discouraged in favor of the more general comparison operators. Additional selectors in this series will not be added to conda-build
.
Because the selector is any valid Python expression, complicated logic is possible:
- if: unix and not win\n then: ...\n- if: (win or linux) and not py27\n then: ...\n
Lists are automatically \"merged\" upwards, so it is possible to group multiple items under a single selector:
tests:\n - script:\n - if: unix\n then:\n - test -d ${PREFIX}/include/xtensor\n - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake\n - if: win\n then:\n - if not exist %LIBRARY_PREFIX%\\include\\xtensor\\xarray.hpp (exit 1)\n - if not exist %LIBRARY_PREFIX%\\lib\\cmake\\xtensor\\xtensorConfigVersion.cmake (exit 1)\n\n# On unix this is rendered to:\ntests:\n - script:\n - test -d ${PREFIX}/include/xtensor\n - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake\n
"},{"location":"reference/recipe_file/#experimental-features","title":"Experimental features","text":"Warning
These are experimental features of rattler-build
and may change or go away completely.
"},{"location":"reference/recipe_file/#jinja-functions","title":"Jinja functions","text":" load_from_file
git.*
functions "},{"location":"tutorials/cpp/","title":"Packaging a C++ package","text":"This tutorial will guide you though making a C++ package with rattler-build
.
"},{"location":"tutorials/cpp/#building-a-header-only-library","title":"Building a Header-only Library","text":"To build a package for the header-only library xtensor
, you need to manage dependencies and ensure proper installation paths.
"},{"location":"tutorials/cpp/#key-steps","title":"Key Steps","text":" Dependencies: Ensure cmake
, ninja
, and a compiler
are available as dependencies.
CMake Installation Prefix: Use the CMAKE_INSTALL_PREFIX
setting to instruct CMake
to install the headers in the correct location.
Unix Systems: Follow the standard Unix prefix:
$PREFIX/include\n$PREFIX/lib\n
Windows Systems: Use a Unix-like prefix but nested in a Library
directory:
$PREFIX/Library/include\n$PREFIX/Library/lib\n
Utilize the handy variables %LIBRARY_PREFIX%
and %LIBRARY_BIN%
to guide CMake
to install the headers and libraries correctly. This approach ensures that the headers and libraries are installed in the correct directories on both Unix and Windows systems.
"},{"location":"tutorials/cpp/#recipe","title":"Recipe","text":"recipe.yamlcontext:\n version: \"0.24.6\"\n\npackage:\n name: xtensor\n version: ${{ version }}\n\nsource:\n url: https://github.com/xtensor-stack/xtensor/archive/${{ version }}.tar.gz\n sha256: f87259b51aabafdd1183947747edfff4cff75d55375334f2e81cee6dc68ef655\n\nbuild:\n number: 0\n script:\n - if: win # (1)!\n then: |\n cmake -GNinja ^\n -D BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% ^\n %SRC_DIR%\n ninja install\n else: |\n cmake -GNinja \\\n -DBUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$PREFIX \\\n $SRC_DIR\n ninja install\n\nrequirements:\n build:\n - ${{ compiler('cxx') }} # (2)!\n - cmake\n - ninja\n host:\n - xtl >=0.7,<0.8\n run:\n - xtl >=0.7,<0.8\n run_constraints: # (3)!\n - xsimd >=8.0.3,<10\n\ntests:\n - package_contents:\n include: # (4)!\n - xtensor/xarray.hpp\n files: # (5)!\n - ${{ \"Library\" if win }}/share/cmake/xtensor/xtensorConfig.cmake\n - ${{ \"Library\" if win }}/share/cmake/xtensor/xtensorConfigVersion.cmake\n\nabout:\n homepage: https://github.com/xtensor-stack/xtensor\n license: BSD-3-Clause\n license_file: LICENSE\n summary: The C++ tensor algebra library\n description: Multi dimensional arrays with broadcasting and lazy computing\n documentation: https://xtensor.readthedocs.io\n repository: https://github.com/xtensor-stack/xtensor\n\nextra:\n recipe-maintainers:\n - some-maintainer\n
The if:
condition allows the user to switch behavior of the build based on some checks like, the operating system. The compiler
function is used to get the C++ compiler for the build system. The run_constraints
section specifies the version range of a package which the package can run \"with\". But which the package doesn't depend on itself. The include
section specifies the header file to tested for existence. The files
section specifies the files to be tested for existence, using a glob pattern. CMAKE_ARGS
It can be tedious to remember all the different variables one needs to pass to CMake to create the perfect build. The cmake
package on conda-forge introduces theCMAKE_ARGS
environment variable. This variable contains the necessary flags to make the package build correctly, also when cross-compiling from one machine to another. Therefore, it is often not necessary to pass any additional flags to the cmake
command. However, because this is a tutorial we will show how to pass the necessary flags to cmake
manually.
For more information please refer to the conda-forge documentation.
"},{"location":"tutorials/cpp/#building-a-c-application","title":"Building A C++ application","text":"In this example, we'll build poppler
, a C++ application for manipulating PDF files from the command line. The final package will install several tools into the bin/
folder. We'll use external build scripts and run actual scripts in the test.
"},{"location":"tutorials/cpp/#key-steps_1","title":"Key Steps","text":" Dependencies:
Build Dependencies: These are necessary for the building process, including cmake
, ninja
, and pkg-config
. Host Dependencies: These are the libraries poppler
links against, such as cairo
, fontconfig
, freetype
, glib
, and others. Compiler Setup: We use the compiler
function to obtain the appropriate C and C++ compilers.
Build Script: The build.script
field points to an external script (poppler-build.sh
) which contains the build commands.
Testing: Simple tests are included to verify that the installed tools (pdfinfo
, pdfunite
, pdftocairo
) are working correctly by running them, and expecting an exit code 0
.
"},{"location":"tutorials/cpp/#recipe_1","title":"Recipe","text":"recipe.yamlcontext:\n version: \"24.01.0\"\n\npackage:\n name: poppler\n version: ${{ version }}\n\nsource:\n url: https://poppler.freedesktop.org/poppler-${{ version }}.tar.xz\n sha256: c7def693a7a492830f49d497a80cc6b9c85cb57b15e9be2d2d615153b79cae08\n\nbuild:\n script: poppler-build.sh\n\nrequirements:\n build:\n - ${{ compiler('c') }} # (1)!\n - ${{ compiler('cxx') }}\n - pkg-config\n - cmake\n - ninja\n host:\n - cairo # (2)!\n - fontconfig\n - freetype\n - glib\n - libboost-headers\n - libjpeg-turbo\n - lcms2\n - libiconv\n - libpng\n - libtiff\n - openjpeg\n - zlib\n\ntests:\n - script:\n - pdfinfo -listenc # (3)!\n - pdfunite --help\n - pdftocairo --help\n
The compiler
jinja function to get the correct compiler for C and C++ on the build system. These are all the dependencies that the library links against. The script test just executes some of the installed tools to check if they are working. These can be as complex as you want. (bash
or cmd.exe
) "},{"location":"tutorials/cpp/#external-build-script","title":"External Build Script","text":"We've defined an external build script in the recipe. This will be searched next to the recipe by the file name given, or the default name build.sh
on unix
or build.bat
on windows are searched for.
poppler-build.sh#! /bin/bash\n\nextra_cmake_args=(\n -GNinja\n -DCMAKE_INSTALL_LIBDIR=lib\n -DENABLE_UNSTABLE_API_ABI_HEADERS=ON\n -DENABLE_GPGME=OFF\n -DENABLE_LIBCURL=OFF\n -DENABLE_LIBOPENJPEG=openjpeg2\n -DENABLE_QT6=OFF\n -DENABLE_QT5=OFF\n -DENABLE_NSS3=OFF\n)\n\nmkdir build && cd build\n\ncmake ${CMAKE_ARGS} \"${extra_cmake_args[@]}\" \\\n -DCMAKE_PREFIX_PATH=$PREFIX \\\n -DCMAKE_INSTALL_PREFIX=$PREFIX \\\n -DTIFF_INCLUDE_DIR=$PREFIX/include \\\n $SRC_DIR\n\nninja\n\n# The `install` command will take care of copying the files to the right place\nninja install\n
"},{"location":"tutorials/cpp/#parsing-the-rattler-build-build-output","title":"Parsing the rattler-build build
Output","text":"When running the rattler-build
command, you might notice some interesting information in the output. Our package will have some run
dependencies, even if we didn't specify any.
These come from the run-exports
of the packages listed in the host
section of the recipe. This is indicated by \"RE of [host: package]\"
in the output.
For example, libcurl
specifies that if you depend on it in the host section, you should also depend on it during runtime with specific version ranges. This ensures proper linking to shared libraries.
Run dependencies:\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 Name \u2506 Spec \u2502\n\u255e\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2561\n\u2502 libcurl \u2506 >=8.5.0,<9.0a0 (RE of [host: libcurl]) \u2502\n\u2502 fontconfig \u2506 >=2.14.2,<3.0a0 (RE of [host: fontconfig]) \u2502\n\u2502 fonts-conda-ecosystem \u2506 (RE of [host: fontconfig]) \u2502\n\u2502 lcms2 \u2506 >=2.16,<3.0a0 (RE of [host: lcms2]) \u2502\n\u2502 gettext \u2506 >=0.21.1,<1.0a0 (RE of [host: gettext]) \u2502\n\u2502 freetype \u2506 >=2.12.1,<3.0a0 (RE of [host: freetype]) \u2502\n\u2502 openjpeg \u2506 >=2.5.0,<3.0a0 (RE of [host: openjpeg]) \u2502\n\u2502 libiconv \u2506 >=1.17,<2.0a0 (RE of [host: libiconv]) \u2502\n\u2502 cairo \u2506 >=1.18.0,<2.0a0 (RE of [host: cairo]) \u2502\n\u2502 libpng \u2506 >=1.6.42,<1.7.0a0 (RE of [host: libpng]) \u2502\n\u2502 libzlib \u2506 >=1.2.13,<1.3.0a0 (RE of [host: zlib]) \u2502\n\u2502 libtiff \u2506 >=4.6.0,<4.7.0a0 (RE of [host: libtiff]) \u2502\n\u2502 libjpeg-turbo \u2506 >=3.0.0,<4.0a0 (RE of [host: libjpeg-turbo]) \u2502\n\u2502 libglib \u2506 >=2.78.3,<3.0a0 (RE of [host: glib]) \u2502\n\u2502 libcxx \u2506 >=16 (RE of [build: clangxx_osx-arm64]) \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
You can also see \"linking\" information in the output, for example on macOS:
[lib/libpoppler-glib.8.26.0.dylib] links against:\n \u251c\u2500 @rpath/libgio-2.0.0.dylib\n \u251c\u2500 @rpath/libgobject-2.0.0.dylib\n \u251c\u2500 /usr/lib/libSystem.B.dylib\n \u251c\u2500 @rpath/libglib-2.0.0.dylib\n \u251c\u2500 @rpath/libpoppler.133.dylib\n \u251c\u2500 @rpath/libfreetype.6.dylib\n \u251c\u2500 @rpath/libc++.1.dylib\n \u251c\u2500 @rpath/libpoppler-glib.8.dylib\n \u2514\u2500 @rpath/libcairo.2.dylib\n
rattler-build
ensures that:
All shared libraries linked against are present in the run dependencies. Missing libraries trigger an overlinking
warning. You don't require any packages in the host that you are not linking against. This triggers an overdepending
warning. "},{"location":"tutorials/go/","title":"Packaging a Go package","text":"This tutorial will guide you through making a Go package with rattler-build
.
When building a recipe for Go, most Go dependencies are linked statically. That means, we should collect their licenses and add them in the package. The go-licenses
tool can help you with this task - as shown in the example below.
"},{"location":"tutorials/go/#the-different-go-compilers","title":"The different Go compilers","text":"The conda-forge
ecosystem provides two go compilers: go-cgo
and go-nocgo
.
By default, if you do not need to link against C libraries, it's recommended to use the go-nocgo
compiler. It generates fat binaries without libc dependencies. The compiler activation scripts will set your CC
, CXX
and related flags to invalid values.
The go-cgo
compiler can generate fat binaries that depend on conda-forge's libc. You should use this compiler if the underlying program needs to link against other C libraries, in which case make sure to add ${{ compiler('c') }}
(cxx
, fortran
, ...) for unix and the m2w64
equivalent for windows.
"},{"location":"tutorials/go/#example-go-recipe","title":"Example Go recipe","text":"This example shows how to package the Temporal CLI.
recipe.yamlcontext:\n version: \"0.13.1\"\n\npackage:\n name: temporal\n version: ${{ version }}\n\nsource:\n url: https://github.com/temporalio/cli/archive/refs/tags/v${{ version }}.tar.gz\n sha256: 9d8812c96d3404490659fec3915dcd23c4142b421ef4cb7e9622bd9a459e1f74\n\nbuild:\n number: 0\n\nrequirements:\n build:\n - ${{ compiler('go-nocgo') }}\n - go-licenses\n\ntests:\n - script:\n - temporal --version\n\nabout:\n homepage: https://temporal.io\n repository: https://github.com/temporalio/cli\n documentation: https://docs.temporal.io/cli\n summary: Temporal CLI\n description: |\n Command-line interface for running Temporal Server and interacting with\n Workflows, Activities, Namespaces, and other parts of Temporal.\n license: MIT\n license_file:\n - LICENSE\n # These license files are generated at build time in the `build.sh` script\n # from all the dependencies of `temporal.io`.\n - license-files/\n
The build script (on Unix) should look something like this:
build.sh# The LDFLAGS are used to set the version of the `temporal` binary. This is a common practice in Go.\nexport LDFLAGS=\"${LDFLAGS} -s -w -X github.com/temporalio/cli/temporalcli.Version=${PKG_VERSION}\"\n\n# Build the `temporal` binary and store it in the `$PREFIX/bin` directory.\ngo build -ldflags \"$LDFLAGS\" -o $PREFIX/bin/temporal ./cmd/temporal\n\n# Store the license files in a separate directory in the $SRC_DIR. These are embedded in the package\n# in the `license_file` section.\ngo-licenses save ./cmd/temporal --save_path=\"$SRC_DIR/license-files/\" || true\n
"},{"location":"tutorials/javascript/","title":"Packaging a Javascript (NPM/NodeJS) package","text":"This tutorial will guide you though making a NodeJS package with rattler-build
. Please note that, while packaging executable applications is possible, the conda ecosystem is not ideal for NPM libraries. NPM supports a number of features that cannot easily be modeled in the conda ecosystem, such as peer dependencies, optional dependencies, and the ability to install multiple versions of the same package.
However, if you need to package a NodeJS application, rattler-build
can help!
"},{"location":"tutorials/javascript/#building-a-nodejs-package","title":"Building a NodeJS Package","text":"In this example, we will build a package for the NodeJS package bibtex-tidy
. We use nodejs
in build and run requirements, and install the package using npm
. NPM comes as part of the NodeJS installation, so we do not need to install it separately.
recipe.yamlcontext:\n version: \"1.14.0\"\n\npackage:\n name: bibtex-tidy\n version: ${{ version }}\n\nsource:\n url: https://registry.npmjs.org/bibtex-tidy/-/bibtex-tidy-${{ version }}.tgz\n sha256: 0a2c1bb73911a7cee36a30ce1fc86feffe39b2d39acd4c94d02aac6f84a00285\n # we do not extract the source code and install the tarball directly as that works better\n file_name: bibtex-tidy-${{ version }}.tgz\n\nbuild:\n number: 0\n script:\n # we use NPM to globally install the bibtex-tidy package\n - npm install -g bibtex-tidy-${{ version }}.tgz --prefix ${{ PREFIX }}\n\nrequirements:\n build:\n - nodejs\n run:\n - nodejs\n\ntests:\n - script:\n - bibtex-tidy --version\n
"},{"location":"tutorials/python/","title":"Writing a Python package","text":"Writing a Python package is fairly straightforward, especially for \"Python-only\" packages. In the second example we will build a package for numpy
which contains compiled code.
"},{"location":"tutorials/python/#a-python-only-package","title":"A Python-only package","text":"The following recipe uses the noarch: python
setting to build a noarch
package that can be installed on any platform without modification. This is very handy for packages that are pure Python and do not contain any compiled extensions.
Additionally, noarch: python
packages work with a range of Python versions (contrary to packages with compiled extensions that are tied to a specific Python version).
recipe.yamlcontext:\n version: \"8.1.2\"\n\npackage:\n name: ipywidgets\n version: ${{ version }}\n\nsource:\n url: https://pypi.io/packages/source/i/ipywidgets/ipywidgets-${{ version }}.tar.gz\n sha256: d0b9b41e49bae926a866e613a39b0f0097745d2b9f1f3dd406641b4a57ec42c9\n\nbuild:\n noarch: python # (1)!\n script: pip install . -v\n\nrequirements:\n # note that there is no build section\n host:\n - pip\n - python >=3.7\n - setuptools\n - wheel\n run:\n - comm >=0.1.3\n - ipython >=6.1.0\n - jupyterlab_widgets >=3.0.10,<3.1.0\n - python >=3.7\n - traitlets >=4.3.1\n - widgetsnbextension >=4.0.10,<4.1.0\n\ntests:\n - python:\n imports:\n - ipywidgets # (2)!\n\nabout:\n homepage: https://github.com/ipython/ipywidgets\n license: BSD-3-Clause\n license_file: LICENSE\n summary: Jupyter Interactive Widgets\n description: |\n ipywidgets are interactive HTML widgets for Jupyter notebooks and the IPython kernel.\n documentation: https://ipywidgets.readthedocs.io/en/latest/\n
The noarch: python
line tells rattler-build
that this package is pure Python and can be one-size-fits-all. noarch
packages can be installed on any platform without modification which is very handy. The imports
section in the tests is used to check that the package is installed correctly and can be imported. "},{"location":"tutorials/python/#running-the-recipe","title":"Running the recipe","text":"To build this recipe, simply run:
rattler-build build --recipe ./ipywidgets\n
"},{"location":"tutorials/python/#a-python-package-with-compiled-extensions","title":"A Python package with compiled extensions","text":"We will build a package for numpy
\u2013 which contains compiled code. Since compiled code is python
version-specific, we will need to specify the python
version explicitly. The best way to do this is with a \"variant_config.yaml\" file:
variants.yamlpython:\n - 3.11\n - 3.12\n
This will replace any python
found in the recipe with the versions specified in the variants.yaml
file.
recipe.yamlcontext:\n version: 2.0.1\n default_abi_level: 1.21\n\npackage:\n name: numpy\n version: ${{ version }}\n\nsource:\n - url: https://github.com/numpy/numpy/releases/download/v${{ version }}/numpy-${{ version }}.tar.gz\n sha256: 485b87235796410c3519a699cfe1faab097e509e90ebb05dcd098db2ae87e7b3\n\nbuild:\n python:\n entry_points:\n - f2py = numpy.f2py.f2py2e:main # [win]\n - numpy-config = numpy._configtool:main\n\nrequirements:\n build:\n - ${{ compiler('c') }}\n - ${{ compiler('cxx') }}\n # note: some `host` dependencies that run at build time (e.g., `cython`, `meson-python`)\n # should ideally be in `build` instead, this is because cross compilation of\n # Python packages in conda-forge uses `crossenv` rather than regular cross compilation.\n host:\n # note: variant is injected here!\n - python\n - pip\n - meson-python\n - pkg-config\n - python-build\n - cython\n - libblas\n - libcblas\n - liblapack\n run:\n - python\n run_exports:\n - numpy >=${{ default_abi_level }},<3.0.0a0\n\ntests:\n - python:\n imports:\n - numpy\n - numpy.fft\n - numpy.linalg\n - numpy.random\n - numpy.ctypeslib\n\n - script:\n - f2py -v\n - numpy-config --cflags\n\nabout:\n homepage: http://numpy.org/\n license: BSD-3-Clause\n license_file: LICENSE.txt\n summary: The fundamental package for scientific computing with Python.\n documentation: https://numpy.org/doc/stable/\n repository: https://github.com/numpy/numpy\n
The build script for Unix:
build.shmkdir builddir\n\n$PYTHON -m build -w -n -x \\\n -Cbuilddir=builddir \\\n -Csetup-args=-Dblas=blas \\\n -Csetup-args=-Dlapack=lapack\n\n$PYTHON -m pip install dist/numpy*.whl\n
The build script for Windows:
build.batmkdir builddir\n\n%PYTHON% -m build -w -n -x ^\n -Cbuilddir=builddir ^\n -Csetup-args=-Dblas=blas ^\n -Csetup-args=-Dlapack=lapack\nif %ERRORLEVEL% neq 0 exit 1\n\n:: `pip install dist\\numpy*.whl` does not work on windows,\n:: so use a loop; there's only one wheel in dist/ anyway\nfor /f %%f in ('dir /b /S .\\dist') do (\n pip install %%f\n if %ERRORLEVEL% neq 0 exit 1\n)\n
"},{"location":"tutorials/python/#running-the-recipe_1","title":"Running the recipe","text":"Running this recipe with the variant config file will build a total of 2 numpy
packages:
rattler-build build --recipe ./numpy\n
At the beginning of the build process, rattler-build
will print the following message to show you the variants it found:
Found variants:\n\nnumpy-1.26.4-py311h5f8ada8_0\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 Variant \u2506 Version \u2502\n\u255e\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2561\n\u2502 python \u2506 3.11 \u2502\n\u2502 target_platform \u2506 osx-arm64 \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\nnumpy-1.26.4-py312h440f24a_0\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 Variant \u2506 Version \u2502\n\u255e\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2561\n\u2502 python \u2506 3.12 \u2502\n\u2502 target_platform \u2506 osx-arm64 \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
"},{"location":"tutorials/rust/","title":"Building a Rust package","text":"We're using rattler-build
to build a Rust package for the cargo-edit
utility. This utility manages Cargo dependencies from the command line.
recipe.yamlcontext:\n version: \"0.11.9\"\n\npackage:\n name: cargo-edit\n version: ${{ version }}\n\nsource:\n url: https://github.com/killercup/cargo-edit/archive/refs/tags/v${{ version }}.tar.gz\n sha256: 46670295e2323fc2f826750cdcfb2692fbdbea87122fe530a07c50c8dba1d3d7\n\nbuild:\n script:\n - cargo-bundle-licenses --format yaml --output ${SRC_DIR}/THIRDPARTY.yml # !(1)\n - $BUILD_PREFIX/bin/cargo install --locked --bins --root ${PREFIX} --path .\n\nrequirements:\n build:\n - ${{ compiler('rust') }}\n - cargo-bundle-licenses\n\ntests:\n - script:\n - cargo-upgrade --help # !(2)\n\nabout:\n homepage: https://github.com/killercup/cargo-edit\n license: MIT\n license_file:\n - LICENSE\n - THIRDPARTY.yml\n description: \"A utility for managing cargo dependencies from the command line.\"\n summary: \"A utility for managing cargo dependencies from the command line.\"\n
Note
The ${{ compiler(...) }}
functions are very useful in the context of cross-compilation. When the function is evaluated it will insert the correct compiler (as selected with the variant config) as well the target_platform
. The \"rendered\" compiler will look like rust_linux-64
when you are targeting the linux-64
platform.
You can read more about this in the cross-compilation section.
The cargo-bundle-licenses
utility is used to bundle all the licenses of the dependencies into a THIRDPARTY.yml
file. This file is then included in the package. You should always include this file in your package when you are redistributing it. Running scripts in bash
or cmd.exe
to test the package build well, expects an exit code of 0
to pass the test. To build this recipe, simply run:
rattler-build build \\\n --recipe ./cargo-edit/recipe.yaml\n
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"#rattler-build-a-fast-conda-package-builder","title":"rattler-build
: A Fast Conda Package Builder","text":"The rattler-build
tooling and library creates cross-platform relocatable binaries / packages from a simple recipe format. The recipe format is heavily inspired by conda-build
and boa
, and the output of a regular rattler-build
run is a package that can be installed using mamba
, rattler
or conda
.
rattler-build
does not have any dependencies on conda-build
or Python and works as a standalone binary.
"},{"location":"#installation","title":"Installation","text":"The recommended way of installing rattler-build
, being a conda-package builder, is through a conda package manager. Next to rattler-build
we are also building pixi
.
With pixi
you can install rattler-build
globally:
pixi global install rattler-build\n
Other options are:
CondaHomebrewArch LinuxBinary conda install rattler-build -c conda-forge\n\nmamba install rattler-build -c conda-forge\nmicromamba install rattler-build -c conda-forge\n\npixi global install rattler-build\npixi add rattler-build # To a pixi project\n
brew install rattler-build\n
pacman -S rattler-build\n
# Download the latest release from the GitHub releases page, for example the linux x86 version with curl:\ncurl -SL --progress-bar https://github.com/prefix-dev/rattler-build/releases/latest/download/rattler-build-x86_64-unknown-linux-musl\n
You can grab version of rattler-build
from the Github Releases."},{"location":"#completion","title":"Completion","text":"When installing rattler-build
you might want to enable shell completion. Afterwards, restart the shell or source the shell config file.
"},{"location":"#bash-default-on-most-linux-systems","title":"Bash (default on most Linux systems)","text":"echo 'eval \"$(rattler-build completion --shell bash)\"' >> ~/.bashrc\n
"},{"location":"#zsh-default-on-macos","title":"Zsh (default on macOS)","text":"echo 'eval \"$(rattler-build completion --shell zsh)\"' >> ~/.zshrc\n
"},{"location":"#powershell-pre-installed-on-all-windows-systems","title":"PowerShell (pre-installed on all Windows systems)","text":"Add-Content -Path $PROFILE -Value '(& rattler-build completion --shell powershell) | Out-String | Invoke-Expression'\n
Failure because no profile file exists
Make sure your profile file exists, otherwise create it with:
New-Item -Path $PROFILE -ItemType File -Force\n
"},{"location":"#fish","title":"Fish","text":"echo 'rattler-build completion --shell fish | source' >> ~/.config/fish/config.fish\n
"},{"location":"#nushell","title":"Nushell","text":"Add the following to the end of your Nushell env file (find it by running $nu.env-path
in Nushell):
mkdir ~/.cache/rattler-build\nrattler-build completion --shell nushell | save -f ~/.cache/rattler-build/completions.nu\n
And add the following to the end of your Nushell configuration (find it by running $nu.config-path
):
use ~/.cache/rattler-build/completions.nu *\n
"},{"location":"#elvish","title":"Elvish","text":"echo 'eval (rattler-build completion --shell elvish | slurp)' >> ~/.elvish/rc.elv\n
"},{"location":"#dependencies","title":"Dependencies","text":"Currently rattler-build
needs some dependencies on the host system which are executed as subprocess. We plan to reduce the number of external dependencies over time by writing what we need in Rust to make rattler-build
fully self-contained.
tar
to unpack tarballs downloaded from the internet in a variety of formats. .gz
, .bz2
and .xz
are widely used and one might have to install the compression packages as well (e.g. gzip
, bzip2
, ...) patch
to patch source code after downloading install_name_tool
is necessary on macOS to rewrite the rpath
of shared libraries and executables to make it relative patchelf
is required on Linux to rewrite the rpath
and runpath
of shared libraries and executables git
to checkout Git repositories (not implemented yet, but will require git
in the future) msvc
on Windows because we cannot ship the MSVC compiler on conda-forge (needs to be installed on the host machine) On Windows, to obtain these dependencies from conda-forge, one can install m2-patch
, m2-bzip2
, m2-gzip
, m2-tar
.
"},{"location":"#github-action","title":"GitHub Action","text":"There is a GitHub Action for rattler-build
. It can be used to install rattler-build
in CI/CD workflows and run a build command. Please check out the GitHub Action documentation for more information.
"},{"location":"#the-recipe-format","title":"The recipe format","text":"Note You can find all examples below in the examples
folder in the codebase and run them with rattler-build
.
A simple example recipe for the xtensor
header-only C++ library:
# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json\n\ncontext:\n name: xtensor\n version: 0.24.6\n\npackage:\n name: ${{ name|lower }}\n version: ${{ version }}\n\nsource:\n url: https://github.com/xtensor-stack/xtensor/archive/${{ version }}.tar.gz\n sha256: f87259b51aabafdd1183947747edfff4cff75d55375334f2e81cee6dc68ef655\n\nbuild:\n number: 0\n script:\n - if: win\n then: |\n cmake -G \"NMake Makefiles\" -D BUILD_TESTS=OFF -D CMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% %SRC_DIR%\n nmake\n nmake install\n else: |\n cmake ${CMAKE_ARGS} -DBUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$PREFIX $SRC_DIR -DCMAKE_INSTALL_LIBDIR=lib\n make install\n\nrequirements:\n build:\n - ${{ compiler('cxx') }}\n - cmake\n - if: unix\n then: make\n host:\n - xtl >=0.7,<0.8\n run:\n - xtl >=0.7,<0.8\n run_constraints:\n - xsimd >=8.0.3,<10\n\ntests:\n - script:\n - if: unix or emscripten\n then:\n - test -d ${PREFIX}/include/xtensor\n - test -f ${PREFIX}/include/xtensor/xarray.hpp\n - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfig.cmake\n - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfigVersion.cmake\n - if: win\n then:\n - if not exist %LIBRARY_PREFIX%\\include\\xtensor\\xarray.hpp (exit 1)\n - if not exist %LIBRARY_PREFIX%\\share\\cmake\\xtensor\\xtensorConfig.cmake (exit 1)\n - if not exist %LIBRARY_PREFIX%\\share\\cmake\\xtensor\\xtensorConfigVersion.cmake (exit 1)\n\nabout:\n homepage: https://github.com/xtensor-stack/xtensor\n license: BSD-3-Clause\n license_file: LICENSE\n summary: The C++ tensor algebra library\n description: Multi dimensional arrays with broadcasting and lazy computing\n documentation: https://xtensor.readthedocs.io\n repository: https://github.com/xtensor-stack/xtensor\n\nextra:\n recipe-maintainers:\n - some-maintainer\n
A recipe for the rich
Python package (using noarch
):
context:\n version: \"13.4.2\"\n\npackage:\n name: \"rich\"\n version: ${{ version }}\n\nsource:\n - url: https://pypi.io/packages/source/r/rich/rich-${{ version }}.tar.gz\n sha256: d653d6bccede5844304c605d5aac802c7cf9621efd700b46c7ec2b51ea914898\n\nbuild:\n # Thanks to `noarch: python` this package works on all platforms\n noarch: python\n script:\n - python -m pip install . -vv\n\nrequirements:\n host:\n - pip\n - poetry-core >=1.0.0\n - python 3.10\n run:\n # sync with normalized deps from poetry-generated setup.py\n - markdown-it-py >=2.2.0\n - pygments >=2.13.0,<3.0.0\n - python 3.10\n - typing_extensions >=4.0.0,<5.0.0\n\ntests:\n - python:\n imports:\n - rich\n pip_check: true\n\nabout:\n homepage: https://github.com/Textualize/rich\n license: MIT\n license_file: LICENSE\n summary: Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal\n description: |\n Rich is a Python library for rich text and beautiful formatting in the terminal.\n\n The Rich API makes it easy to add color and style to terminal output. Rich\n can also render pretty tables, progress bars, markdown, syntax highlighted\n source code, tracebacks, and more \u2014 out of the box.\n documentation: https://rich.readthedocs.io\n repository: https://github.com/Textualize/rich\n
A recipe for the curl
library:
context:\n version: \"8.0.1\"\n\npackage:\n name: curl\n version: ${{ version }}\n\nsource:\n url: http://curl.haxx.se/download/curl-${{ version }}.tar.bz2\n sha256: 9b6b1e96b748d04b968786b6bdf407aa5c75ab53a3d37c1c8c81cdb736555ccf\n\nbuild:\n number: 0\n\nrequirements:\n build:\n - ${{ compiler('c') }}\n - if: win\n then:\n - cmake\n - ninja\n - if: unix\n then:\n - make\n - perl\n - pkg-config\n - libtool\n host:\n - if: linux\n then:\n - openssl\n\nabout:\n homepage: http://curl.haxx.se/\n license: MIT/X derivate (http://curl.haxx.se/docs/copyright.html)\n license_file: COPYING\n summary: tool and library for transferring data with URL syntax\n description: |\n Curl is an open source command line tool and library for transferring data\n with URL syntax. It is used in command lines or scripts to transfer data.\n documentation: https://curl.haxx.se/docs/\n repository: https://github.com/curl/curl\n
For the curl
library recipe, two additional script files (build.sh
and build.bat
) are needed.
build.sh
#!/bin/bash\n\n# Get an updated config.sub and config.guess\ncp $BUILD_PREFIX/share/libtool/build-aux/config.* .\n\nif [[ $target_platform =~ linux.* ]]; then\n USESSL=\"--with-openssl=${PREFIX}\"\nelse\n USESSL=\"--with-secure-transport\"\nfi;\n\n./configure \\\n --prefix=${PREFIX} \\\n --host=${HOST} \\\n ${USESSL} \\\n --with-ca-bundle=${PREFIX}/ssl/cacert.pem \\\n --disable-static --enable-shared\n\nmake -j${CPU_COUNT} ${VERBOSE_AT}\nmake install\n\n# Includes man pages and other miscellaneous.\nrm -rf \"${PREFIX}/share\"\n
build.bat
mkdir build\n\ncmake -GNinja ^\n -DCMAKE_BUILD_TYPE=Release ^\n -DBUILD_SHARED_LIBS=ON ^\n -DCMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% ^\n -DCMAKE_PREFIX_PATH=%LIBRARY_PREFIX% ^\n -DCURL_USE_SCHANNEL=ON ^\n -DCURL_USE_LIBSSH2=OFF ^\n -DUSE_ZLIB=ON ^\n -DENABLE_UNICODE=ON ^\n %SRC_DIR%\n\nIF %ERRORLEVEL% NEQ 0 exit 1\n\nninja install --verbose\n
"},{"location":"authentication_and_upload/","title":"Server authentication","text":""},{"location":"authentication_and_upload/#authenticating-with-a-server","title":"Authenticating with a server","text":"You may want to use private channels for which you need to be authenticated. To do this ephemerally you can use the RATTLER_AUTH_FILE
environment variable to point to a JSON file with the following structure:
{\n \"*.prefix.dev\": {\n \"BearerToken\": \"your_token\"\n },\n \"otherhost.com\": {\n \"BasicHTTP\": {\n \"username\": \"your_username\",\n \"password\": \"your_password\"\n }\n },\n \"anaconda.org\": {\n \"CondaToken\": \"your_token\"\n }\n}\n
The keys are the host names. You can use wildcard specifiers here (e.g. *.prefix.dev
to match all subdomains of prefix.dev
, such as repo.prefix.dev
). This will allow you to also obtain packages from any private channels that you have access to.
The following known authentication methods are supported:
BearerToken
: prefix.dev CondaToken
: anaconda.org, quetz BasicHTTP
: artifactory "},{"location":"authentication_and_upload/#uploading-packages","title":"Uploading packages","text":"If you want to upload packages, then rattler-build comes with a built-in upload
command. There are 4 options:
prefix.dev
: you can create public or private channels on the prefix.dev hosted server anaconda.org
: you can upload packages to the free anaconda.org server quetz
: you can host your own quetz server and upload packages to it artifactory
: you can upload packages to a JFrog Artifactory server The command is:
rattler-build upload <server> <package_files>\n
Note: you can also use the RATTLER_AUTH_FILE
environment variable to authenticate with the server.
"},{"location":"authentication_and_upload/#prefixdev","title":"prefix.dev","text":""},{"location":"authentication_and_upload/#trusted-publishing-via-oidc","title":"Trusted publishing via OIDC","text":"rattler-build
supports authentication with https://prefix.dev through OIDC with GitHub Actions. An API key is no longer required, rattler-build can manage the complete authentication workflow for you. You only have to set up a specific repository and workflow under \"Trusted Publishers\" on prefix.dev.
Here you can find an example GitHub Actions workflow
.github/workflows/build.ymlpermissions:\n contents: read\n id-token: write\n\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - name: Build conda package\n uses: prefix-dev/rattler-build-action@v0.2.19\n\n - name: Upload all packages\n shell: bash\n run: |\n shopt -s nullglob\n EXIT_CODE=0\n for pkg in $(find output -type f \\( -name \"*.conda\" -o -name \"*.tar.bz2\" \\) ); do\n if ! rattler-build upload prefix -c my-channel \"${pkg}\"; then\n EXIT_CODE=1\n fi\n done\n exit $EXIT_CODE\n
"},{"location":"authentication_and_upload/#token","title":"Token","text":"To upload to prefix.dev, you need to have an account. You can then create a token in the settings of your account. The token is used to authenticate the upload.
export PREFIX_API_KEY=<your_token>\nrattler-build upload prefix -c <channel> <package_files>\n
You can also use the --api-key=$PREFIX_API_KEY
option to pass the token directly to the command. Note that you need to have created the channel on the prefix.dev website before you can upload to it.
"},{"location":"authentication_and_upload/#quetz","title":"Quetz","text":"You need to pass a token and API key to upload to a channel on your own Quetz server. The token is used to authenticate the upload.
export QUETZ_API_KEY=<your_token>\nrattler-build upload quetz -u <url> -c <channel> <package_files>\n
"},{"location":"authentication_and_upload/#artifactory","title":"Artifactory","text":"To upload to an Artifactory server, you need to pass a username and password. The username and password are used to authenticate the upload.
export ARTIFACTORY_USERNAME=<your_username>\nexport ARTIFACTORY_PASSWORD=<your_password>\nrattler-build upload artifactory -u <url> -c <channel> <package_files>\n
"},{"location":"authentication_and_upload/#anacondaorg","title":"anaconda.org","text":"To upload to anaconda.org, you need to specify the owner and API key. The API key is used to authenticate the upload.
The owner is the owner of the distribution, for example, your user name or organization.
One can also specify a label such as dev
for release candidates using the -c
flag. The default value is main
.
You can also add the --force
argument to forcibly upload a new package (and overwrite any existing ones).
export ANACONDA_API_KEY=<your_token>\nrattler-build upload anaconda -o <your_username> -c <label> <package_files>\n
"},{"location":"automatic_linting/","title":"Enabling Automatic Linting in VSCode","text":"Our new recipe format adheres to a strict JSON schema, which you can access here.
This schema is implemented using pydantic
and can be rendered into a JSON schema file. The YAML language server extension in VSCode is capable of recognizing this schema, providing useful hints during the editing process.
To enable automatic linting with the YAML language server, you need to add the following line at the beginning of your recipe file:
# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json\n
Alternatively, if you prefer not to add this line to your file, you can install the JSON Schema Store Catalog extension. This extension will also enable automatic linting for your recipe files.
"},{"location":"build_options/","title":"Advanced build options","text":"There are some specialized build options to control various features:
prefix replacement variant configuration encoded file type These are all found under the build
key in the recipe.yaml
.
"},{"location":"build_options/#include-only-certain-files-in-the-package","title":"Include only certain files in the package","text":"Sometimes you may want to include only a subset of the files installed by the build process in your package. For this, the files
key can be used. Only new files are considered for inclusion (ie. files that were not in the host environment beforehand).
recipe.yamlbuild:\n # select files to be included in the package\n # this can be used to remove files from the package, even if they are installed in the\n # environment\n files: list of globs\n
For example, to only include the header files in a package, you could use:
recipe.yamlbuild:\n files:\n - include/**/*.h\n
Glob patterns throughout the recipe file can also use a flexible include
/ exclude
pair, such as:
recipe.yamlbuild:\n files:\n include:\n - include/**/*.h\n exclude:\n - include/**/private.h\n
"},{"location":"build_options/#glob-evaluation","title":"Glob evaluation","text":"Glob patterns are used throughout the build options to specify files. The patterns are matched against the relative path of the file in the build directory. Patterns can contain *
to match any number of characters, ?
to match a single character, and **
to match any number of directories.
For example:
*.txt
matches all files ending in .txt
**/*.txt
matches all files ending in .txt
in any directory **/test_*.txt
matches all files starting with test_
and ending in .txt
in any directory foo/
matches all files under the foo
directory The globs are always evaluated relative to the prefix directory. If you have no include
globs, but an exclude
glob, then all files are included except those that match the exclude
glob. This is equivalent to include: ['**']
.
"},{"location":"build_options/#always-include-and-always-copy-files","title":"Always include and always copy files","text":"There are some options that control the inclusion of files in the final package.
The always_include_files
option can be used to include files even if they are already in the environment as part of some other host dependency. This is normally \"clobbering\" and should be used with caution (since packages should not have any overlapping files).
The always_copy_files
option can be used to copy files instead of linking them. This is useful for files that might be modified inside the environment (e.g. configuration files). Normally, files are linked from a central cache into the environment to save space \u2013 that means that files modified in one environment will be modified in all environments. This is not always desirable, and in that case you can use the always_copy_files
option.
??? note \"How always_copy_files
works\" The always_copy_files
option works by setting the no_link
option in the info/paths.json
to true
for the files in question. This means that the files are copied instead of linked when the package is installed.
recipe.yamlbuild:\n # include files even if they are already in the environment\n # as part of some other host dependency\n always_include_files: list of globs\n\n # do not soft- or hard-link these files, but always copy them was `no_link`\n always_copy_files: list of globs\n
"},{"location":"build_options/#merge-build-and-host-environments","title":"Merge build and host environments","text":"In very rare cases you might want to merge the build and host environments to obtain the \"legacy\" behavior of conda-build.
recipe.yamlbuild:\n # merge the build and host environments (used in many R packages on Windows)\n merge_build_and_host_envs: bool (defaults to false)\n
"},{"location":"build_options/#prefix-detection-replacement-options","title":"Prefix detection / replacement options","text":"During installation time the \"install\"-prefix is injected into text and binary files. Sometimes this is not desired, and sometimes the user might want closer control over the automatic text/binary detection.
The main difference between prefix replacement for text and binary files is that for binary files, the prefix string is padded with null bytes to match the length of the original prefix. The original prefix is the very long placeholder string that you might have seen in the build process.
On Windows, binary prefix replacement is never performed.
recipe.yamlpackage:\n name: mypackage\n version: 1.0\n\nbuild:\n # settings concerning the prefix detection in files\n prefix_detection:\n # force the file type of the given files to be TEXT or BINARY\n # for prefix replacement\n force_file_type:\n # force TEXT file type (list of globs)\n text: list of globs\n # force binary file type (list of globs)\n binary: list of globs\n\n # ignore all or specific files for prefix replacement`\n ignore: bool | [path] (defaults to false)\n\n # whether to detect binary files with prefix or not\n # defaults to true on Unix and (always) false on Windows\n ignore_binary_files: bool\n
"},{"location":"build_options/#variant-configuration","title":"Variant configuration","text":"To control the variant precisely you can use the \"variant configuration\" options.
A variant package has the same version number, but different \"hash\" and potentially different dependencies or build options. Variant keys are extracted from the variant_config.yaml
file and usually any used Jinja variables or dependencies without version specifier are used as variant keys.
Variant keys can also be forcibly set or ignored with the use_keys
and ignore_keys
options.
In order to decide which of the variant packages to prefer and install by default, the down_prioritize_variant
option can be used. The higher the value, the less preferred the variant is.
More about variants can be found in the variant documentation.
The following options are available in the build
section to control the variant configuration:
recipe.yamlbuild:\n # settings for the variant\n variant:\n # Keys to forcibly use for the variant computation\n # even if they are not in the dependencies\n use_keys: list of strings\n\n # Keys to forcibly ignore for the variant computation\n # even if they are in the dependencies\n ignore_keys: list of strings\n\n # used to prefer this variant less\n down_prioritize_variant: integer (defaults to 0, higher is less preferred)\n
"},{"location":"build_options/#dynamic-linking-configuration","title":"Dynamic linking configuration","text":"After the package is built, rattler-build performs some \"post-processing\" on the binaries and libraries.
This entails making the shared libraries relocatable and checking that all linked libraries are present in the run requirements. The following settings control this behavior.
With the rpath
option you can forcibly set the rpath
of the shared libraries. The path is relative to the install prefix. Any rpath
setting is ignored on Windows.
The rpath_allowlist
option can be used to allow the rpath
to point to locations outside of the environment. This is useful if you want to link against libraries that are not part of the conda environment (e.g. proprietary software).
If you want to stop rattler-build
from relocating the binaries, you can set binary_relocation
to false
. If you want to only relocate some binaries, you can select the relevant ones with a glob pattern.
To read more about rpath
s and how rattler-build creates relocatable binary packages, see the internals docs.
If you link against some libraries (possibly even outside of the prefix, in a system location), then you can use the missing_dso_allowlist
to allow linking against these and suppress any warnings. This list is pre-populated with a list of known system libraries on the different operating systems.
As part of the post-processing, rattler-build
checks for overlinking and overdepending. \"Overlinking\" is when a binary links against a library that is not specified in the run requirements. This is usually a mistake because the library would not be present in the environment when the package is installed.
Conversely, \"overdepending\" is when a library is part of the run requirements, but is not actually used by any of the binaries/libraries in the package.
recipe.yamlbuild:\n # settings for shared libraries and executables\n dynamic_linking:\n # linux only, list of rpaths relative to the installation prefix\n rpaths: list of paths (defaults to ['lib/'])\n\n # Allow runpath / rpath to point to these locations\n # outside of the environment\n rpath_allowlist: list of globs\n\n # whether to relocate binaries or not. If this is a list of paths, then\n # only the listed paths are relocated\n binary_relocation: bool (defaults to true) | list of globs\n\n # Allow linking against libraries that are not in the run requirements\n missing_dso_allowlist: list of globs\n\n # what to do when detecting overdepending\n overdepending_behavior: \"ignore\" or \"error\" # (defaults to \"error\")\n\n # what to do when detecting overlinking\n overlinking_behavior: \"ignore\" or \"error\" # (defaults to \"error\")\n
"},{"location":"build_script/","title":"Build scripts","text":"The build.sh
file is the build script for Linux and macOS and build.bat
is the build script for Windows. These scripts contain the logic that carries out your build steps. Anything that your build script copies into the $PREFIX
or %PREFIX%
folder will be included in your output package.
For example, this build.sh
:
build.shmkdir -p $PREFIX/bin\ncp $RECIPE_DIR/my_script_with_recipe.sh $PREFIX/bin/super-cool-script.sh\n
There are many environment variables defined for you to use in build.sh and build.bat. Please see environment variables for more information.
build.sh
and build.bat
are optional. You can instead use the build/script
key in your recipe.yaml
, with each value being either a string command or a list of string commands. Any commands you put there must be able to run on every platform for which you build. For example, you can't use the cp
command because cmd.exe
won't understand it on Windows.
build.sh
is run with bash
and build.bat
is run with cmd.exe
.
recipe.yamlbuild:\n script:\n - if: unix\n then:\n - mkdir -p $PREFIX/bin\n - cp $RECIPE_DIR/my_script_with_recipe.sh $PREFIX/bin/super-cool-script.sh\n - if: win\n then:\n - mkdir %PREFIX%\\bin\n - copy %RECIPE_DIR%\\my_script_with_recipe.bat %PREFIX%\\bin\\super-cool-script.bat\n
"},{"location":"build_script/#environment-variables","title":"Environment variables","text":"There are many environment variables that are automatically set during the build process.
However, you can also set your own environment variables easily in the script
section of your recipe:
recipe.yamlbuild:\n script:\n # Either use `content` or `file` to specify the script\n # Note: this script only works on Unix :)\n content: |\n echo $FOO\n echo $BAR\n echo \"Secret value: $BAZ\"\n env:\n # hard coded value for `FOO`\n FOO: \"foo\"\n # Forward a value from the \"outer\" environment\n # Without `default=...`, the build process will error if `BAR` is not set\n BAR: ${{ env.get(\"BAR\", default=\"NOBAR\") }}\n secrets:\n # This value is a secret and will be masked in the logs and not stored in the rendered recipe\n # The value needs to be available as an environment variable in the outer environment\n - BAZ\n
"},{"location":"build_script/#alternative-script-interpreters","title":"Alternative script interpreters","text":"With rattler-build
and the new recipe syntax you can select an interpreter
for your script.
So far, the following interpreters are supported:
bash
(default on Unix) cmd.exe
(default on Windows) nushell
python
Note
Using alternative interpreters is less battle-tested than using bash
or cmd.exe
. If you encounter any issues, please open an issue.
"},{"location":"build_script/#using-nushell","title":"Using nushell
","text":"In order to use nushell
you can select the interpreter: nu
or have a build.nu
file in your recipe directory. Nushell works on Windows, macOS and Linux with the same syntax.
recipe.yamlbuild:\n script:\n interpreter: nu\n content: |\n echo \"Hello from nushell!\"\n\n# Note: it's required to have `nushell` in the `build` section of your recipe!\nrequirements:\n build:\n - nushell\n
"},{"location":"build_script/#using-python","title":"Using python
","text":"In order to use python
you can select the interpreter: python
or have a build.py
file in your recipe directory and python
in the requirements/build
section.
recipe.yamlbuild:\n script:\n interpreter: python\n content: |\n print(\"Hello from Python!\")\n\n# Note: it's required to have `python` in the `build` section of your recipe!\nrequirements:\n build:\n - python\n
"},{"location":"build_script/#default-environment-variables-set-during-the-build-process","title":"Default environment variables set during the build process","text":"During the build process, the following environment variables are set, on Windows with build.bat
and on macOS and Linux with build.sh
. By default, these are the only variables available to your build script. Unless otherwise noted, no variables are inherited from the shell environment in which you invoke conda-build
. To override this behavior, see :ref:inherited-env-vars
.
ARCH
Either 32
or 64
, to specify whether the build is 32-bit or 64-bit. The value depends on the ARCH environment variable and defaults to the architecture the interpreter running conda was compiled with.
CMAKE_GENERATOR
The CMake generator string for the current build environment. On Linux systems, this is always Unix Makefiles
. On Windows, it is generated according to the Visual Studio version activated at build time, for example, Visual Studio 9 2008 Win64
.
CONDA_BUILD=1
Always set to indicate that the conda-build process is running.
CPU_COUNT
Represents the number of CPUs on the system.
SHLIB_EXT
Denotes the shared library extension specific to the operating system (e.g. .so
for Linux, .dylib
for macOS, and .dll
for Windows).
HTTP_PROXY
, HTTPS_PROXY
Inherited from the user's shell environment, specifying the HTTP and HTTPS proxy settings.
LANG
Inherited from the user's shell environment, defining the system language and locale settings.
MAKEFLAGS
Inherited from the user's shell environment. This can be used to set additional arguments for the make command, such as -j2 to utilize 2 CPU cores for building the recipe.
PY_VER
Specifies the Python version against which the build is occurring. This can be modified with a variant_config.yaml
file.
PATH
Inherited from the user's shell environment and augmented with the activated host and build prefixes.
PREFIX
The build prefix to which the build script should install the software.
PKG_BUILDNUM
Indicates the build number of the package currently being built.
PKG_NAME
The name of the package that is being built.
PKG_VERSION
The version of the package currently under construction.
PKG_BUILD_STRING
The complete build string of the package being built, including the hash (e.g. py311h21422ab_0).
PKG_HASH
Represents the hash of the package being built, excluding the leading 'h' (e.g. 21422ab). This is applicable from conda-build 3.0 onwards.
PYTHON
The path to the Python executable in the host prefix. Python is installed in the host prefix only when it is listed as a host requirement.
R
The path to the R executable in the build prefix. R is installed in the build prefix only when it is listed as a build requirement.
RECIPE_DIR
The directory where the recipe is located.
SP_DIR
The location of Python's site-packages, where Python libraries are installed.
SRC_DIR
The path to where the source code is unpacked or cloned. If the source file is not a recognized archive format, this directory contains a copy of the source file.
STDLIB_DIR
The location of Python's standard library.
build_platform
Represents the native subdirectory of the conda executable, indicating the platform for which the build is occurring.
Removed from conda-build
are: - NPY_VER
- PY3K
"},{"location":"build_script/#windows","title":"Windows","text":"Unix-style packages on Windows are built in a special Library
directory under the build prefix. The environment variables listed in the following table are defined only on Windows.
Variable Description LIBRARY_BIN
<build prefix>\\Library\\bin
. LIBRARY_INC
<build prefix>\\Library\\include
. LIBRARY_LIB
<build prefix>\\Library\\lib
. LIBRARY_PREFIX
<build prefix>\\Library
. SCRIPTS
<build prefix>\\Scripts
. Not yet supported in rattler-build
:
CYGWIN_PREFIX
VS_MAJOR
VS_VERSION
VS_YEAR
Additionally, the following variables are forwarded from the environment:
ALLUSERSPROFILE
APPDATA
CommonProgramFiles
CommonProgramFiles(x86)
CommonProgramW6432
COMPUTERNAME
ComSpec
HOMEDRIVE
HOMEPATH
LOCALAPPDATA
LOGONSERVER
NUMBER_OF_PROCESSORS
PATHEXT
ProgramData
ProgramFiles
ProgramFiles(x86)
ProgramW6432
PROMPT
PSModulePath
PUBLIC
SystemDrive
SystemRoot
TEMP
TMP
USERDOMAIN
USERNAME
USERPROFILE
windir
PROCESSOR_ARCHITEW6432
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
"},{"location":"build_script/#unix","title":"Unix","text":"The environment variables listed in the following table are defined only on macOS and Linux.
Variable Description HOME
Standard $HOME environment variable. PKG_CONFIG_PATH
Path to pkgconfig
directory, defaults to `$PREFIX/lib/pkgconfig SSL_CERT_FILE
Path to SSL_CERT_FILE
file. CFLAGS
Empty, can be forwarded from env to set additional arguments to C compiler. CXXFLAGS
Same as CFLAGS for C++ compiler. LDFLAGS
Empty, additional flags to be passed to the linker when linking object files into an executable or shared object."},{"location":"build_script/#macos","title":"macOS","text":"The environment variables listed in the following table are defined only on macOS.
Variable Description MACOSX_DEPLOYMENT_TARGET
Same as the Anaconda Python macOS deployment target. Currently 10.9
for intel 32- and 64bit macOS, and 11.0 for arm64. OSX_ARCH
i386
or x86_64
or arm64
, depending on the target platform"},{"location":"build_script/#linux","title":"Linux","text":"The environment variable listed in the following table is defined only on Linux.
Variable Description LD_RUN_PATH
Defaults to <build prefix>/lib
. QEMU_LD_PREFIX
The prefix used by QEMU's user mode emulation for library paths. QEMU_UNAME
Set qemu uname release string to 'uname'. DEJAGNU
The path to the dejagnu testing framework used by the GCC test suite. DISPLAY
The X11 display to use for graphical applications. BUILD
Target triple ({build_arch}-conda_{build_distro}-linux-gnu
) where build_distro is one of cos6
or cos7
, for Centos 6 or 7"},{"location":"compilers/","title":"Compilers and cross-compilation","text":"To use a compiler in your project, it's best to use the ${{ compiler('lang') }}
template function. The compiler function works by taking a language, determining the configured compiler for that language, and adding some information about the target platform to the selected compiler. To configure a compiler for a specific language, the variant_config.yaml
file can be used.
For example, in a recipe that uses a C-compiler, you can use the following code:
requirements:\n build:\n - ${{ compiler('c') }}\n
To set the compiler that you want to use, create a variant config that looks like the following:
c_compiler:\n - gcc\n\n# optionally you can specify a version\nc_compiler_version:\n - 9.3.0\n
When the template function is evaluated, it will look something like: gcc_linux-64 9.3.0
. You can define your own compilers. For example, for Rust you can use ${{ compiler('rust') }}
and rust_compiler_{version}
in your variant config.
"},{"location":"compilers/#cross-compilation","title":"Cross-compilation","text":"Cross-compilation is supported by rattler-build
and the compiler template function is part of what makes it possible. When you want to cross-compile from linux-64
to linux-aarch64
(i.e. intel to ARM), you can pass --target-platform linux-aarch64
to the rattler-build
command. This will cause the compiler template function to select a compiler that is configured for linux-aarch64
. The above example would resolve to gcc_linux-aarch64 9.3.0
. Provided that the package is available for linux-64
(your build platform), the compilation should succeed.
The distinction between the build
and host
sections begins to make sense when thinking about cross-compilation. The build
environment is resolved to packages that need to run at compilation time. For example, cmake
, gcc
, and autotools
are all tools that need to be executed. Therefore, the build
environment resolves to packages for the linux-64
architecture (in our example). On the other hand, the host
packages resolve to linux-aarch64
- those are packages that we want to link against.
# packages that need to run at build time (cmake, gcc, autotools, etc.)\n# in the platform that rattler-build is executed on (the build_platform)\nbuild:\n - cmake\n - ${{ compiler('c') }}\n# packages that we want to link against in the architecture we are\n# cross-compiling to the target_platform\nhost:\n - libcurl\n - openssl\n
"},{"location":"converting_from_conda_build/","title":"Converting a recipe from conda-build","text":"The recipe format of rattler-build
differs in some aspects from conda-build
. This document aims to help you convert a recipe from conda-build
to rattler-build
.
"},{"location":"converting_from_conda_build/#automatic-conversion","title":"Automatic conversion","text":"To convert a recipe from meta.yaml
to recipe.yaml
you can use the automatic conversion utility.
To install conda-recipe-manager
, run
pixi global install conda-recipe-manager\n# or\nconda install -c conda-forge conda-recipe-manager\n
Then, run the conversion utility:
conda-recipe-manager convert my-recipe/meta.yaml\n
This will print the converted recipe to the console. You can save it to a file by redirecting the output:
conda-recipe-manager convert my-recipe/meta.yaml > recipe.yaml\n
To learn more about the tool, or contribute, find the repository here.
"},{"location":"converting_from_conda_build/#converting-jinja-and-selectors","title":"Converting Jinja and selectors","text":"To use jinja
in the new recipes, you need to keep in mind two conversions. The {% set version = \"1.2.3\" %}
syntax is replaced by the context
section in the new recipe format.
{% set version = \"1.2.3\" %}\n
becomes
context:\n version: \"1.2.3\"\n
To use the values or other Jinja expressions (e.g. from the variant config) you can use the ${{ version }}
syntax. Note the $
sign before the curly braces - it makes Jinja fully compatible with the YAML format.
meta.yaml# instead of\npackage:\n version: \"{{ version }}\"\nsource:\n url: https://example.com/foo-{{ version }}.tar.gz\n
becomes
recipe.yamlpackage:\n version: ${{ version }}\nsource:\n url: https://example.com/foo-${{ version }}.tar.gz\n
"},{"location":"converting_from_conda_build/#converting-selectors","title":"Converting selectors","text":"conda-build
has a line based \"selector\" system, to e.g. disable certain fields on Windows vs. Unix.
In rattler-build we\u00a0use two different syntaxes: an if/else/then
map or a inline jinja expression.
A typical selector in conda-build
looks something like this:
meta.yamlrequirements:\n host:\n - pywin32 # [win]\n
To convert this to rattler-build
syntax, you can use one of the following two syntaxes:
recipe.yamlrequirements:\n host:\n - ${{ \"pywin32\" if win }} # empty strings are automatically filtered\n # or\n - if: win\n then:\n - pywin32 # this list extends the outer list\n
"},{"location":"converting_from_conda_build/#converting-the-recipe-script","title":"Converting the recipe script","text":"We still support the build.sh
script, but the bld.bat
script was renamed to build.bat
in order to be more consistent with the build.sh
script.
You can also choose a different name for your script:
build:\n # note: if there is no extension, we will try to find .sh on unix and .bat on windows\n script: my_build_script\n
There are also new ways of writing scripts, for example with nushell
or python
Variant keys in build scripts
conda-build
tries to analyze the build scripts for any usage of variant keys. We do not attempt that. If you want to use variant keys in your build script that are not used anywhere else you need to manually add them to your script environment, e.g.
recipe.yamlbuild:\n script:\n content: echo $MY_VARIANT\n env:\n MY_VARIANT: ${{ my_variant }}\n
"},{"location":"converting_from_conda_build/#converting-the-recipe-structure","title":"Converting the recipe structure","text":"There are a few differences in the recipe structure. However, the schema will tell you quite easily what is expected and you should see red squiggly lines in your editor (e.g. VSCode) if you make a mistake.
Here are a few differences:
"},{"location":"converting_from_conda_build/#converting-the-test-section","title":"Converting the test section","text":"The test
section is renamed to tests
and is a list of independent tests. Each test runs in its own environment.
Let's have a look at converting an existing test section:
meta.yamltest:\n imports:\n - mypackage\n commands:\n - mypackage --version\n
This would now be split into two tests:
recipe.yamltests:\n - script:\n - mypackage --version\n - python:\n imports:\n - mypackage\n # by default we perform a `pip check` in the python test but\n # it can be disabled by setting this to false\n pip_check: false\n
The script
tests also take a requirements
section with run
and build
requirements. The build
requirements can be used to install emulators and similar tools that need to run to execute tests in a cross-compilation environment.
"},{"location":"experimental_features/","title":"Experimental features","text":"Warning
These are experimental features of rattler-build
and may change or go away completely.
Currently only the build
and rebuild
commands support the following experimental features.
To enable them, use the --experimental
flag with the command. Or, use the environment variable, RATTLER_BUILD_EXPERIMENTAL=true
.
"},{"location":"experimental_features/#jinja-functions","title":"Jinja functions","text":""},{"location":"experimental_features/#load_from_filefile_path","title":"load_from_file(<file_path>)
","text":"The Jinja function load_from_file
allows loading from files; specifically, it allows loading from toml
, json
, and yaml
file types to an object to allow it to fetch things directly from the file. It loads all other files as strings.
"},{"location":"experimental_features/#usage","title":"Usage","text":"load_from_file
is useful when there is a project description in a well-defined project file such as Cargo.toml
, package.json
, pyproject.toml
, package.yaml
, or stack.yaml
. It enables the recipe to be preserved in as simple a state as possible, especially when there is no need to keep the changes in sync; some example use cases for this are with CI/CD infrastructure or when there is a well-defined output format.
Below is an example loading a Cargo.toml
inside of the rattler-build
GitHub repository:
recipe.yamlcontext:\n name: ${{ load_from_file(\"Cargo.toml\").package.name }}\n version: ${{ load_from_file(\"Cargo.toml\").package.version }}\n source_url: ${{ load_from_file(\"Cargo.toml\").package.homepage }}\n rust_toolchain: ${{ load_from_file(\"rust-toolchains\") }}\n\npackage:\n name: ${{ name }}\n version: ${{ version }}\n\nsource:\n git: ${{ source_url }}\n tag: ${{ source_tag }}}}\n\nrequirements:\n build:\n - rust ==${{ rust_toolchain }}\n\nbuild:\n script: cargo build --release -p ${{ name }}\n\ntest:\n - script: cargo test -p ${{ name }}\n - script: cargo test -p rust-test -- --test-threads=1\n\nabout:\n home: ${{ source_url }}\n repository: ${{ source_url }}\n documentation: ${{ load_from_file(\"Cargo.toml\").package.documentation }}\n summary: ${{ load_from_file(\"Cargo.toml\").package.description }}\n license: ${{ load_from_file(\"Cargo.toml\").package.license }}\n
"},{"location":"experimental_features/#git-functions","title":"git
functions","text":"git
functions are useful for getting the latest tag and commit hash. These can be used in the context
section of the recipe, to fetch version information from a repository.
Examples # latest tag in the repo\ngit.latest_tag(<git_repo_url>)\n\n# latest tag revision(aka, hash of tag commit) in the repo\ngit.latest_tag_rev(<git_repo_url>)\n\n# latest commit revision(aka, hash of head commit) in the repo\ngit.head_rev(<git_repo_url>)\n
"},{"location":"experimental_features/#usage_1","title":"Usage","text":"These can be useful for automating minor things inside of the recipe itself, such as if the current version is the latest version or if the current hash is the latest hash, etc.
recipe.yamlcontext:\n git_repo_url: \"https://github.com/prefix-dev/rattler-build\"\n latest_tag: ${{ git.latest_tag( git_repo_url ) }}\n\npackage:\n name: \"rattler-build\"\n version: ${{ latest_tag }}\n\nsource:\n git: ${{ git_repo_url }}\n tag: ${{ latest_tag }}\n
There is currently no guarantee of caching for repo fetches when using git
functions. This may lead to some performance issues.
"},{"location":"highlevel/","title":"What is rattler-build
?","text":"rattler-build
is a tool to build and package software so that it can be installed on any operating system \u2013 with any compatible package manager such as mamba
, conda
, or rattler
. We are also intending for rattler-build
to be used as a library to drive builds of packages from any other recipe format in the future.
"},{"location":"highlevel/#how-does-rattler-build-work","title":"How does rattler-build
work?","text":"Building of packages consists of several steps. It all begins with a recipe.yaml
file that specifies how the package is to be built and what the dependencies are. From the recipe file, rattler-build
executes several steps:
Rendering: Parse the recipe file and evaluate conditionals, Jinja expressions, and variables, and variants.
Fetch source: Retrieve specified source files, such as .tar.gz
files, git
repositories, local paths. Additionally, this step will apply patches that can be specified alongside the source file.
Install build environments: Download and install dependencies into temporary \"host\" and \"build\" workspaces. Any dependencies that are needed at build time are installed in this step.
Build source: Execute the build script to build/compile the source code and install it into the host environment.
Prepare package files: Collect all files that are new in the \"host\" environment and apply some transformations if necessary; specifically, we edit the rpath
on Linux
and macOS
to make binaries relocatable.
Package: Bundle all the files in a package and write out any additional metadata into the info/index.json
, info/about.json
, and info/paths.json
files. This also creates the test files that are bundled with the package.
Test: Run any tests specified in the recipe. The package is considered done if it passes all the tests, otherwise its moved to broken/
in the output directory.
After this process, a package is created. This package can be uploaded to somewhere like a custom prefix.dev private or public channel.
"},{"location":"highlevel/#how-to-run-rattler-build","title":"How to run rattler-build
","text":"Running rattler-build
is straightforward. It can be done on the command line:
rattler-build build --recipe myrecipe/recipe.yaml\n
A custom channel that is not conda-forge (the default) can be specified like so:
rattler-build build -c robostack --recipe myrecipe/recipe.yaml\n
You can also use the --recipe-dir
argument if you want to build all the packages in a directory:
rattler-build build --recipe-dir myrecipes/\n
"},{"location":"highlevel/#overview-of-a-recipeyaml","title":"Overview of a recipe.yaml
","text":"A recipe.yaml
file is separated into multiple sections and can conditionally include or exclude sections. Recipe files also support a limited amount of string interpolation with Jinja (specifically minijinja
in our case).
A simple example of a recipe file for the zlib
package would look as follows:
recipe.yaml# variables from the context section can be used in the rest of the recipe\n# in jinja expressions\ncontext:\n version: 1.2.13\n\npackage:\n name: zlib\n version: ${{ version }}\n\nsource:\n url: http://zlib.net/zlib-${{ version }}.tar.gz\n sha256: b3a24de97a8fdbc835b9833169501030b8977031bcb54b3b3ac13740f846ab30\n\nbuild:\n # build numbers can be set arbitrarily\n number: 0\n script:\n # build script to install the package into the $PREFIX (host prefix)\n - if: unix\n then:\n - ./configure --prefix=$PREFIX\n - make -j$CPU_COUNT\n - if: win\n then:\n - cmake -G \"Ninja\" -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=%LIBRARY_PREFIX%\n - ninja install\n\nrequirements:\n build:\n # compiler is a special function.\n - ${{ compiler(\"c\") }}\n # The following two dependencies are only needed on Windows,\n # and thus conditionally selected\n - if: win\n then:\n - cmake\n - ninja\n - if: unix\n then:\n - make\n
The sections of a recipe are:
sections description context
Defines variables that can be used in the Jinja context later in the recipe (e.g. name and version are commonly interpolated in strings) package
This section defines the name and version of the package you are currently building and will be the name of the final output source
Defines where the source code is going to be downloaded from and checksums build
Settings for the build and the build script requirements
Allows the definition of build, host, run and run-constrained dependencies"},{"location":"internals/","title":"What does rattler-build
do to build a package?","text":"rattler-build
creates conda packages which are relocatable packages. These packages are built up with some rules and conventions in mind.
"},{"location":"internals/#what-goes-into-a-package","title":"What goes into a package?","text":"Generally speaking, any new files that are copied into the $PREFIX
directory at build time are part of the new package. However, there is some filtering going on to exclude unwanted files, and noarch: python
packages have special handling as well. The rules are as follows:
"},{"location":"internals/#filtering","title":"Filtering","text":""},{"location":"internals/#general-file-filtering","title":"General File Filtering","text":"Certain files are filtered out to prevent them from being included in the package. These include:
.pyo files: Optimized Python files are not included because they are considered harmful. .la files: Libtool archive files that are not needed at runtime. .DS_Store files: macOS-specific files that are irrelevant to the package. .git files and directories: Version control files, including .gitignore
and the .git
directory, which are not needed in the package. share/info/dir This file is ignored because it would be written from multiple packages. "},{"location":"internals/#special-handling-for-noarch-python-packages","title":"Special Handling for noarch: python
Packages","text":"For packages marked as noarch: python
, special transformations are applied to ensure compatibility across different platforms:
Stripping Python Library Prefix: The \"lib/pythonX.X\" prefix is removed, retaining only the \"site-packages\" part of the path. Skipping __pycache__
Directories and .pyc
Files: These are excluded and recreated during installation (they are specific to the Python version). Replacing bin
and Scripts
Directories: On Unix systems, the bin
directory is replaced with python-scripts
. On Windows systems, the Scripts
directory is replaced with python-scripts
. Remove explicitly mentioned entrypoints: For noarch: python
packages, entry points registered in the package are also taken into account. Files in the bin
or Scripts
directories that match entry points are excluded to avoid duplications. "},{"location":"internals/#symlink-handling","title":"Symlink Handling","text":"Symlinks are carefully managed to ensure they are relative rather than absolute, which aids in making the package relocatable:
Absolute symlinks pointing within the $PREFIX
are converted to relative symlinks. On Unix systems, this conversion is handled directly by creating new relative symlinks. On Windows, a warning is issued since symlink creation requires administrator privileges. "},{"location":"internals/#making-packages-relocatable-with-rattler-build","title":"Making Packages Relocatable with rattler-build
","text":"Often, the most challenging aspect of building a package using rattler-build
is making it relocatable. A relocatable package can be installed into any prefix, allowing it to be used outside the environment in which it was built. This is in contrast to a non-relocatable package, which can only be utilized within its original build environment.
rattler-build
automatically performs the following actions to make packages relocatable:
Binary object file conversion: Binary object files are converted to use relative paths using install_name_tool
on macOS and patchelf
on Linux. This uses $ORIGIN
for elf files on Linux and @loader_path
for Mach-O files on macOS to make the rpath
relative to the executable / shared library. Text file prefix registration: Any text file without NULL
bytes containing the placeholder prefix have the registered prefix replaced with the install prefix. Binary file prefix detection and registration: Binary files containing the build prefix can be automatically registered. The registered files will have their build prefix replaced with the install prefix at install time. This works by padding the install prefix with null terminators, such that the length of the binary file remains the same. The build prefix must be long enough to accommodate any reasonable installation prefix. On macOS and Linux, rattler-build
pads the build prefix to 255 characters by appending _placehold
to the end of the build directory name. "},{"location":"multiple_output_cache/","title":"The cache for multiple outputs","text":"Note
The \"multi-output\" cache is a little bit different from a compilation cache. If you look for tips and tricks on how to use sccache
or ccache
with rattler-build
, please refer to the tips and tricks section.
Sometimes you build a package and want to split the contents into multiple sub-packages. For example, when building a C/C++ package, you might want to create multiple packages for the runtime requirements (library), and the development time requirements such as header files.
The \"cache\" output makes this easy. It allows you to specify a single top-level cache that can produce arbitrary files, that can then be used in other packages.
Let's take a look at an example:
recipe.yamlrecipe:\n name: mypackage\n version: '0.1.0'\n\ncache:\n source:\n - url: https://example.com/library.tar.gz\n sha256: 1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef\n\n requirements:\n build:\n - ${{ compiler('c') }}\n\n build:\n script:\n - mkdir -p $PREFIX/lib\n - mkdir -p $PREFIX/include\n - echo \"This is the library\" > lib/library.txt\n - echo \"This is the header\" > include/header.txt\n\noutputs:\n - package:\n name: mypackage-library\n build:\n files:\n - lib/*\n\n - package:\n name: mypackage-headers\n build:\n files:\n - include/*\n
Note
Since this is an experimental feature, you need to pass the --experimental
flag to enable parsing of the cache
top-level section.
In this example, we have a single package called mypackage
that creates two outputs: mypackage-library
and mypackage-headers
. The cache output will run like a regular output, but after the build is finished, the files will be copied to a \"cache\" directory (in your output folder, under output/build_cache
).
The files in the cache folder are then copied into the $PREFIX
of each output package. Since they are \"new\" files in the prefix, they will be included in the output package. The easiest way to select a subset of the files in the prefix is by using the files
field in the output definition. You can use a list of globs to select only the files that you want.
For something more complicated you can also use include
and exclude
fields in the files
selector. Please refer to the the build options documentation.
"},{"location":"multiple_output_cache/#run-exports-from-the-cache","title":"Run exports from the cache","text":"Since the cache output also has build- and host requirements we need to additionally take care of eventual \"run-exports\" from the cache output. Run exports from the cache-dependencies are handled very similar to the run exports from a given output. We append any run exports to the outputs.
If the cache has an \"ignore run exports\" section, than we apply those filters at the cache level. If the output ignores any run exports, then we also ignore the run-exports if they would come from the cache.
"},{"location":"multiple_output_cache/#source-code-in-the-cache","title":"Source code in the cache","text":"The cache output has its own source
section. For every output, the (dirty) source is restored from the cache directory. Outputs can layer additional files on top of the cache source. However, if you already ran cmake
in the cache output, you can continue from where the build left off. This is useful when you want to e.g. build additional components (such as Python bindings) on top of the already-built library.
"},{"location":"multiple_output_cache/#c-example-that-builds-python-bindings-on-top-of-a-library","title":"C++ Example that builds Python bindings on top of a library","text":"You can find an example (with source code) here: Link.
variants.yamlpython:\n - \"3.12.*\"\n - \"3.11.*\"\n
And the corresponding recipe:
recipe.yamlrecipe:\nname: calculator\nversion: 1.0.0\n\ncache:\nsource:\n path: ../\n\nrequirements:\n build:\n - ${{ compiler('cxx') }}\n - cmake\n - ninja\nbuild:\n script:\n # make sure that `alternative_name.md` is not present\n - test ! -f ./alternative_name.md\n - mkdir build\n - cd build\n - cmake $SRC_DIR -GNinja ${CMAKE_ARGS}\n - ninja install\n\noutputs:\n# this first output will include all files installed during the cache build\n- package:\n name: libcalculator\n\n requirements:\n run_exports:\n - ${{ pin_subpackage('libcalculator') }}\n# This output will build the Python bindings using CMake and then create new\n# packages with the Python bindings\n- package:\n name: py-calculator\n source:\n - path: ../README.md\n file_name: alternative_name.md\n\n requirements:\n build:\n - ${{ compiler('cxx') }}\n - cmake\n - ninja\n host:\n - pybind11\n - python\n - libcalculator\n\n build:\n script:\n # assert that the README.md file is present\n - test -f ./alternative_name.md\n - cd build\n - cmake $SRC_DIR -GNinja ${CMAKE_ARGS} -DBUILD_PYTHON_BINDINGS=ON\n - ninja install\n
"},{"location":"package_spec/","title":"Package specification","text":"rattler-build
produces \"conda\" packages. These packages work with the mamba
and conda
package managers, and they work cross-platform on Windows, Linux and macOS.
By default, a conda package is a tar.bz2
archive which contains:
Metadata under the info/
directory A collection of files that are installed directly into an install prefix The format is identical across platforms and operating systems. During the install process, all files are extracted into the install prefix, except the ones in info/
. Installing a conda package into an environment is similar to executing the following commands:
cd <environment prefix>\ntar xjf mypkg-1.0.0-h2134.tar.bz2\n
Only files, including symbolic links, are part of a conda package. Directories are not included. Directories are created and removed as needed, but you cannot create an empty directory from the tar archive directly.
There is also a newer archive type, suffixed with .conda
. This archive type consists of an outer \"zip\" archive that is not compressed, and two inner archives that are compressed with zstd
, which is very fast for decompression.
The inner archives are split into info
and pkg
files, which makes it possible to extract only the info
part of the archive (only the metadata), which is often smaller in size.
"},{"location":"package_spec/#package-filename","title":"Package filename","text":"A conda package conforms to the following filename:
<name>-<version>-<hash>.tar.bz2 OR <name>-<version>-<hash>.conda\n
"},{"location":"package_spec/#special-files-in-packages","title":"Special files in packages","text":"There are some special files in a package:
activation and deactivation scripts that are executed when the environment is activated or deactivated post-link and pre-unlink scripts that are executed when the package is installed or uninstalled You can read more about these files in the activation scripts and other special files section.
"},{"location":"package_spec/#package-metadata","title":"Package metadata","text":"The info/
directory contains all metadata about a package. Files in this location are not installed under the install prefix. Although you are free to add any file to this directory, conda only inspects the content of the files discussed below:
"},{"location":"package_spec/#infoindexjson","title":"info/index.json
","text":"This file contains basic information about the package, such as name, version, build string, and dependencies. The content of this file is stored in repodata.json
, which is the repository index file, hence the name index.json
. The JSON object is a dictionary containing the keys shown below.
name: string
The lowercase name of the package. May contain lowercase characters, underscores, and dashes.
version: string
The package version. May not contain \"-
\". Acknowledges PEP 440.
build: string
The build string. May not contain \"-
\". Differentiates builds of packages with otherwise identical names and versions, such as:
A build with other dependencies, such as Python 3.4 instead of Python 2.7. A bug fix in the build process. Some different optional dependencies, such as MKL versus ATLAS linkage. Nothing in conda actually inspects the build string. Strings such as np18py34_1
are designed only for human readability and conda never parses them. build_number: integer
A non-negative integer representing the build number of the package. Unlike the build string, the build_number
is inspected by conda. Conda uses it to sort packages that have otherwise identical names and versions to determine the latest one. This is important because new builds that contain bug fixes for the way a package is built may be added to a repository.
depends: list of match specs
A list of dependency specifications, where each element is a string. These come from the run
section of the recipe or any run exports of dependencies.
constrains: list of match specs
A list of optional dependency constraints. The packages listed under constrains
are not installed by default, but if they are installed they have to respect the constraints.
subdir: string
The subdir (like linux-64
) of this package.
arch: string
Optional. The architecture the package is built for. EXAMPLE: x86_64
. This key is generally not used (duplicate information from sudir
).
platform: string
Optional. The OS that the package is built for, e.g. osx
. This key is generally not used (duplicate information from sudir
).
"},{"location":"package_spec/#infopathsjson","title":"info/paths.json
","text":"The paths.json
file lists all files that are installed into the environment.
It consists of a list of path entries, each with the following keys:
_path: string
The relative path of the file
path_type: optional, string
The type of linking, can be hardlink
, softlink
, or directory
. Default is hardlink
.
file_mode: - optional, string
The file mode can be binary
or text
. This is only relevant for prefix replacement.
prefix_placeholder: optional, string
The prefix placeholder string that is encoded in the text or binary file, which is replaced at installation time. Note that this prefix placeholder uses /
even on Windows.
no_link: bool, optional
Determines whether this file should be linked or not when installing the package (linking the file from the cache into the environment). Defaults to false
.
sha256: string
The SHA256
hash of the file. For symbolic links it contains the SHA256
hash of the file pointed to.
size_in_bytes: number
The size, in bytes, of the file. For symbolic links, it contains the file size of the file pointed to.
Due to the way the binary replacement works, the placeholder prefix must be longer than the install prefix.
"},{"location":"package_spec/#infolicense","title":"info/license/<...>
","text":"All licenses mentioned in the recipe are copied to this folder.
"},{"location":"package_spec/#infoaboutjson","title":"info/about.json
","text":"Optional file. Contains the entries of the \"about\" section of the recipe of the recipe.yaml
file. The following keys are added to info/about.json
if present in the build recipe:
Renamed fields
The new recipe spec renamed a few fields (from conda-build's original implementation). This means that some fields in the about.json
file still have the old names (for backwards compatibility), while you would generally use different names in the recipe.
home: url (from about.homepage)
The URL of the homepage of the package.
dev_url: url (from about.repository)
The URL of the development repository of the package.
doc_url: url (from about.documentation)
The URL of the documentation of the package.
license_url: url
The URL of the license of the package.
license: string (from about.license)
The SPDX license identifier of the package.
summary: string
A short summary of the package.
description: string
A longer description of the package.
license_family: string
(this field is not used anymore as we rely on SPDX license identifiers)
"},{"location":"package_spec/#inforecipe","title":"info/recipe/<...>
","text":"A directory containing the full contents of the build recipe. This folder also contains a rendered version of the recipe (rendered_recipe.yaml
). This rendered version is used for the rebuild
command. However, note that currently this format is still in flux and can change at any time.
You can also use --no-include-recipe
to disable the inclusion of the recipe in the package.
"},{"location":"rebuild/","title":"Rebuilding a package","text":"The rebuild
command allows you to rebuild a package from an existing package. The main use case is to examine if a package can be rebuilt in a reproducible manner. You can read more about reproducible builds here.
"},{"location":"rebuild/#usage","title":"Usage","text":"rattler-build rebuild ./mypkg-0.1.0-h60d57d3_0.tar.bz2\n
"},{"location":"rebuild/#how-it-works","title":"How it works","text":"The recipe is \"rendered\" and stored into the package. The way the recipe is rendered is subject to change. For the moment, the rendered recipe is stored as info/recipe/rendered_recipe.yaml
. It includes the exact package versions that were used at build time. When rebuilding, we use the package resolutions from the rendered recipe, and execute the same build script as the original package.
We also take great care to sort files in a deterministic manner as well as erasing any time stamps. The SOURCE_DATE_EPOCH
environment variable is set to the same timestamp as the original build for additional determinism (some build tools use this variable to set timestamps).
"},{"location":"rebuild/#how-to-check-the-reproducibility-of-a-package","title":"How to check the reproducibility of a package","text":"There is an excellent tool called diffoscope
that allows you to compare two packages and see the differences. You can install it with pixi
:
pixi global install diffoscope\n
To compare two packages, you can use the following command:
rattler-build rebuild ./build0.tar.bz2\ndiffoscope ./build0.tar.bz2 ./mypkg-0.1.0-h60d57d3_0.tar.bz2\n
"},{"location":"recipe_generation/","title":"Generating recipes for different ecosystems","text":"Rattler-build has some builtin functionality to generate recipes for different (existing) ecosystems.
Currently we support the following ecosystems:
pypi
(Python) - generates a recipe for a Python package cran
(R) - generates a recipe for an R package To generate a recipe for a Python package, you can use the following command:
rattler-build generate-recipe pypi jinja2\n
This will generate a recipe for the jinja2
package from PyPI and print it to the console. To turn it into a recipe, you can either pipe the stdout to a file or use the -w
flag. The -w
flag will create a new folder with the recipe in it.
The generated recipe for jinja2
will look something like:
recipe.yamlpackage:\n name: jinja2\n version: 3.1.4\n\nsource:\n- url: https://files.pythonhosted.org/packages/ed/55/39036716d19cab0747a5020fc7e907f362fbf48c984b14e62127f7e68e5d/jinja2-3.1.4.tar.gz\n sha256: 4a3aee7acbbe7303aede8e9648d13b8bf88a429282aa6122a993f0ac800cb369\n\nbuild:\n script: python -m pip install .\n\nrequirements:\n host:\n - flit_core <4\n - python >=3.7\n - pip\n run:\n - python >=3.7\n - markupsafe >=2.0\n # - babel >=2.7 # extra == 'i18n'\n\ntests: []\n\nabout:\n summary: A very fast and expressive template engine.\n documentation: https://jinja.palletsprojects.com/\n
"},{"location":"recipe_generation/#generating-recipes-for-r-packages","title":"Generating recipes for R packages","text":"To generate a recipe for an R package, you can use the following command:
rattler-build generate-recipe cran dplyr\n
The R
recipe generation supports some additional flags:
-u/--universe
select an R universe to use (e.g. bioconductor
) -t/--tree
generate multiple recipes, for every dependency as well R packages will be prefixed with r-
to avoid name conflicts with Python packages. The generated recipe for dplyr
will look something like:
recipe.yamlpackage:\n name: r-dplyr\n version: 1.1.4\n\nsource:\n- url: https://cran.r-project.org/src/contrib/dplyr_1.1.4.tar.gz\n md5: e3066ea859b26e0d3b992c476ea3af2e\n\nbuild:\n script: R CMD INSTALL --build .\n python: {}\n\nrequirements:\n host:\n - r-base >=3.5.0\n run:\n - r-cli >=3.4.0\n - r-generics\n - r-glue >=1.3.2\n - r-lifecycle >=1.0.3\n - r-magrittr >=1.5\n - r-methods\n - r-pillar >=1.9.0\n - r-r6\n - r-rlang >=1.1.0\n - r-tibble >=3.2.0\n - r-tidyselect >=1.2.0\n - r-utils\n - r-vctrs >=0.6.4\n # - r-bench # suggested\n # - r-broom # suggested\n # - r-callr # suggested\n # - r-covr # suggested\n # - r-dbi # suggested\n # - r-dbplyr >=2.2.1 # suggested\n # - r-ggplot2 # suggested\n # - r-knitr # suggested\n # - r-lahman # suggested\n # - r-lobstr # suggested\n # - r-microbenchmark # suggested\n # - r-nycflights13 # suggested\n # - r-purrr # suggested\n # - r-rmarkdown # suggested\n # - r-rmysql # suggested\n # - r-rpostgresql # suggested\n # - r-rsqlite # suggested\n # - r-stringi >=1.7.6 # suggested\n # - r-testthat >=3.1.5 # suggested\n # - r-tidyr >=1.3.0 # suggested\n # - r-withr # suggested\n\nabout:\n homepage: https://dplyr.tidyverse.org, https://github.com/tidyverse/dplyr\n summary: A Grammar of Data Manipulation\n description: |-\n A fast, consistent tool for working with data frame like\n objects, both in memory and out of memory.\n license: MIT\n license_file: LICENSE\n repository: https://github.com/cran/dplyr\n
Tip
You can use the generated recipes to build your own \"forge\" with rattler-build
. Read more about it in the Building your own forge section.
"},{"location":"selectors/","title":"Selectors in recipes","text":"Recipe and variant configuration files can utilize selectors to conditionally add, remove, or modify dependencies, configuration options, or even skip recipe execution based on specific conditions.
Selectors are implemented using an if / then / else
map, which is a valid YAML dictionary. The condition is evaluated using minijinja
and follows the same syntax as a Python expression.
During rendering, several variables are set based on the platform and variant being built. For example, the unix
variable is true for macOS and Linux, while win
is true for Windows. Consider the following recipe executed on Linux:
requirements:\n host:\n - if: unix\n then: unix-tool\n - if: win\n then: win-tool\n
This will be evaluated as:
requirements:\n host:\n - unix-tool\n
The line containing the Windows-specific configuration is removed. Multiple items can also be selected, such as:
host:\n - if: linux\n then:\n - linux-tool-1\n - linux-tool-2\n - linux-tool-3\n
For Linux, this will result in:
host:\n - linux-tool-1\n - linux-tool-2\n - linux-tool-3\n
Other examples often found in the wild:
if: build_platform != target_platform ... # true if cross-platform build\nif: osx and arm64 ... # true for apple silicon (osx-arm64)\nif: linux and (aarch64 or ppc64le)) ... # true for linux ppc64le or linux-aarch64\n
"},{"location":"selectors/#available-variables","title":"Available variables","text":"The following variables are available during rendering of the recipe:
Variable Description target_platform
the configured target_platform
for the build build_platform
the configured build_platform
for the build linux
\"true\" if target_platform
is Linux osx
\"true\" if target_platform
is OSX / macOS win
\"true\" if target_platform
is Windows unix
\"true\" if target_platform
is a Unix (macOS or Linux) x86
, x86_64
x86 32/64-bit Architecture aarch64
, arm64
64-bit Arm (these are the same but are both supported for legacy) armV6l
, armV7l
32-bit Arm ppc64
, s390x
, Big endian ppc64le
Little endian riscv32
, riscv64
The RISC-V Architecture wasm32
The WebAssembly Architecture"},{"location":"selectors/#variant-selectors","title":"Variant selectors","text":"To select based on variant configuration you can use the names in the selectors as well. For example, if the build uses python: 3.8
as a variant, we can use if: python == \"3.8\"
to enable a dependency for only when the Python version is 3.8.
String comparison
The comparison is a string comparison done by minijinja
, so it is important to use the correct string representation of the variant. Use the match
function to compare versions.
variants.yamlpython:\n - 3.8\n - 3.9\n
recipe.yamlrequirements:\n host:\n - if: python == \"3.8\" # (1)!\n then: mydep\n else: otherdep\n
This will only add mydep
when the Python version is 3.8. This comparison is a string comparison, so it is important to use the correct string representation of the variant. "},{"location":"selectors/#the-match-function","title":"The match
function","text":"!!! note \"Rename from cmp
to match
\" The cmp
function has been renamed to match
to better reflect its purpose.
Inside selectors, one can use a special match
function to test if the selected variant version has a matching version. For example, having the following variants file, we could use the these tests:
variants.yamlpython:\n - 3.8\n - 3.9\n
recipe.yaml- if: match(python, \"3.8\") # true, false\n then: mydep\n- if: match(python, \">=3.8\") # true, true\n then: mydep\n- if: match(python, \"<3.8\") # false, false (1)\n then: mydep\n
else:
would also have worked here. This function eliminates the need to implement any Python-specific conda-build
selectors (such as py3k
, py38
, etc.) or the py
and npy
integers.
Please note that during the initial phase of rendering we do not know the variant, and thus the match
condition always evaluates to true
.
"},{"location":"selectors/#selector-evaluation","title":"Selector evaluation","text":"Except for the rattler-build specific selectors, the selectors are evaluated using the minijinja
engine. This means that the selectors are evaluated by minijinja
thus Python like expressions. Some notable options are:
- if: python == \"3.8\" # equal\n- if: python != \"3.8\" # not equal\n- if: python and linux # true if python variant is set and the target_platform is linux\n- if: python and not linux # true if python variant is set and the target_platform is not linux\n- if: python and (linux or osx) # true if python variant is set and the target_platform is linux or osx\n
"},{"location":"special_files/","title":"Activation scripts and other special files","text":"A conda
package can contain \"special\" files in the prefix. These files are scripts that are executed during activation, installation, or uninstallation process.
If possible, they should be avoided since they execute arbitrary code at installation time and slow down the installation and activation process.
"},{"location":"special_files/#activation-scripts","title":"Activation scripts","text":"The activation scripts are executed when the environment containing the package is activated (e.g. when doing micromamba activate myenv
or pixi run ...
).
The scripts are located in special folders:
etc/conda/activate.d/{script.sh/bat}
- scripts in this folder are executed before the environment is activated etc/conda/deactivate.d/{script.sh/bat}
- scripts in this folder are executed when the environment is deactivated The scripts are executed in lexicographical order, so you can prefix them with numbers to control the order of execution.
To add a script to the package, just make sure that you install the file in this folder. For example, on Linux:
mkdir -p $PREFIX/etc/conda/activate.d\ncp activate-mypkg.sh $PREFIX/etc/conda/activate.d/10-activate-mypkg.sh\n\nmkdir -p $PREFIX/etc/conda/deactivate.d\ncp deactivate-mypkg.sh $PREFIX/etc/conda/deactivate.d/10-deactivate-mypkg.sh\n
"},{"location":"special_files/#post-link-and-pre-unlink-scripts","title":"Post-link and pre-unlink scripts","text":"The post-link
and pre-unlink
scripts are executed when the package is installed or uninstalled. They are both heavily discouraged but implemented for compatibility with conda in rattler-build
since version 0.17.
For a post-link
script to be executed when a package is installed, the built package needs to have a .<package_name>-post-link.{sh/bat}
in its bin/
folder. The same is applicable for pre-unlink
scripts, just with the name .<package_name>-pre-unlink.{sh/bat}
(note the leading period). For example, for a package mypkg
, you would need to have a .mypkg-post-link.sh
in its bin/
folder.
To make sure the scripts are included in the correct location, use your recipe's build script or build/script
key. For example, assuming you have a post-link.sh
script in your source, alongside the recipe in the recipe's folder, the following configuration will copy it correctly:
build:\n ...\n script:\n - ...\n - mkdir -p $PREFIX/bin\n - cp $RECIPE_DIR/post-link.sh $PREFIX/bin/.mypkg-post-link.sh\n - chmod +x $PREFIX/bin/.mypkg-post-link.sh\n
The $PREFIX
and $RECIPE_DIR
environment variables will be set during the build process to help you specify the correct paths.
"},{"location":"testing/","title":"Testing packages","text":"When you are developing a package, you should write tests for it. The tests are automatically executed right after the package build has finished.
The tests from the test section are actually packaged into your package and can also be executed straight from the existing package.
The idea behind adding the tests into the package is that you can execute the tests independently from building the package. That is also why we are shipping a test
subcommand that takes as input an existing package and executes the tests:
rattler-build test --package-file ./xtensor-0.24.6-h60d57d3_0.tar.bz2\n
Running the above command will extract the package and create a clean environment where the package and dependencies are installed. Then the tests are executed in this newly-created environment.
If you inspect the package contents, you would find the test files under info/test/*
.
"},{"location":"testing/#how-tests-are-translated","title":"How tests are translated","text":"The tests
section allows you to specify the following things:
tests:\n - script:\n # commands to run to test the package. If any of the commands\n # returns with an error code, the test is considered failed.\n - echo \"Hello world\"\n - pytest ./tests\n\n # additional requirements at test time\n requirements:\n run:\n - pytest\n\n files:\n # Extra files to be copied to the test directory from the \"work directory\"\n source:\n - tests/\n - test.py\n - *.sh\n recipe:\n - more_tests/*.py\n\n # This test section tries to import the Python modules and errors if it can't\n - python:\n imports:\n - mypkg\n - mypkg.subpkg\n
When you are writing a test for your package, additional files are created and added to your package. These files are placed under the info/tests/{index}/
folder for each test.
For a script test:
All the files are copied straight into the test folder (under info/tests/{index}/
) The script is turned into a run_test.sh
or run_test.bat
file The extra requirements are stored as a JSON file called test_time_dependencies.json
For a Python import test:
A JSON file is created that is called python_test.json
and stores the imports to be tested and whether to execute pip check
or not. This file is placed under info/tests/{index}/
For a downstream test:
A JSON file is created that is called downstream_test.json
and stores the downstream tests to be executed. This file is placed under info/tests/{index}/
"},{"location":"testing/#legacy-tests","title":"Legacy tests","text":"Legacy tests (from conda-build
) are still supported for execution. These tests are stored as files under the info/test/
folder.
The files are:
run_test.sh
(Unix) run_test.bat
(Windows) run_test.py
(for the Python import tests) test_time_dependencies.json
(for additional dependencies at test time) Additionally, the info/test/
folder contains all the files specified in the test section as source_files
and files
. The tests are executed pointing to this directory as the current working directory.
"},{"location":"tips_and_tricks/","title":"Tips and tricks for rattler-build","text":"This section contains some tips and tricks for using rattler-build
.
"},{"location":"tips_and_tricks/#using-sccache-or-ccache-with-rattler-build","title":"Using sccache or ccache with rattler-build
","text":"When debugging a recipe it can help a lot to use sccache
or ccache
. You can install both tools e.g. with pixi global install sccache
.
To use them with a CMake project, you can use the following variables:
export CMAKE_C_COMPILER_LAUNCHER=sccache\nexport CMAKE_CXX_COMPILER_LAUNCHER=sccache\n\n# or more generally\n\nexport C=\"sccache $C\"\nexport CXX=\"sccache $CXX\"\n
However, both ccache
and sccache
are sensitive to changes in the build location. Since rattler-build
, by default, always creates a new build directory with the timestamp, you need to use the --no-build-id
flag. This will disable the time stamp in the build directory and allow ccache
and sccache
to cache the build.
rattler-build build --no-build-id --recipe ./path/to/recipe.yaml\n
"},{"location":"tips_and_tricks/#building-your-own-forge","title":"Building your own \"forge\"","text":"You might want to publish your own software packages to a channel you control. These might be packages that are not available in the main conda-forge channel, or proprietary packages, or packages that you have modified in some way.
Doing so is pretty straightforward with rattler-build
and a CI provider of your choice. We have a number of example repositories for \"custom\" forges:
rust-forge: This repository builds a number of Rust packages for Windows, macOS and Linux on top of Github Actions. r-forge: The same idea, but for R
packages "},{"location":"tips_and_tricks/#directory-structure","title":"Directory structure","text":"To create your own forge, you should create a number of sub-directories where each sub-directory should contain at most one recipe. With the --recipe-dir
flag of rattler-build, the program will go and collect all recipes it finds in the given directory or sub-directories.
We can combine this with the --skip-existing=all
flag which will skip all packages that are already built locally or in the channel (if you upload them). Using all
will also look at the repodata.json
file in the channel to see if the package is already there. Packages are skipped based on their complete name, including the version and build string.
To note: the build string changes if the variant configuration changes! So if you update a package in the variant configuration, the packages that need rebuilding should be rebuilt.
Note
You can generate recipes for different ecosystems with the rattler-build generate-recipe
command. Read more about it in the Generating recipes section.
"},{"location":"tips_and_tricks/#ci-setup","title":"CI setup","text":"As an example, the following is the CI setup for rust-forge
. The workflow uses rattler-build
to build and upload packages to a custom channel on https://prefix.dev \u2013 but you can also use rattler-build
to upload to your own quetz
instance, or a channel on anaconda.org
.
Example CI setup for rust-forge
The following is an example of a Github Actions workflow for rust-forge
:
.github/workflows/forge.ymlname: Build all packages\n\non:\n push:\n branches:\n - main\n workflow_dispatch:\n pull_request:\n branches:\n - main\n\njobs:\n build:\n strategy:\n matrix:\n include:\n - { target: linux-64, os: ubuntu-20.04 }\n - { target: win-64, os: windows-latest }\n # force older macos-13 to get x86_64 runners\n - { target: osx-64, os: macos-13 }\n - { target: osx-arm64, os: macos-14 }\n fail-fast: false\n\n runs-on: ${{ matrix.os }}\n steps:\n - uses: actions/checkout@v4\n with:\n fetch-depth: 2\n - uses: prefix-dev/setup-pixi@v0.5.1\n with:\n pixi-version: v0.24.2\n cache: true\n\n - name: Run code in changed subdirectories\n shell: bash\n env:\n TARGET_PLATFORM: ${{ matrix.target }}\n\n run: |\n pixi run rattler-build build --recipe-dir . \\\n --skip-existing=all --target-platform=$TARGET_PLATFORM \\\n -c conda-forge -c https://prefix.dev/rust-forge\n\n - name: Upload all packages\n shell: bash\n # do not upload on PR\n if: github.event_name == 'push'\n env:\n PREFIX_API_KEY: ${{ secrets.PREFIX_API_KEY }}\n run: |\n # ignore errors because we want to ignore duplicate packages\n for file in output/**/*.conda; do\n pixi run rattler-build upload prefix -c rust-forge \"$file\" || true\n done\n
"},{"location":"tui/","title":"Terminal User Interface","text":"rattler-build
offers a terminal user interface for building multiple packages and viewing the logs.
To launch the TUI, run the build
command with the --tui
flag as shown below:
$ rattler-build build -r recipe.yaml --tui\n
Note
rattler-build-tui
is gated behind the tui
feature flag to avoid extra dependencies. Build the project with --features tui
arguments to enable the TUI functionality.
"},{"location":"tui/#key-bindings","title":"Key Bindings","text":"Key Action \u23ce Build a Build all j/k Next/previous package up/down/left/right Scroll logs e Edit recipe (via $EDITOR
) c, : Open command prompt (available commands: edit
) q, ctrl-c, esc, Quit"},{"location":"variants/","title":"Variant configuration","text":"rattler-build
can automatically build multiple variants of a given package. For example, a Python package might need multiple variants per Python version (especially if it is a binary package such as numpy
).
For this use case, one can specify variant configuration files. A variant configuration file has 2 special entries and a list of packages with variants. For example:
variants.yaml# special entry #1, the zip keys\nzip_keys:\n- [python, numpy]\n\n# special entry #2, the pin_run_as_build key\npin_run_as_build:\n numpy:\n max_pin: 'x.x'\n\n# entries per package version that users are interested in\npython:\n# Note that versions are _strings_ (not numbers)\n- \"3.8\"\n- \"3.9\"\n- \"3.10\"\n\nnumpy:\n- \"1.12\"\n- \"1.12\"\n- \"1.20\"\n
If we have a recipe, that has a build
, host
or run
dependency on python
we will build multiple variants of this package, one for each configured python
version (\"3.8\", \"3.9\" and \"3.10\").
For example:
# ...\nrequirements:\n host:\n - python\n
... will be rendered as (for the first variant):
# ...\nrequirements:\n host:\n- python 3.8*\n
Note that variants are only applied if the requirement doesn't specify any constraints. If the requirement would be python >3.8,<3.10
then the variant entry would be ignored.
"},{"location":"variants/#automatic-variantsyaml-discovery","title":"Automatic variants.yaml
discovery","text":"rattler-build
automatically includes the variant configuration from a variants.yaml
file next to a recipe. Use the --ignore-recipe-variants
option to disable automatic discovery of variants.yaml
files next to the recipes.
To include a variant config file from another location or include multiple configuration files use the --variant-config
option:
rattler-build build --variant-config ~/user_variants.yaml --variant-config /opt/rattler-build/global_variants.yaml --recipe myrecipe.yaml\n
"},{"location":"variants/#package-hash-from-variant","title":"Package hash from variant","text":"You might have wondered what the role of the build string is. The build string is (if not explicitly set) computed from the variant configuration. It serves as a mechanism to discern different build configurations that produce a package with the same name and version.
The hash is computed by dumping all of the variant configuration values that are used by a given recipe into a JSON file, and then hashing that JSON file.
For example, in our python
example, we would get a variant configuration file that looks something like:
{\n \"python\": \"3.8\"\n}\n
This JSON string is then hashed with the MD5 hash algorithm, and produces the hash. For certain packages (such as Python packages) special rules exists, and the py<Major.Minor>
version is prepended to the hash, so that the final hash would look something like py38h123123
.
"},{"location":"variants/#zip-keys","title":"Zip keys","text":"Zip keys modify how variants are combined. Usually, each variant key that has multiple entries is expanded to a build matrix. For example, if we have:
python: [\"3.8\", \"3.9\"]\nnumpy: [\"1.12\", \"1.14\"]\n
...then we obtain 4 variants for a recipe that uses both numpy
and python
:
- python 3.8, numpy 1.12\n- python 3.8, numpy 1.14\n- python 3.9, numpy 1.12\n- python 3.9, numpy 1.14\n
However, if we use the zip_keys
and specify:
zip_keys: [\"python\", \"numpy\"]\npython: [\"3.8\", \"3.9\"]\nnumpy: [\"1.12\", \"1.14\"]\n
...then the versions are \"zipped up\" and we only get 2 variants. Note that both python
and numpy
need to specify the exact same number of versions to make this work.
The resulting variants with the zip applied are:
- python 3.8, numpy 1.12\n- python 3.9, numpy 1.14\n
"},{"location":"variants/#pin-run-as-build","title":"Pin run as build","text":"The pin_run_as_build
key allows the user to inject additional pins. Usually, the run_exports
mechanism is used to specify constraints for runtime dependencies from build time dependencies, but pin_run_as_build
offers a mechanism to override that if the package does not contain a run exports file.
For example:
pin_run_as_build:\n libcurl:\n min_pin: 'x'\n max_pin: 'x'\n
If we now have a recipe that uses libcurl
in the host
and run
dependencies like:
requirements:\n host:\n - libcurl\n run:\n - libcurl\n
During resolution, libcurl
might be evaluated to libcurl 8.0.1 h13284
. Our new runtime dependency then looks like:
requirements:\n host:\n - libcurl 8.0.1 h13284\n run:\n - libcurl >=8,<9\n
"},{"location":"variants/#prioritizing-variants","title":"Prioritizing variants","text":"You might produce multiple variants for a package, but want to define a priority for a given variant. The variant with the highest priority would be the default package that is selected by the resolver.
There are two mechanisms to make this possible: mutex
packages and the down_prioritize_variant
option in the recipe.
"},{"location":"variants/#the-down_prioritize_variant-option","title":"The down_prioritize_variant
option","text":"Note
It is not always necessary to use the down_prioritize_variant
option - only if the solver has no other way to prefer a given variant. For example, if you have a package that has multiple variants for different Python versions, the solver will automatically prefer the variant with the highest Python version.
The down_prioritize_variant
option allows you to specify a variant that should be down-prioritized. For example:
recipe.yamlbuild:\n variant:\n use_keys:\n # use cuda from the variant config, e.g. to build multiple CUDA variants\n - cuda\n # this will down-prioritize the cuda variant versus other variants of the package\n down_prioritize_variant: ${{ 1 if cuda else 0 }}\n
"},{"location":"variants/#mutex-packages","title":"Mutex packages","text":"Another way to make sure the right variants are selected are \"mutex\" packages. A mutex package is a package that is mutually exclusive. We use the fact that only one package of a given name can be installed at a time (the solver has to choose).
A mutex package might be useful to make sure that all packages that depend on BLAS are compiled against the same BLAS implementation. The mutex package will serve the purpose that \"openblas
\" and \"mkl
\" can never be installed at the same time.
We could define a BLAS mutex package like this:
variant_config.yamlblas_variant:\n - \"openblas\"\n - \"mkl\"\n
And then the recipe.yaml
for the mutex
package could look like this:
recipe.yamlpackage:\n name: blas_mutex\n version: 1.0\n\nbuild:\n string: ${{ blas_variant }}${{ hash }}_${{ build_number }}\n variant_config:\n # make sure that `openblas` is preferred over `mkl`\n down_prioritize_variant: ${{ 1 if blas_variant == \"mkl\" else 0 }}\n
This will create two package: blas_mutex-1.0-openblas
and blas_mutex-1.0-mkl
. Only one of these packages can be installed at a time because they share the same name. The solver will then only select one of these two packages.
The blas
package in turn should have a run_export
for the blas_mutex
package, so that any package that links against blas
also has a dependency on the correct blas_mutex
package:
recipe.yamlpackage:\n name: openblas\n version: 1.0\n\nrequirements:\n # any package depending on openblas should also depend on the correct blas_mutex package\n run_export:\n # Add a run export on _any_ version of the blas_mutex package whose build string starts with \"openblas\"\n - blas_mutex * openblas*\n
Then the recipe of a package that wants to build two variants, one for openblas
and one for mkl
could look like this:
recipe.yamlpackage:\n name: fastnumerics\n version: 1.0\n\nrequirements:\n host:\n # build against both openblas and mkl\n - ${{ blas_variant }}\n run:\n # implicitly adds the correct blas_mutex package through run exports\n # - blas_mutex * ${{ blas_variant }}*\n
"},{"location":"reference/cli/","title":"Command-Line Help for rattler-build
","text":"This document contains the help content for the rattler-build
command-line program.
"},{"location":"reference/cli/#rattler-build","title":"rattler-build
","text":"Usage: rattler-build [OPTIONS] [COMMAND]
"},{"location":"reference/cli/#subcommands","title":"Subcommands:","text":" build
\u2014 Build a package from a recipe test
\u2014 Run a test for a single package rebuild
\u2014 Rebuild a package from a package file instead of a recipe upload
\u2014 Upload a package completion
\u2014 Generate shell completion script generate-recipe
\u2014 Generate a recipe from PyPI or CRAN auth
\u2014 Handle authentication to external channels "},{"location":"reference/cli/#options","title":"Options:","text":" -v
, --verbose
Increase logging verbosity
-q
, --quiet
Decrease logging verbosity
--log-style <LOG_STYLE>
Logging style
Default value: fancy
Possible values: fancy
: Use fancy logging output json
: Use JSON logging output plain
: Use plain logging output --wrap-log-lines <WRAP_LOG_LINES>
Wrap log lines at the terminal width. This is automatically disabled on CI (by detecting the CI
environment variable)
Possible values: true
, false
--color <COLOR>
Enable or disable colored output from rattler-build. Also honors the CLICOLOR
and CLICOLOR_FORCE
environment variable
Default value: auto
Possible values: always
: Always use colors never
: Never use colors auto
: Use colors when the output is a terminal "},{"location":"reference/cli/#build","title":"build
","text":"Build a package from a recipe
Usage: rattler-build build [OPTIONS]
"},{"location":"reference/cli/#options_1","title":"Options:","text":" -r
, --recipe <RECIPE>
The recipe file or directory containing recipe.yaml
. Defaults to the current directory
--recipe-dir <RECIPE_DIR>
The directory that contains recipes
--up-to <UP_TO>
Build recipes up to the specified package
--build-platform <BUILD_PLATFORM>
The build platform to use for the build (e.g. for building with emulation, or rendering)
Default value: current platform --target-platform <TARGET_PLATFORM>
The target platform for the build
--host-platform <HOST_PLATFORM>
The host platform for the build. If set, it will be used to determine also the target_platform (as long as it is not noarch)
Default value: current platform -c
, --channel <CHANNEL>
Add a channel to search for dependencies in
Default value: conda-forge
-m
, --variant-config <VARIANT_CONFIG>
Variant configuration files for the build
--ignore-recipe-variants
Do not read the variants.yaml
file next to a recipe
Possible values: true
, false
--render-only
Render the recipe files without executing the build
Possible values: true
, false
--with-solve
Render the recipe files with solving dependencies
Possible values: true
, false
--keep-build
Keep intermediate build artifacts after the build
Possible values: true
, false
--no-build-id
Don't use build id(timestamp) when creating build directory name
Possible values: true
, false
--compression-threads <COMPRESSION_THREADS>
The number of threads to use for compression (only relevant when also using --package-format conda
)
--use-zstd
Enable support for repodata.json.zst
Default value: true
Possible values: true
, false
--use-bz2
Enable support for repodata.json.bz2
Default value: true
Possible values: true
, false
--experimental
Enable experimental features
Possible values: true
, false
--auth-file <AUTH_FILE>
Path to an auth-file to read authentication information from
--channel-priority <CHANNEL_PRIORITY>
Channel priority to use when solving
--tui
Launch the terminal user interface
Default value: false
Possible values: true
, false
--extra-meta <EXTRA_META>
Extra metadata to include in about.json
"},{"location":"reference/cli/#modifying-result","title":"Modifying result","text":" --package-format <PACKAGE_FORMAT>
The package format to use for the build. Can be one of tar-bz2
or conda
. You can also add a compression level to the package format, e.g. tar-bz2:<number>
(from 1 to 9) or conda:<number>
(from -7 to 22).
--no-include-recipe
Don't store the recipe in the final package
Possible values: true
, false
--no-test
Do not run tests after building (deprecated, use --test=skip
instead)
Default value: false
Possible values: true
, false
--test <TEST>
The strategy to use for running tests
Default value: native-and-emulated
Possible values: skip
: Skip the tests native
: Run the tests only if the build platform is the same as the host platform. Otherwise, skip the tests. If the target platform is noarch, the tests are always executed native-and-emulated
: Always run the tests --color-build-log
Don't force colors in the output of the build script
Default value: true
Possible values: true
, false
--output-dir <OUTPUT_DIR>
Output directory for build artifacts.
--skip-existing <SKIP_EXISTING>
Whether to skip packages that already exist in any channel If set to none
, do not skip any packages, default when not specified. If set to local
, only skip packages that already exist locally, default when using --skip-existing. If set to
all`, skip packages that already exist in any channel
Default value: none
Possible values: none
: Do not skip any packages local
: Skip packages that already exist locally all
: Skip packages that already exist in any channel --noarch-build-platform <NOARCH_BUILD_PLATFORM>
Define a \"noarch platform\" for which the noarch packages will be built for. The noarch builds will be skipped on the other platforms
"},{"location":"reference/cli/#test","title":"test
","text":"Run a test for a single package
This creates a temporary directory, copies the package file into it, and then runs the indexing. It then creates a test environment that installs the package and any extra dependencies specified in the package test dependencies file.
With the activated test environment, the packaged test files are run:
info/test/run_test.sh
or info/test/run_test.bat
on Windows * info/test/run_test.py
These test files are written at \"package creation time\" and are part of the package.
Usage: rattler-build test [OPTIONS] --package-file <PACKAGE_FILE>
"},{"location":"reference/cli/#options_2","title":"Options:","text":" -c
, --channel <CHANNEL>
Channels to use when testing
-p
, --package-file <PACKAGE_FILE>
The package file to test
--compression-threads <COMPRESSION_THREADS>
The number of threads to use for compression
--use-zstd
Enable support for repodata.json.zst
Default value: true
Possible values: true
, false
--use-bz2
Enable support for repodata.json.bz2
Default value: true
Possible values: true
, false
--experimental
Enable experimental features
Possible values: true
, false
--auth-file <AUTH_FILE>
Path to an auth-file to read authentication information from
--channel-priority <CHANNEL_PRIORITY>
Channel priority to use when solving
"},{"location":"reference/cli/#modifying-result_1","title":"Modifying result","text":""},{"location":"reference/cli/#rebuild","title":"rebuild
","text":"Rebuild a package from a package file instead of a recipe
Usage: rattler-build rebuild [OPTIONS] --package-file <PACKAGE_FILE> --test <TEST>
"},{"location":"reference/cli/#options_3","title":"Options:","text":" -p
, --package-file <PACKAGE_FILE>
The package file to rebuild
--no-test
Do not run tests after building (deprecated, use --test=skip
instead)
Default value: false
Possible values: true
, false
--compression-threads <COMPRESSION_THREADS>
The number of threads to use for compression
--use-zstd
Enable support for repodata.json.zst
Default value: true
Possible values: true
, false
--use-bz2
Enable support for repodata.json.bz2
Default value: true
Possible values: true
, false
--experimental
Enable experimental features
Possible values: true
, false
--auth-file <AUTH_FILE>
Path to an auth-file to read authentication information from
--channel-priority <CHANNEL_PRIORITY>
Channel priority to use when solving
"},{"location":"reference/cli/#modifying-result_2","title":"Modifying result","text":" --test <TEST>
The strategy to use for running tests
Possible values: skip
: Skip the tests native
: Run the tests only if the build platform is the same as the host platform. Otherwise, skip the tests. If the target platform is noarch, the tests are always executed native-and-emulated
: Always run the tests --output-dir <OUTPUT_DIR>
Output directory for build artifacts.
"},{"location":"reference/cli/#upload","title":"upload
","text":"Upload a package
Usage: rattler-build upload [OPTIONS] [PACKAGE_FILES]... <COMMAND>
"},{"location":"reference/cli/#subcommands_1","title":"Subcommands:","text":" quetz
\u2014 Upload to a Quetz server. Authentication is used from the keychain / auth-file artifactory
\u2014 Options for uploading to a Artifactory channel. Authentication is used from the keychain / auth-file prefix
\u2014 Options for uploading to a prefix.dev server. Authentication is used from the keychain / auth-file anaconda
\u2014 Options for uploading to a Anaconda.org server "},{"location":"reference/cli/#arguments","title":"Arguments:","text":""},{"location":"reference/cli/#options_4","title":"Options:","text":" --use-zstd
Enable support for repodata.json.zst
Default value: true
Possible values: true
, false
--use-bz2
Enable support for repodata.json.bz2
Default value: true
Possible values: true
, false
--experimental
Enable experimental features
Possible values: true
, false
--auth-file <AUTH_FILE>
Path to an auth-file to read authentication information from
--channel-priority <CHANNEL_PRIORITY>
Channel priority to use when solving
"},{"location":"reference/cli/#modifying-result_3","title":"Modifying result","text":""},{"location":"reference/cli/#quetz","title":"quetz
","text":"Upload to a Quetz server. Authentication is used from the keychain / auth-file
Usage: rattler-build upload quetz [OPTIONS] --url <URL> --channel <CHANNEL>
"},{"location":"reference/cli/#options_5","title":"Options:","text":" -u
, --url <URL>
The URL to your Quetz server
-c
, --channel <CHANNEL>
The URL to your channel
-a
, --api-key <API_KEY>
The Quetz API key, if none is provided, the token is read from the keychain / auth-file
"},{"location":"reference/cli/#artifactory","title":"artifactory
","text":"Options for uploading to a Artifactory channel. Authentication is used from the keychain / auth-file
Usage: rattler-build upload artifactory [OPTIONS] --url <URL> --channel <CHANNEL>
"},{"location":"reference/cli/#options_6","title":"Options:","text":" -u
, --url <URL>
The URL to your Artifactory server
-c
, --channel <CHANNEL>
The URL to your channel
-r
, --username <USERNAME>
Your Artifactory username
-p
, --password <PASSWORD>
Your Artifactory password
"},{"location":"reference/cli/#prefix","title":"prefix
","text":"Options for uploading to a prefix.dev server. Authentication is used from the keychain / auth-file
Usage: rattler-build upload prefix [OPTIONS] --channel <CHANNEL>
"},{"location":"reference/cli/#options_7","title":"Options:","text":" -u
, --url <URL>
The URL to the prefix.dev server (only necessary for self-hosted instances)
Default value: https://prefix.dev
-c
, --channel <CHANNEL>
The channel to upload the package to
-a
, --api-key <API_KEY>
The prefix.dev API key, if none is provided, the token is read from the keychain / auth-file
"},{"location":"reference/cli/#anaconda","title":"anaconda
","text":"Options for uploading to a Anaconda.org server
Usage: rattler-build upload anaconda [OPTIONS] --owner <OWNER>
"},{"location":"reference/cli/#options_8","title":"Options:","text":" -o
, --owner <OWNER>
The owner of the distribution (e.g. conda-forge or your username)
-c
, --channel <CHANNEL>
The channel / label to upload the package to (e.g. main / rc)
-a
, --api-key <API_KEY>
The Anaconda API key, if none is provided, the token is read from the keychain / auth-file
-u
, --url <URL>
The URL to the Anaconda server
Default value: https://api.anaconda.org
-f
, --force
Replace files on conflict
Default value: false
Possible values: true
, false
"},{"location":"reference/cli/#completion","title":"completion
","text":"Generate shell completion script
Usage: rattler-build completion --shell <SHELL>
"},{"location":"reference/cli/#options_9","title":"Options:","text":""},{"location":"reference/cli/#generate-recipe","title":"generate-recipe
","text":"Generate a recipe from PyPI or CRAN
Usage: rattler-build generate-recipe <COMMAND>
"},{"location":"reference/cli/#subcommands_2","title":"Subcommands:","text":" pypi
\u2014 Generate a recipe for a Python package from PyPI cran
\u2014 Generate a recipe for an R package from CRAN "},{"location":"reference/cli/#pypi","title":"pypi
","text":"Generate a recipe for a Python package from PyPI
Usage: rattler-build generate-recipe pypi [OPTIONS] <PACKAGE>
"},{"location":"reference/cli/#arguments_1","title":"Arguments:","text":""},{"location":"reference/cli/#options_10","title":"Options:","text":" -w
, --write
Whether to write the recipe to a folder
Possible values: true
, false
-u
, --use-mapping
Whether to use the conda-forge PyPI name mapping
Default value: true
Possible values: true
, false
-t
, --tree
Whether to generate recipes for all dependencies
Possible values: true
, false
"},{"location":"reference/cli/#cran","title":"cran
","text":"Generate a recipe for an R package from CRAN
Usage: rattler-build generate-recipe cran [OPTIONS] <PACKAGE>
"},{"location":"reference/cli/#arguments_2","title":"Arguments:","text":""},{"location":"reference/cli/#options_11","title":"Options:","text":" -u
, --universe <UNIVERSE>
The R Universe to fetch the package from (defaults to cran
)
-t
, --tree
Whether to create recipes for the whole dependency tree or not
Possible values: true
, false
-w
, --write
Whether to write the recipe to a folder
Possible values: true
, false
"},{"location":"reference/cli/#auth","title":"auth
","text":"Handle authentication to external channels
Usage: rattler-build auth <COMMAND>
"},{"location":"reference/cli/#subcommands_3","title":"Subcommands:","text":" login
\u2014 Store authentication information for a given host logout
\u2014 Remove authentication information for a given host "},{"location":"reference/cli/#login","title":"login
","text":"Store authentication information for a given host
Usage: rattler-build auth login [OPTIONS] <HOST>
"},{"location":"reference/cli/#arguments_3","title":"Arguments:","text":""},{"location":"reference/cli/#options_12","title":"Options:","text":" --token <TOKEN>
The token to use (for authentication with prefix.dev)
--username <USERNAME>
The username to use (for basic HTTP authentication)
--password <PASSWORD>
The password to use (for basic HTTP authentication)
--conda-token <CONDA_TOKEN>
The token to use on anaconda.org / quetz authentication
"},{"location":"reference/cli/#logout","title":"logout
","text":"Remove authentication information for a given host
Usage: rattler-build auth logout <HOST>
"},{"location":"reference/cli/#arguments_4","title":"Arguments:","text":" This document was generated automatically by clap-markdown
.
"},{"location":"reference/jinja/","title":"Jinja","text":"rattler-build
comes with a couple of useful Jinja functions and filters that can be used in the recipe.
"},{"location":"reference/jinja/#functions","title":"Functions","text":""},{"location":"reference/jinja/#the-compiler-function","title":"The compiler function","text":"The compiler function can be used to put together a compiler that works for the current platform and the compilation \"target_platform
\". The syntax looks like: ${{ compiler('c') }}
where 'c'
signifies the programming language that is used.
This function evaluates to <compiler>_<target_platform> <compiler_version>
. For example, when compiling on linux
and to linux-64
, this function evaluates to gcc_linux-64
.
The values can be influenced by the variant_configuration
. The <lang>_compiler
and <lang>_compiler_version
variables are the keys with influence. See below for an example:
"},{"location":"reference/jinja/#usage-in-a-recipe","title":"Usage in a recipe","text":"recipe.yamlrequirements:\n build:\n - ${{ compiler('c') }}\n
With a corresponding variant_configuration:
variant_configuration.yamlc_compiler:\n- clang\nc_compiler_version:\n- 9.0\n
The variables shown above would select the clang
compiler in version 9.0
. Note that the final output will still contain the target_platform
, so that the full compiler will read clang_linux-64 9.0
when compiling with --target-platform linux-64
.
rattler-build
defines some default compilers for the following languages (inherited from conda-build
):
c
: gcc
on Linux, clang
on osx
and vs2017
on Windows cxx
: gxx
on Linux, clangxx
on osx
and vs2017
on Windows fortran
: gfortran
on Linux, gfortran
on osx
and vs2017
on Windows rust
: rust
"},{"location":"reference/jinja/#the-stdlib-function","title":"The stdlib
function","text":"The stdlib
function closely mirrors the compiler function. It can be used to put together a standard library that works for the current platform and the compilation \"target_platform
\".
Usage: ${{ stdlib('c') }}
Results in <stdlib>_<target_platform> <stdlib_version>
. And uses the variant variables <lang>_stdlib
and <lang>_stdlib_version
to influence the output.
"},{"location":"reference/jinja/#usage-in-a-recipe_1","title":"Usage in a recipe:","text":"recipe.yamlrequirements:\n build:\n # these are usually paired!\n - ${{ compiler('c') }}\n - ${{ stdlib('c') }}\n
With a corresponding variant_configuration:
variant_configuration.yaml# these are the values `conda-forge` uses in their pinning file\n# found at https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml\nc_stdlib:\n- sysroot\nc_stdlib_version:\n- 2.17\n
"},{"location":"reference/jinja/#the-pin-functions","title":"The pin
functions","text":"A pin is created based on the version input (from a subpackage or a package resolution).
The pin functions take the following three arguments:
lower_bound
(default: \"x.x.x.x.x.x\"
): The lower bound pin expression to be used. When set to None
, no lower bound is set. upper_bound
(default: \"x\"
): The maximum pin to be used. When set to None
, no upper bound is set. The lower bound and upper bound can either be a \"pin expression\" (only x
and .
are allowed) or a hard-coded version string.
A \"pin expression\" is applied to the version input to create the lower and upper bounds. For example, if the version is 3.10.5
with a lower_bound=\"x.x\", upper_bound=\"x.x.x\"
, the lower bound will be 3.10
and the upper bound will be 3.10.6.0a0
. A pin expression for the upper_bound
will increment the last selected segment of the version by 1
, and append .0a0
to the end to prevent any alpha versions from being selected.
If the last segment of the version contains a letter (e.g. 9e
or 1.1.1j
), then incrementing the version will set that letter to a
, e.g. 9e
will become 10a
, and 1.1.1j
will become 1.1.2a
. In this case, also no 0a0
is appended to the end.
Sometimes you want to strongly connect your outputs. This can be achieved with the following input:
exact=True
(default: False
): This will pin the version exactly to the version of the output, incl. the build string. To override the lower or upper bound with a hard-coded value, you can use the following input:
lower_bound
(default: None
): This will override the lower bound with the given value. upper_bound
(default: None
): This will override the upper bound with the given value. Both lower_bound
and upper_bound
expect a valid version string (e.g. 1.2.3
).
To add an build-string matching expression, you can use the build
argument:
build
(default: None
): This will add a build string matching expression to the pin. The build string matching expression is a string that is used to match the build string with the match spec. For example, if the build string is py38_0
, the build string matching expression could be py38*
or to match exactly py38_0
. The build
and exact
options are mutually exclusive. "},{"location":"reference/jinja/#the-pin_subpackage-function","title":"The pin_subpackage
function","text":" ${{ pin_subpackage(\"mypkg\", lower_bound=\"x.x\", upper_bound=\"x.x\") }}
creates a pin to another output in the recipe. With an input of 3.1.5
, this would create a pin of mypkg >=3.1,<3.2.0a0
. ${{ pin_subpackage(\"other_output\", exact=True) }}
creates a pin to another output in the recipe with an exact version. ${{ pin_subpackage(\"other_output\", lower_bound=\"1.2.3\", upper_bound=\"1.2.4\") }}
creates a pin to another output in the recipe with a lower bound of 1.2.3
and an upper bound of 1.2.4
. This is equivalent to writing other_output >=1.2.3,<1.2.4
. ${{ pin_subpackage(\"foo\", build=\"py38*\") }}
creates a matchspec like foo >=3.1,<3.2.0a0 py38*
. "},{"location":"reference/jinja/#the-pin_compatible-function","title":"The pin_compatible
function","text":"The pin compatible function works exactly as the pin_subpackage
function, but it pins the package in the run requirements based on the resolved package of the host
or build
section.
pin_compatible
pins a package in the run requirements based on the resolved package of the host
or build
section. "},{"location":"reference/jinja/#the-cdt-function","title":"The cdt
function","text":" ${{ cdt(\"mypkg\") }}
creates a cross-dependency to another output in the recipe. This function helps add Core Dependency Tree packages as dependencies by converting packages as required according to hard-coded logic. See below for an example of how this function can be used:
# on x86_64 system\ncdt('package-name') # outputs: package-name-cos6-x86_64\n# on aarch64 system\ncdt('package-name') # outputs: package-name-cos6-aarch64\n
"},{"location":"reference/jinja/#the-hash-variable","title":"The hash
variable","text":" ${{ hash }}
is the variant hash and is useful in the build string computation. "},{"location":"reference/jinja/#the-version_to_buildstring-function","title":"The version_to_buildstring
function","text":" ${{ python | version_to_buildstring }}
converts a version from the variant to a build string (it removes the .
character and takes only the first two elements of the version). "},{"location":"reference/jinja/#the-env-object","title":"The env
object","text":"You can use the env
object to retrieve environment variables and forward them to your build script. ${{ env.get(\"MY_ENV_VAR\") }}
will return the value of the environment variable MY_ENV_VAR
or throw an error if it is not set.
To supply a default value when the environment variable is not set, you can use ${{ env.get(\"MY_ENV_VAR\", default=\"default_value\") }}
. In this case, if MY_ENV_VAR
is not set, the value default_value
will be returned (and no error is thrown).
You can also check for the existence of an environment variable:
${{ env.exists(\"MY_ENV_VAR\") }}
will return true
if the environment variable MY_ENV_VAR
is set and false
otherwise. "},{"location":"reference/jinja/#filters","title":"Filters","text":"A feature of jinja
is called \"filters\". Filters are functions that can be applied to variables in a template expression.
The syntax for a filter is {{ variable | filter_name }}
. A filter can also take arguments, such as ... | replace('foo', 'bar')
.
The following Jinja filters are available, taken from the upstream minijinja
library:
replace
: replace a string with another string (e.g. \"{{ 'foo' | replace('oo', 'aa') }}\"
will return \"faa\"
) lower
: convert a string to lowercase (e.g. \"{{ 'FOO' | lower }}\"
will return \"foo\"
) upper
: convert a string to uppercase (e.g. \"{{ 'foo' | upper }}\"
will return \"FOO\"
) - int
: convert a string to an integer (e.g. \"{{ '42' | int }}\"
will return 42
) abs
: return the absolute value of a number (e.g. \"{{ -42 | abs }}\"
will return 42
) bool
: convert a value to a boolean (e.g. \"{{ 'foo' | bool }}\"
will return true
) default
: return a default value if the value is falsy (e.g. \"{{ '' | default('foo') }}\"
will return \"foo\"
) first
: return the first element of a list (e.g. \"{{ [1, 2, 3] | first }}\"
will return 1
) - last
: return the last element of a list (e.g. \"{{ [1, 2, 3] | last }}\"
will return 3
) length
: return the length of a list (e.g. \"{{ [1, 2, 3] | length }}\"
will return 3
) list
: convert a string to a list (e.g. \"{{ 'foo' | list }}\"
will return ['f', 'o', 'o']
) join
: join a list with a separator (e.g. \"{{ [1, 2, 3] | join('.') }}\"
will return \"1.2.3\"
) min
: return the minimum value of a list (e.g. \"{{ [1, 2, 3] | min }}\"
will return 1
) max
: return the maximum value of a list (e.g. \"{{ [1, 2, 3] | max }}\"
will return 3
) reverse
: reverse a list (e.g. \"{{ [1, 2, 3] | reverse }}\"
will return [3, 2, 1]
) slice
: slice a list (e.g. \"{{ [1, 2, 3] | slice(1, 2) }}\"
will return [2]
) batch
: This filter works pretty much like slice
just the other way round. It returns a list of lists with the given number of items. If you provide a second parameter this is used to fill up missing items. sort
: sort a list (e.g. \"{{ [3, 1, 2] | sort }}\"
will return [1, 2, 3]
) trim
: remove leading and trailing whitespace from a string (e.g. \"{{ ' foo ' | trim }}\"
will return \"foo\"
) unique
: remove duplicates from a list (e.g. \"{{ [1, 2, 1, 3] | unique }}\"
will return [1, 2, 3]
) split
: split a string into a list (e.g. \"{{ '1.2.3' | split('.') | list }}\"
will return ['1', '2', '3']
). By default, splits on whitespace. Removed filters The following filters are removed from the builtins:
attr
indent
select
selectattr
dictsort
reject
rejectattr
round
map
title
capitalize
urlencode
escape
pprint
safe
items
float
tojson
"},{"location":"reference/jinja/#extra-filters-for-recipes","title":"Extra filters for recipes","text":""},{"location":"reference/jinja/#the-version_to_buildstring-filter","title":"The version_to_buildstring
filter","text":" ${{ python | version_to_buildstring }}
converts a version from the variant to a build string (it removes the .
character and takes only the first two elements of the version). For example the following:
context:\n cuda: \"11.2.0\"\n\nbuild:\n string: ${{ hash }}_cuda${{ cuda_version | version_to_buildstring }}\n
Would evaluate to a abc123_cuda112
(assuming the hash was abc123
).
"},{"location":"reference/jinja/#various-remarks","title":"Various remarks","text":""},{"location":"reference/jinja/#inline-conditionals-with-jinja","title":"Inline conditionals with Jinja","text":"The new recipe format allows for inline conditionals with Jinja. If they are falsey, and no else
branch exists, they will render to an empty string (which is, for example in a list or dictionary, equivalent to a YAML null
).
When a recipe is rendered, all values that are null
must be filtered from the resulting YAML.
requirements:\n host:\n - ${{ \"numpy\" if cuda == \"yes\" }}\n
If cuda
is not equal to yes, the first item of the host requirements will be empty (null) and thus filtered from the final list.
This must also work for dictionary values. For example:
build:\n number: ${{ 100 if cuda == \"yes\" }}\n # or an `else` branch can be used, of course\n number: ${{ 100 if cuda == \"yes\" else 0 }}\n
"},{"location":"reference/recipe_file/","title":"The recipe spec","text":"rattler-build
implements a new recipe spec, different from the traditional \"meta.yaml
\" file used in conda-build
. A recipe has to be stored as a recipe.yaml
file.
"},{"location":"reference/recipe_file/#history","title":"History","text":"A discussion was started on what a new recipe spec could or should look like. The fragments of this discussion can be found here.
The reason for a new spec are:
make it easier to parse (i.e. \"pure YAML\"); conda-build
uses a mix of comments and Jinja to achieve a great deal of flexibility, but it's hard to parse the recipe with a computer iron out some inconsistencies around multiple outputs (build
vs. build/script
and more) remove any need for recursive parsing & solving finally, the initial implementation in boa
relied on conda-build
; rattler-build
removes any dependency on Python or conda-build
and reimplements everything in Rust "},{"location":"reference/recipe_file/#major-differences-from-conda-build","title":"Major differences from conda-build
","text":""},{"location":"reference/recipe_file/#spec","title":"Spec","text":"The recipe spec has the following parts:
context
: to set up variables that can later be used in Jinja string interpolation package
: defines name, version etc. of the top-level package source
: points to the sources that need to be downloaded in order to build the recipe build
: defines how to build the recipe and what build number to use requirements
: defines requirements of the top-level package test
: defines tests for the top-level package outputs
: a recipe can have multiple outputs. Each output can and should have a package
, requirements
and test
section "},{"location":"reference/recipe_file/#spec-reference","title":"Spec reference","text":"The spec is also made available through a JSON Schema (which is used for validation). The schema (and pydantic
source file) can be found in this repository: recipe-format
To use with VSCode(yaml-plugin) and other IDEs: Either start the document with the following line:
# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json\n
Or, using yaml.schemas
, yaml.schemas: {\n \"https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json\": \"**/recipe.yaml\",\n}\n
Read more about this here. See more in the automatic linting chapter.
"},{"location":"reference/recipe_file/#examples","title":"Examples","text":"recipe.yaml# this sets up \"context variables\" (in this case name and version) that\n# can later be used in Jinja expressions\ncontext:\n version: 1.1.0\n name: imagesize\n\n# top level package information (name and version)\npackage:\n name: ${{ name }}\n version: ${{ version }}\n\n# location to get the source from\nsource:\n url: https://pypi.io/packages/source/${{ name[0] }}/${{ name }}/${{ name }}-${{ version }}.tar.gz\n sha256: f3832918bc3c66617f92e35f5d70729187676313caa60c187eb0f28b8fe5e3b5\n\n# build number (should be incremented if a new build is made, but version is not incrementing)\nbuild:\n number: 1\n script: python -m pip install .\n\n# the requirements at build and runtime\nrequirements:\n host:\n - python\n - pip\n run:\n - python\n\n# tests to validate that the package works as expected\ntests:\n - python:\n imports:\n - imagesize\n\n# information about the package\nabout:\n homepage: https://github.com/shibukawa/imagesize_py\n license: MIT\n summary: 'Getting image size from png/jpeg/jpeg2000/gif file'\n description: |\n This module analyzes jpeg/jpeg2000/png/gif image header and\n return image size.\n repository: https://github.com/shibukawa/imagesize_py\n documentation: https://pypi.python.org/pypi/imagesize\n\n# the below is conda-forge specific!\nextra:\n recipe-maintainers:\n - somemaintainer\n
"},{"location":"reference/recipe_file/#package-section","title":"Package section","text":"Specifies package information.
package:\n name: bsdiff4\n version: \"2.1.4\"\n
name: The lower case name of the package. It may contain \"-
\", but no spaces. version: The version number of the package. Use the PEP-386 verlib conventions. Cannot contain \"-
\". YAML interprets version numbers such as 1.0 as floats, meaning that 0.10 will be the same as 0.1. To avoid this, put the version number in quotes so that it is interpreted as a string. "},{"location":"reference/recipe_file/#source-section","title":"Source section","text":"Specifies where the source code of the package is coming from. The source may come from a tarball file, git
, hg
, or svn
. It may be a local path and it may contain patches.
"},{"location":"reference/recipe_file/#source-from-tarball-or-zip-archive","title":"Source from tarball or zip
archive","text":"source:\n url: https://pypi.python.org/packages/source/b/bsdiff4/bsdiff4-1.1.4.tar.gz\n md5: 29f6089290505fc1a852e176bd276c43\n sha1: f0a2c9a30073449cfb7d171c57552f3109d93894\n sha256: 5a022ff4c1d1de87232b1c70bde50afbb98212fd246be4a867d8737173cf1f8f\n
If an extracted archive contains only 1 folder at its top level, its contents will be moved 1 level up, so that the extracted package contents sit in the root of the work folder.
"},{"location":"reference/recipe_file/#specifying-a-file-name","title":"Specifying a file name","text":"For URL and local paths you can specify a file name. If the source is an archive and a file name is set, automatic extraction is disabled.
source:\n url: https://pypi.python.org/packages/source/b/bsdiff4/bsdiff4-1.1.4.tar.gz\n # will put the file in the work directory as `bsdiff4-1.1.4.tar.gz`\n file_name: bsdiff4-1.1.4.tar.gz\n
"},{"location":"reference/recipe_file/#source-from-git","title":"Source from git
","text":"source:\n git: https://github.com/ilanschnell/bsdiff4.git\n # branch: master # note: defaults to fetching the repo's default branch\n
You can use rev
to pin the commit version directly:
source:\n git: https://github.com/ilanschnell/bsdiff4.git\n rev: \"50a1f7ed6c168eb0815d424cba2df62790f168f0\"\n
Or you can use the tag
:
source:\n git: https://github.com/ilanschnell/bsdiff4.git\n tag: \"1.1.4\"\n
git
can also be a relative path to the recipe directory:
source:\n git: ../../bsdiff4/.git\n tag: \"1.1.4\"\n
Furthermore, if you want to fetch just the current \"HEAD
\" (this may result in non-deterministic builds), then you can use depth
.
source:\n git: https://github.com/ilanschnell/bsdiff4.git\n depth: 1 # note: the behaviour defaults to -1\n
Note: tag
or rev
may not be available within commit depth range, hence we don't allow using rev
or the tag
and depth
of them together if not set to -1
.
source:\n git: https://github.com/ilanschnell/bsdiff4.git\n tag: \"1.1.4\"\n depth: 1 # error: use of `depth` with `rev` is invalid, they are mutually exclusive\n
When you want to use git-lfs
, you need to set lfs: true
. This will also pull the lfs
files from the repository.
source:\n git: ../../bsdiff4/.git\n tag: \"1.1.4\"\n lfs: true # note: defaults to false\n
"},{"location":"reference/recipe_file/#source-from-a-local-path","title":"Source from a local path","text":"If the path is relative, it is taken relative to the recipe directory. The source is copied to the work directory before building.
source:\n path: ../src\n use_gitignore: false # note: defaults to true\n
By default, all files in the local path that are ignored by git
are also ignored by rattler-build
. You can disable this behavior by setting use_gitignore
to false
.
"},{"location":"reference/recipe_file/#patches","title":"Patches","text":"Patches may optionally be applied to the source.
source:\n #[source information here]\n patches:\n - my.patch # the patch file is expected to be found in the recipe\n
"},{"location":"reference/recipe_file/#destination-path","title":"Destination path","text":"Within rattler-build
's work directory, you may specify a particular folder to place the source into. rattler-build
will always drop you into the same folder ([build folder]/work
), but it's up to you whether you want your source extracted into that folder, or nested deeper. This feature is particularly useful when dealing with multiple sources, but can apply to recipes with single sources as well.
source:\n #[source information here]\n target_directory: my-destination/folder\n
"},{"location":"reference/recipe_file/#source-from-multiple-sources","title":"Source from multiple sources","text":"Some software is most easily built by aggregating several pieces.
The syntax is a list of source dictionaries. Each member of this list follows the same rules as the single source. All features for each member are supported.
Example:
source:\n - url: https://package1.com/a.tar.bz2\n target_directory: stuff\n - url: https://package1.com/b.tar.bz2\n target_directory: stuff\n - git: https://github.com/mamba-org/boa\n target_directory: boa\n
Here, the two URL tarballs will go into one folder, and the git
repo is checked out into its own space. git
will not clone into a non-empty folder.
"},{"location":"reference/recipe_file/#build-section","title":"Build section","text":"Specifies build information.
Each field that expects a path can also handle a glob pattern. The matching is performed from the top of the build environment, so to match files inside your project you can use a pattern similar to the following one: \"**/myproject/**/*.txt\"
. This pattern will match any .txt
file found in your project. Quotation marks (\"\"
) are required for patterns that start with a *
.
Recursive globbing using **
is also supported.
"},{"location":"reference/recipe_file/#build-number-and-string","title":"Build number and string","text":"The build number should be incremented for new builds of the same version. The number defaults to 0
. The build string cannot contain \"-
\". The string defaults to the default rattler-build
build string plus the build number.
build:\n number: 1\n string: abc\n
"},{"location":"reference/recipe_file/#dynamic-linking","title":"Dynamic linking","text":"This section contains settings for the shared libraries and executables.
build:\n dynamic_linking:\n rpath_allowlist: [\"/usr/lib/**\"]\n
"},{"location":"reference/recipe_file/#python-entry-points","title":"Python entry points","text":"The following example creates a Python entry point named \"bsdiff4
\" that calls bsdiff4.cli.main_bsdiff4()
.
build:\n python:\n entry_points:\n - bsdiff4 = bsdiff4.cli:main_bsdiff4\n - bspatch4 = bsdiff4.cli:main_bspatch4\n
"},{"location":"reference/recipe_file/#script","title":"Script","text":"By default, rattler-build
uses a build.sh
file on Unix (macOS and Linux) and a build.bat
file on Windows, if they exist in the same folder as the recipe.yaml
file. With the script parameter you can either supply a different filename or write out short build scripts. You may need to use selectors to use different scripts for different platforms.
build:\n # A very simple build script\n script: pip install .\n\n # The build script can also be a list\n script:\n - pip install .\n - echo \"hello world\"\n - if: unix\n then:\n - echo \"unix\"\n
"},{"location":"reference/recipe_file/#skipping-builds","title":"Skipping builds","text":"Lists conditions under which rattler-build
should skip the build of this recipe. Particularly useful for defining recipes that are platform-specific. By default, a build is never skipped.
build:\n skip:\n - win\n ...\n
"},{"location":"reference/recipe_file/#architecture-independent-packages","title":"Architecture-independent packages","text":"Allows you to specify \"no architecture\" when building a package, thus making it compatible with all platforms and architectures. Architecture-independent packages can be installed on any platform.
Assigning the noarch
key as generic
tells conda
to not try any manipulation of the contents.
build:\n noarch: generic\n
noarch: generic
is most useful for packages such as static JavaScript assets and source archives. For pure Python packages that can run on any Python version, you can use the noarch: python
value instead:
build:\n noarch: python\n
Note
At the time of this writing, noarch
packages should not make use of preprocess-selectors: noarch
packages are built with the directives which evaluate to true
in the platform it is built on, which probably will result in incorrect/incomplete installation in other platforms.
"},{"location":"reference/recipe_file/#include-build-recipe","title":"Include build recipe","text":"The recipe and rendered recipe.yaml
file are included in the package_metadata
by default. You can disable this by passing --no-include-recipe
on the command line.
Note
There are many more options in the build section. These additional options control how variants are computed, prefix replacements, and more. See the full build options for more information.
"},{"location":"reference/recipe_file/#requirements-section","title":"Requirements section","text":"Specifies the build and runtime requirements. Dependencies of these requirements are included automatically.
Versions for requirements must follow the conda
/mamba
match specification. See build-version-spec
.
"},{"location":"reference/recipe_file/#build","title":"Build","text":"Tools required to build the package.
These packages are run on the build system and include things such as version control systems (git
, svn
) make tools (GNU make, Autotool, CMake) and compilers (real cross, pseudo-cross, or native when not cross-compiling), and any source pre-processors.
Packages which provide \"sysroot
\" files, like the CDT
packages (see below), also belong in the build
section.
requirements:\n build:\n - git\n - cmake\n
"},{"location":"reference/recipe_file/#host","title":"Host","text":"Represents packages that need to be specific to the target platform when the target platform is not necessarily the same as the native build platform. For example, in order for a recipe to be \"cross-capable\", shared libraries requirements must be listed in the host
section, rather than the build
section, so that the shared libraries that get linked are ones for the target platform, rather than the native build platform. You should also include the base interpreter for packages that need one. In other words, a Python package would list python
here and an R package would list mro-base
or r-base
.
requirements:\n build:\n - ${{ compiler('c') }}\n - if: linux\n then:\n - ${{ cdt('xorg-x11-proto-devel') }}\n host:\n - python\n
Note
When both \"build
\" and \"host
\" sections are defined, the build
section can be thought of as \"build tools\" - things that run on the native platform, but output results for the target platform (e.g. a cross-compiler that runs on linux-64
, but targets linux-armv7
).
The PREFIX
environment variable points to the host prefix. With respect to activation during builds, both the host and build environments are activated. The build prefix is activated before the host prefix so that the host prefix has priority over the build prefix. Executables that don't exist in the host prefix should be found in the build prefix.
The build
and host
prefixes are always separate when both are defined, or when ${{ compiler() }}
Jinja2 functions are used. The only time that build
and host
are merged is when the host
section is absent, and no ${{ compiler() }}
Jinja2 functions are used in meta.yaml
.
"},{"location":"reference/recipe_file/#run","title":"Run","text":"Packages required to run the package.
These are the dependencies that are installed automatically whenever the package is installed. Package names should follow the package match specifications.
requirements:\n run:\n - python\n - six >=1.8.0\n
To build a recipe against different versions of NumPy and ensure that each version is part of the package dependencies, list numpy
as a requirement in recipe.yaml
and use a conda_build_config.yaml
file with multiple NumPy versions.
"},{"location":"reference/recipe_file/#run-constraints","title":"Run constraints","text":"Packages that are optional at runtime but must obey the supplied additional constraint if they are installed.
Package names should follow the package match specifications.
requirements:\n run_constraints:\n - optional-subpackage ==${{ version }}\n
For example, let's say we have an environment that has package \"a\" installed at version 1.0. If we install package \"b\" that has a run_constraints
entry of \"a >1.0
\", then mamba
would need to upgrade \"a\" in the environment in order to install \"b\".
This is especially useful in the context of virtual packages, where the run_constraints
dependency is not a package that mamba
manages, but rather a virtual package that represents a system property that mamba
can't change. For example, a package on Linux may impose a run_constraints
dependency on __glibc >=2.12
. This is the version bound consistent with CentOS 6. Software built against glibc 2.12 will be compatible with CentOS 6. This run_constraints
dependency helps mamba
, conda
or pixi
tell the user that a given package can't be installed if their system glibc version is too old.
"},{"location":"reference/recipe_file/#run-exports","title":"Run exports","text":"Packages may have runtime requirements such as shared libraries (e.g. zlib
), which are required for linking at build time, and for resolving the link at run time. With run_exports
packages runtime requirements can be implicitly added. run_exports
are weak by default, these two requirements for the zlib
package are therefore equivalent:
recipe.yaml for zlib requirements:\n run_exports:\n - ${{ pin_subpackage('libzlib', exact=True) }}\n
recipe.yaml for zlib requirements:\n run_exports:\n weak:\n - ${{ pin_subpackage('libzlib', exact=True) }}\n
The alternative to weak
is strong
. For gcc
this would look like this:
recipe.yaml for gcc requirements:\n run_exports:\n strong:\n - ${{ pin_subpackage('libgcc', exact=True) }}\n
weak
exports will only be implicitly added as runtime requirement, if the package is a host dependency. strong
exports will be added for both build and host dependencies. In the following example you can see the implicitly added runtime dependencies.
recipe.yaml of some package using gcc and zlib requirements:\n build:\n - gcc # has a strong run export\n host:\n - zlib # has a (weak) run export\n # - libgcc <-- implicitly added by gcc\n run:\n # - libgcc <-- implicitly added by gcc\n # - libzlib <-- implicitly added by libzlib\n
"},{"location":"reference/recipe_file/#ignore-run-exports","title":"Ignore run exports","text":"There maybe cases where an upstream package has a problematic run_exports
constraint. You can ignore it in your recipe by listing the upstream package name in the ignore_run_exports
section in requirements
.
You can ignore them by package name, or by naming the runtime dependency directly.
requirements:\n ignore_run_exports:\n from_package:\n - zlib\n
Using a runtime dependency name:
requirements:\n ignore_run_exports:\n by_name:\n - libzlib\n
Note
ignore_run_exports
only applies to runtime dependencies coming from an upstream package.
"},{"location":"reference/recipe_file/#tests-section","title":"Tests section","text":"rattler-build
supports four different types of tests. The \"script test\" installs the package and runs a list of commands. The \"Python test\" attempts to import a list of Python modules and runs pip check
. The \"downstream test\" runs the tests of a downstream package that reverse depends on the package being built. And lastly, the \"package content test\" checks if the built package contains the mentioned items.
The tests
section is a list of these items:
tests:\n - script:\n - echo \"hello world\"\n requirements:\n run:\n - pytest\n files:\n source:\n - test-data.txt\n\n - python:\n imports:\n - bsdiff4\n pip_check: true # this is the default\n - downstream: numpy\n
"},{"location":"reference/recipe_file/#script-test","title":"Script test","text":"The script test has 3 top-level keys: script
, files
and requirements
. Only the script
key is required.
"},{"location":"reference/recipe_file/#test-commands","title":"Test commands","text":"Commands that are run as part of the test.
tests:\n - script:\n - echo \"hello world\"\n - bsdiff4 -h\n - bspatch4 -h\n
"},{"location":"reference/recipe_file/#extra-test-files","title":"Extra test files","text":"Test files that are copied from the source work directory into the temporary test directory and are needed during testing (note that the source work directory is otherwise not available at all during testing).
You can also include files that come from the recipe
folder. They are copied into the test directory as well.
At test execution time, the test directory is the current working directory.
tests:\n - script:\n - ls\n files:\n source:\n - myfile.txt\n - tests/\n - some/directory/pattern*.sh\n recipe:\n - extra-file.txt\n
"},{"location":"reference/recipe_file/#test-requirements","title":"Test requirements","text":"In addition to the runtime requirements, you can specify requirements needed during testing. The runtime requirements that you specified in the \"run
\" section described above are automatically included during testing (because the built package is installed as it regularly would be).
In the build
section you can specify additional requirements that are only needed on the build system for cross-compilation (e.g. emulators or compilers).
tests:\n - script:\n - echo \"hello world\"\n requirements:\n build:\n - myemulator\n run:\n - nose\n
"},{"location":"reference/recipe_file/#python-tests","title":"Python tests","text":"For this test type you can list a set of Python modules that need to be importable. The test will fail if any of the modules cannot be imported.
The test will also automatically run pip check
to check for any broken dependencies. This can be disabled by setting pip_check: false
in the YAML.
tests:\n - python:\n imports:\n - bsdiff4\n - bspatch4\n pip_check: true # can be left out because this is the default\n
Internally this will write a small Python script that imports the modules:
import bsdiff4\nimport bspatch4\n
"},{"location":"reference/recipe_file/#check-for-package-contents","title":"Check for package contents","text":"Checks if the built package contains the mentioned items. These checks are executed directly at the end of the build process to make sure that all expected files are present in the package.
tests:\n - package_contents:\n # checks for the existence of files inside $PREFIX or %PREFIX%\n # or, checks that there is at least one file matching the specified `glob`\n # pattern inside the prefix\n files:\n - etc/libmamba/test.txt\n - etc/libmamba\n - etc/libmamba/*.mamba.txt\n\n # checks for the existence of `mamba/api/__init__.py` inside of the\n # Python site-packages directory (note: also see Python import checks)\n site_packages:\n - mamba.api\n\n\n # looks in $PREFIX/bin/mamba for unix and %PREFIX%\\Library\\bin\\mamba.exe on Windows\n # note: also check the `commands` and execute something like `mamba --help` to make\n # sure things work fine\n bin:\n - mamba\n\n # searches for `$PREFIX/lib/libmamba.so` or `$PREFIX/lib/libmamba.dylib` on Linux or macOS,\n # on Windows for %PREFIX%\\Library\\lib\\mamba.dll & %PREFIX%\\Library\\bin\\mamba.bin\n lib:\n - mamba\n\n # searches for `$PREFIX/include/libmamba/mamba.hpp` on unix, and\n # on Windows for `%PREFIX%\\Library\\include\\libmamba\\mamba.hpp`\n include:\n - libmamba/mamba.hpp\n
"},{"location":"reference/recipe_file/#downstream-tests","title":"Downstream tests","text":"Warning
Downstream tests are not yet implemented in rattler-build
.
A downstream test can mention a single package that has a dependency on the package being built. The test will install the package and run the tests of the downstream package with our current package as a dependency.
Sometimes downstream packages do not resolve. In this case, the test is ignored.
tests:\n - downstream: numpy\n
"},{"location":"reference/recipe_file/#outputs-section","title":"Outputs section","text":"Explicitly specifies packaging steps. This section supports multiple outputs, as well as different package output types. The format is a list of mappings.
When using multiple outputs, certain top-level keys are \"forbidden\": package
and requirements
. Instead of package
, a top-level recipe
key can be defined. The recipe.name
is ignored but the recipe.version
key is used as default version for each output. Other \"top-level\" keys are merged into each output (e.g. the about
section) to avoid repetition. Each output is a complete recipe, and can have its own build
, requirements
, and test
sections.
recipe:\n # the recipe name is ignored\n name: some\n version: 1.0\n\noutputs:\n - package:\n # version is taken from recipe.version (1.0)\n name: some-subpackage\n\n - package:\n name: some-other-subpackage\n version: 2.0\n
Each output acts like an independent recipe and can have their own script
, build_number
, and so on.
outputs:\n - package:\n name: subpackage-name\n build:\n script: install-subpackage\n
If script
lacks a file extension, the appropriate extension for the platform will be appended, e.g. the above will run install-subpackage.sh
in bash
on most platforms and install-subpackage.bat
in cmd.exe
on Windows.
Each output is built independently. You should take care of not packaging the same files twice.
"},{"location":"reference/recipe_file/#subpackage-requirements","title":"Subpackage requirements","text":"Like a top-level recipe, a subpackage may have zero or more dependencies listed as build, host or run requirements.
The dependencies listed as subpackage build requirements are available only during the packaging phase of that subpackage.
outputs:\n - package:\n name: subpackage-name\n requirements:\n build:\n - some-dep\n run:\n - some-dep\n
You can also use the pin_subpackage
function to pin another output from the same recipe.
outputs:\n - package:\n name: libtest\n - package:\n name: test\n requirements:\n build:\n - ${{ pin_subpackage('libtest', max_pin='x.x') }}\n
The outputs are topologically sorted by the dependency graph which is taking the pin_subpackage
invocations into account. When using pin_subpackage(name, exact=True)
a special behavior is used where the name
package is injected as a \"variant\" and the variant matrix is expanded appropriately. For example, when you have the following situation, with a variant_config.yaml
file that contains openssl: [1, 3]
:
outputs:\n - package:\n name: libtest\n requirements:\n host:\n - openssl\n - package:\n name: test\n requirements:\n build:\n - ${{ pin_subpackage('libtest', exact=True) }}\n
Due to the variant config file, this will build two versions of libtest
. We will also build two versions of test
, one that depends on libtest (openssl 1)
and one that depends on libtest (openssl 3)
.
"},{"location":"reference/recipe_file/#about-section","title":"About section","text":"Specifies identifying information about the package. The information displays in the package server.
about:\n homepage: https://example.com/bsdiff4\n license: BSD-3-Clause # (1)!\n license_file: LICENSE\n summary: binary diff and patch using the BSDIFF4-format\n description: |\n Long description of bsdiff4 ...\n repository: https://github.com/ilanschnell/bsdiff4\n documentation: https://docs.com\n
Only the SPDX specifiers are allowed, more info here: SPDX If you want another license type LicenseRef-<YOUR-LICENSE>
can be used, e.g. license: LicenseRef-Proprietary
"},{"location":"reference/recipe_file/#license-file","title":"License file","text":"Adds a file containing the software license to the package metadata. Many licenses require the license statement to be distributed with the package. The filename is relative to the source or recipe directory. The value can be a single filename or a YAML list for multiple license files. Values can also point to directories with license information. Directory entries must end with a /
suffix (this is to lessen unintentional inclusion of non-license files; all the directory's contents will be unconditionally and recursively added).
about:\n license_file:\n - LICENSE\n - vendor-licenses/\n
"},{"location":"reference/recipe_file/#extra-section","title":"Extra section","text":"A schema-free area for storing non-conda
-specific metadata in standard YAML form.
Example: To store recipe maintainers information extra:\n maintainers:\n - name of maintainer\n
"},{"location":"reference/recipe_file/#templating-with-jinja","title":"Templating with Jinja","text":"rattler-build
supports limited Jinja templating in the recipe.yaml
file.
You can set up Jinja variables in the context
section:
context:\n name: \"test\"\n version: \"5.1.2\"\n # later keys can reference previous keys\n # and use jinja functions to compute new values\n major_version: ${{ version.split('.')[0] }}\n
Later in your recipe.yaml
you can use these values in string interpolation with Jinja:
source:\n url: https://github.com/mamba-org/${{ name }}/v${{ version }}.tar.gz\n
Jinja has built-in support for some common string manipulations.
In rattler-build, complex Jinja is completely disallowed as we try to produce YAML that is valid at all times. So you should not use any {% if ... %}
or similar Jinja constructs that produce invalid YAML. Furthermore, instead of plain double curly brackets Jinja statements need to be prefixed by $
, e.g. ${{ ... }}
:
package:\n name: {{ name }} # WRONG: invalid yaml\n name: ${{ name }} # correct\n
For more information, see the Jinja template documentation and the list of available environment variables env-vars
.
Jinja templates are evaluated during the build process.
"},{"location":"reference/recipe_file/#additional-jinja2-functionality-in-rattler-build","title":"Additional Jinja2 functionality in rattler-build","text":"Besides the default Jinja2 functionality, additional Jinja functions are available during the rattler-build
process: pin_compatible
, pin_subpackage
, and compiler
.
The compiler function takes c
, cxx
, fortran
and other values as argument and automatically selects the right (cross-)compiler for the target platform.
build:\n - ${{ compiler('c') }}\n
The pin_subpackage
function pins another package produced by the recipe with the supplied parameters.
Similarly, the pin_compatible
function will pin a package according to the specified rules.
"},{"location":"reference/recipe_file/#pin-expressions","title":"Pin expressions","text":"rattler-build
knows pin expressions. A pin expression can have a min_pin
, max_pin
and exact
value. A max_pin
and min_pin
are specified with a string containing only x
and .
, e.g. max_pin=\"x.x.x\"
would signify to pin the given package to <1.2.3
(if the package version is 1.2.2
, for example).
A pin with min_pin=\"x.x\",max_pin=\"x.x\"
for a package of version 1.2.2
would evaluate to >=1.2,<1.3.0a0
.
If exact=true
, then the hash
is included, and the package is pinned exactly, e.g. ==1.2.2 h1234
. This is a unique package variant that cannot exist more than once, and thus is \"exactly\" pinned.
"},{"location":"reference/recipe_file/#pin-subpackage","title":"Pin subpackage","text":"Pin subpackage refers to another package from the same recipe file. It is commonly used in the build/run_exports
section to export a run export from the package, or with multiple outputs to refer to a previous build.
It looks something like:
package:\n name: mypkg\n version: \"1.2.3\"\n\nrequirements:\n run_exports:\n # this will evaluate to `mypkg <1.3`\n - ${{ pin_subpackage(name, max_pin='x.x') }}\n
"},{"location":"reference/recipe_file/#pin-compatible","title":"Pin compatible","text":"Pin compatible lets you pin a package based on the version retrieved from the variant file (if the pinning from the variant file needs customization).
For example, if the variant specifies a pin for numpy: 1.11
, one can use pin_compatible
to relax it:
requirements:\n host:\n # this will select nupy 1.11\n - numpy\n run:\n # this will export `numpy >=1.11,<2`, instead of the stricter `1.11` pin\n - ${{ pin_compatible('numpy', min_pin='x.x', max_pin='x') }}\n
"},{"location":"reference/recipe_file/#the-env-jinja-functions","title":"The env Jinja functions","text":"You can access the current environment variables using the env
object in Jinja.
There are three functions:
env.get(\"ENV_VAR\")
will insert the value of \"ENV_VAR\" into the recipe. env.get(\"ENV_VAR\", default=\"undefined\")
will insert the value of ENV_VAR
into the recipe or, if ENV_VAR
is not defined, the specified default value (in this case \"undefined\") env.exists(\"ENV_VAR\")
returns a boolean true of false if the env var is set to any value This can be used for some light templating, for example:
build:\n string: ${{ env.get(\"GIT_BUILD_STRING\") }}_${{ PKG_HASH }}\n
"},{"location":"reference/recipe_file/#match-function","title":"match
function","text":"This function matches the first argument (the package version) against the second argument (the version spec) and returns the resulting boolean. This only works for packages defined in the \"variant_config.yaml\" file.
recipe.yamlmatch(python, '>=3.4')\n
For example, you could require a certain dependency only for builds against python 3.4 and above:
recipe.yamlrequirements:\n build:\n - if: match(python, '>=3.4')\n then:\n - some-dep\n
With a corresponding variant config that looks like the following:
variant_config.yamlpython: [\"3.2\", \"3.4\", \"3.6\"]\n
Example: match
usage example
"},{"location":"reference/recipe_file/#cdt-function","title":"cdt
function","text":"This function helps add Core Dependency Tree packages as dependencies by converting packages as required according to hard-coded logic.
# on x86_64 system\ncdt('package-name') # outputs: package-name-cos6-x86_64\n# on aarch64 system\ncdt('package-name') # outputs: package-name-cos6-aarch64\n
Example: cdt
usage example
"},{"location":"reference/recipe_file/#preprocessing-selectors","title":"Preprocessing selectors","text":"You can add selectors to any item, and the selector is evaluated in a preprocessing stage. If a selector evaluates to true
, the item is flattened into the parent element. If a selector evaluates to false
, the item is removed.
Selectors can use if ... then ... else
as follows:
source:\n - if: not win\n then:\n - url: http://path/to/unix/source\n else:\n - url: http://path/to/windows/source\n\n# or the equivalent with two if conditions:\n\nsource:\n - if: unix\n then:\n - url: http://path/to/unix/source\n - if: win\n then:\n - url: http://path/to/windows/source\n
A selector is a valid Python statement that is executed. You can read more about them in the \"Selectors in recipes\" chapter.
The use of the Python version selectors, py27
, py34
, etc. is discouraged in favor of the more general comparison operators. Additional selectors in this series will not be added to conda-build
.
Because the selector is any valid Python expression, complicated logic is possible:
- if: unix and not win\n then: ...\n- if: (win or linux) and not py27\n then: ...\n
Lists are automatically \"merged\" upwards, so it is possible to group multiple items under a single selector:
tests:\n - script:\n - if: unix\n then:\n - test -d ${PREFIX}/include/xtensor\n - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake\n - if: win\n then:\n - if not exist %LIBRARY_PREFIX%\\include\\xtensor\\xarray.hpp (exit 1)\n - if not exist %LIBRARY_PREFIX%\\lib\\cmake\\xtensor\\xtensorConfigVersion.cmake (exit 1)\n\n# On unix this is rendered to:\ntests:\n - script:\n - test -d ${PREFIX}/include/xtensor\n - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake\n
"},{"location":"reference/recipe_file/#experimental-features","title":"Experimental features","text":"Warning
These are experimental features of rattler-build
and may change or go away completely.
"},{"location":"reference/recipe_file/#jinja-functions","title":"Jinja functions","text":" load_from_file
git.*
functions "},{"location":"tutorials/cpp/","title":"Packaging a C++ package","text":"This tutorial will guide you though making a C++ package with rattler-build
.
"},{"location":"tutorials/cpp/#building-a-header-only-library","title":"Building a Header-only Library","text":"To build a package for the header-only library xtensor
, you need to manage dependencies and ensure proper installation paths.
"},{"location":"tutorials/cpp/#key-steps","title":"Key Steps","text":" Dependencies: Ensure cmake
, ninja
, and a compiler
are available as dependencies.
CMake Installation Prefix: Use the CMAKE_INSTALL_PREFIX
setting to instruct CMake
to install the headers in the correct location.
Unix Systems: Follow the standard Unix prefix:
$PREFIX/include\n$PREFIX/lib\n
Windows Systems: Use a Unix-like prefix but nested in a Library
directory:
$PREFIX/Library/include\n$PREFIX/Library/lib\n
Utilize the handy variables %LIBRARY_PREFIX%
and %LIBRARY_BIN%
to guide CMake
to install the headers and libraries correctly. This approach ensures that the headers and libraries are installed in the correct directories on both Unix and Windows systems.
"},{"location":"tutorials/cpp/#recipe","title":"Recipe","text":"recipe.yamlcontext:\n version: \"0.24.6\"\n\npackage:\n name: xtensor\n version: ${{ version }}\n\nsource:\n url: https://github.com/xtensor-stack/xtensor/archive/${{ version }}.tar.gz\n sha256: f87259b51aabafdd1183947747edfff4cff75d55375334f2e81cee6dc68ef655\n\nbuild:\n number: 0\n script:\n - if: win # (1)!\n then: |\n cmake -GNinja ^\n -D BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% ^\n %SRC_DIR%\n ninja install\n else: |\n cmake -GNinja \\\n -DBUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$PREFIX \\\n $SRC_DIR\n ninja install\n\nrequirements:\n build:\n - ${{ compiler('cxx') }} # (2)!\n - cmake\n - ninja\n host:\n - xtl >=0.7,<0.8\n run:\n - xtl >=0.7,<0.8\n run_constraints: # (3)!\n - xsimd >=8.0.3,<10\n\ntests:\n - package_contents:\n include: # (4)!\n - xtensor/xarray.hpp\n files: # (5)!\n - ${{ \"Library\" if win }}/share/cmake/xtensor/xtensorConfig.cmake\n - ${{ \"Library\" if win }}/share/cmake/xtensor/xtensorConfigVersion.cmake\n\nabout:\n homepage: https://github.com/xtensor-stack/xtensor\n license: BSD-3-Clause\n license_file: LICENSE\n summary: The C++ tensor algebra library\n description: Multi dimensional arrays with broadcasting and lazy computing\n documentation: https://xtensor.readthedocs.io\n repository: https://github.com/xtensor-stack/xtensor\n\nextra:\n recipe-maintainers:\n - some-maintainer\n
The if:
condition allows the user to switch behavior of the build based on some checks like, the operating system. The compiler
function is used to get the C++ compiler for the build system. The run_constraints
section specifies the version range of a package which the package can run \"with\". But which the package doesn't depend on itself. The include
section specifies the header file to tested for existence. The files
section specifies the files to be tested for existence, using a glob pattern. CMAKE_ARGS
It can be tedious to remember all the different variables one needs to pass to CMake to create the perfect build. The cmake
package on conda-forge introduces theCMAKE_ARGS
environment variable. This variable contains the necessary flags to make the package build correctly, also when cross-compiling from one machine to another. Therefore, it is often not necessary to pass any additional flags to the cmake
command. However, because this is a tutorial we will show how to pass the necessary flags to cmake
manually.
For more information please refer to the conda-forge documentation.
"},{"location":"tutorials/cpp/#building-a-c-application","title":"Building A C++ application","text":"In this example, we'll build poppler
, a C++ application for manipulating PDF files from the command line. The final package will install several tools into the bin/
folder. We'll use external build scripts and run actual scripts in the test.
"},{"location":"tutorials/cpp/#key-steps_1","title":"Key Steps","text":" Dependencies:
Build Dependencies: These are necessary for the building process, including cmake
, ninja
, and pkg-config
. Host Dependencies: These are the libraries poppler
links against, such as cairo
, fontconfig
, freetype
, glib
, and others. Compiler Setup: We use the compiler
function to obtain the appropriate C and C++ compilers.
Build Script: The build.script
field points to an external script (poppler-build.sh
) which contains the build commands.
Testing: Simple tests are included to verify that the installed tools (pdfinfo
, pdfunite
, pdftocairo
) are working correctly by running them, and expecting an exit code 0
.
"},{"location":"tutorials/cpp/#recipe_1","title":"Recipe","text":"recipe.yamlcontext:\n version: \"24.01.0\"\n\npackage:\n name: poppler\n version: ${{ version }}\n\nsource:\n url: https://poppler.freedesktop.org/poppler-${{ version }}.tar.xz\n sha256: c7def693a7a492830f49d497a80cc6b9c85cb57b15e9be2d2d615153b79cae08\n\nbuild:\n script: poppler-build.sh\n\nrequirements:\n build:\n - ${{ compiler('c') }} # (1)!\n - ${{ compiler('cxx') }}\n - pkg-config\n - cmake\n - ninja\n host:\n - cairo # (2)!\n - fontconfig\n - freetype\n - glib\n - libboost-headers\n - libjpeg-turbo\n - lcms2\n - libiconv\n - libpng\n - libtiff\n - openjpeg\n - zlib\n\ntests:\n - script:\n - pdfinfo -listenc # (3)!\n - pdfunite --help\n - pdftocairo --help\n
The compiler
jinja function to get the correct compiler for C and C++ on the build system. These are all the dependencies that the library links against. The script test just executes some of the installed tools to check if they are working. These can be as complex as you want. (bash
or cmd.exe
) "},{"location":"tutorials/cpp/#external-build-script","title":"External Build Script","text":"We've defined an external build script in the recipe. This will be searched next to the recipe by the file name given, or the default name build.sh
on unix
or build.bat
on windows are searched for.
poppler-build.sh#! /bin/bash\n\nextra_cmake_args=(\n -GNinja\n -DCMAKE_INSTALL_LIBDIR=lib\n -DENABLE_UNSTABLE_API_ABI_HEADERS=ON\n -DENABLE_GPGME=OFF\n -DENABLE_LIBCURL=OFF\n -DENABLE_LIBOPENJPEG=openjpeg2\n -DENABLE_QT6=OFF\n -DENABLE_QT5=OFF\n -DENABLE_NSS3=OFF\n)\n\nmkdir build && cd build\n\ncmake ${CMAKE_ARGS} \"${extra_cmake_args[@]}\" \\\n -DCMAKE_PREFIX_PATH=$PREFIX \\\n -DCMAKE_INSTALL_PREFIX=$PREFIX \\\n -DTIFF_INCLUDE_DIR=$PREFIX/include \\\n $SRC_DIR\n\nninja\n\n# The `install` command will take care of copying the files to the right place\nninja install\n
"},{"location":"tutorials/cpp/#parsing-the-rattler-build-build-output","title":"Parsing the rattler-build build
Output","text":"When running the rattler-build
command, you might notice some interesting information in the output. Our package will have some run
dependencies, even if we didn't specify any.
These come from the run-exports
of the packages listed in the host
section of the recipe. This is indicated by \"RE of [host: package]\"
in the output.
For example, libcurl
specifies that if you depend on it in the host section, you should also depend on it during runtime with specific version ranges. This ensures proper linking to shared libraries.
Run dependencies:\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 Name \u2506 Spec \u2502\n\u255e\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2561\n\u2502 libcurl \u2506 >=8.5.0,<9.0a0 (RE of [host: libcurl]) \u2502\n\u2502 fontconfig \u2506 >=2.14.2,<3.0a0 (RE of [host: fontconfig]) \u2502\n\u2502 fonts-conda-ecosystem \u2506 (RE of [host: fontconfig]) \u2502\n\u2502 lcms2 \u2506 >=2.16,<3.0a0 (RE of [host: lcms2]) \u2502\n\u2502 gettext \u2506 >=0.21.1,<1.0a0 (RE of [host: gettext]) \u2502\n\u2502 freetype \u2506 >=2.12.1,<3.0a0 (RE of [host: freetype]) \u2502\n\u2502 openjpeg \u2506 >=2.5.0,<3.0a0 (RE of [host: openjpeg]) \u2502\n\u2502 libiconv \u2506 >=1.17,<2.0a0 (RE of [host: libiconv]) \u2502\n\u2502 cairo \u2506 >=1.18.0,<2.0a0 (RE of [host: cairo]) \u2502\n\u2502 libpng \u2506 >=1.6.42,<1.7.0a0 (RE of [host: libpng]) \u2502\n\u2502 libzlib \u2506 >=1.2.13,<1.3.0a0 (RE of [host: zlib]) \u2502\n\u2502 libtiff \u2506 >=4.6.0,<4.7.0a0 (RE of [host: libtiff]) \u2502\n\u2502 libjpeg-turbo \u2506 >=3.0.0,<4.0a0 (RE of [host: libjpeg-turbo]) \u2502\n\u2502 libglib \u2506 >=2.78.3,<3.0a0 (RE of [host: glib]) \u2502\n\u2502 libcxx \u2506 >=16 (RE of [build: clangxx_osx-arm64]) \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
You can also see \"linking\" information in the output, for example on macOS:
[lib/libpoppler-glib.8.26.0.dylib] links against:\n \u251c\u2500 @rpath/libgio-2.0.0.dylib\n \u251c\u2500 @rpath/libgobject-2.0.0.dylib\n \u251c\u2500 /usr/lib/libSystem.B.dylib\n \u251c\u2500 @rpath/libglib-2.0.0.dylib\n \u251c\u2500 @rpath/libpoppler.133.dylib\n \u251c\u2500 @rpath/libfreetype.6.dylib\n \u251c\u2500 @rpath/libc++.1.dylib\n \u251c\u2500 @rpath/libpoppler-glib.8.dylib\n \u2514\u2500 @rpath/libcairo.2.dylib\n
rattler-build
ensures that:
All shared libraries linked against are present in the run dependencies. Missing libraries trigger an overlinking
warning. You don't require any packages in the host that you are not linking against. This triggers an overdepending
warning. "},{"location":"tutorials/go/","title":"Packaging a Go package","text":"This tutorial will guide you through making a Go package with rattler-build
.
When building a recipe for Go, most Go dependencies are linked statically. That means, we should collect their licenses and add them in the package. The go-licenses
tool can help you with this task - as shown in the example below.
"},{"location":"tutorials/go/#the-different-go-compilers","title":"The different Go compilers","text":"The conda-forge
ecosystem provides two go compilers: go-cgo
and go-nocgo
.
By default, if you do not need to link against C libraries, it's recommended to use the go-nocgo
compiler. It generates fat binaries without libc dependencies. The compiler activation scripts will set your CC
, CXX
and related flags to invalid values.
The go-cgo
compiler can generate fat binaries that depend on conda-forge's libc. You should use this compiler if the underlying program needs to link against other C libraries, in which case make sure to add ${{ compiler('c') }}
(cxx
, fortran
, ...) for unix and the m2w64
equivalent for windows.
"},{"location":"tutorials/go/#example-go-recipe","title":"Example Go recipe","text":"This example shows how to package the Temporal CLI.
recipe.yamlcontext:\n version: \"0.13.1\"\n\npackage:\n name: temporal\n version: ${{ version }}\n\nsource:\n url: https://github.com/temporalio/cli/archive/refs/tags/v${{ version }}.tar.gz\n sha256: 9d8812c96d3404490659fec3915dcd23c4142b421ef4cb7e9622bd9a459e1f74\n\nbuild:\n number: 0\n\nrequirements:\n build:\n - ${{ compiler('go-nocgo') }}\n - go-licenses\n\ntests:\n - script:\n - temporal --version\n\nabout:\n homepage: https://temporal.io\n repository: https://github.com/temporalio/cli\n documentation: https://docs.temporal.io/cli\n summary: Temporal CLI\n description: |\n Command-line interface for running Temporal Server and interacting with\n Workflows, Activities, Namespaces, and other parts of Temporal.\n license: MIT\n license_file:\n - LICENSE\n # These license files are generated at build time in the `build.sh` script\n # from all the dependencies of `temporal.io`.\n - license-files/\n
The build script (on Unix) should look something like this:
build.sh# The LDFLAGS are used to set the version of the `temporal` binary. This is a common practice in Go.\nexport LDFLAGS=\"${LDFLAGS} -s -w -X github.com/temporalio/cli/temporalcli.Version=${PKG_VERSION}\"\n\n# Build the `temporal` binary and store it in the `$PREFIX/bin` directory.\ngo build -ldflags \"$LDFLAGS\" -o $PREFIX/bin/temporal ./cmd/temporal\n\n# Store the license files in a separate directory in the $SRC_DIR. These are embedded in the package\n# in the `license_file` section.\ngo-licenses save ./cmd/temporal --save_path=\"$SRC_DIR/license-files/\" || true\n
"},{"location":"tutorials/javascript/","title":"Packaging a Javascript (NPM/NodeJS) package","text":"This tutorial will guide you though making a NodeJS package with rattler-build
. Please note that, while packaging executable applications is possible, the conda ecosystem is not ideal for NPM libraries. NPM supports a number of features that cannot easily be modeled in the conda ecosystem, such as peer dependencies, optional dependencies, and the ability to install multiple versions of the same package.
However, if you need to package a NodeJS application, rattler-build
can help!
"},{"location":"tutorials/javascript/#building-a-nodejs-package","title":"Building a NodeJS Package","text":"In this example, we will build a package for the NodeJS package bibtex-tidy
. We use nodejs
in build and run requirements, and install the package using npm
. NPM comes as part of the NodeJS installation, so we do not need to install it separately.
recipe.yamlcontext:\n version: \"1.14.0\"\n\npackage:\n name: bibtex-tidy\n version: ${{ version }}\n\nsource:\n url: https://registry.npmjs.org/bibtex-tidy/-/bibtex-tidy-${{ version }}.tgz\n sha256: 0a2c1bb73911a7cee36a30ce1fc86feffe39b2d39acd4c94d02aac6f84a00285\n # we do not extract the source code and install the tarball directly as that works better\n file_name: bibtex-tidy-${{ version }}.tgz\n\nbuild:\n number: 0\n script:\n # we use NPM to globally install the bibtex-tidy package\n - npm install -g bibtex-tidy-${{ version }}.tgz --prefix ${{ PREFIX }}\n\nrequirements:\n build:\n - nodejs\n run:\n - nodejs\n\ntests:\n - script:\n - bibtex-tidy --version\n
"},{"location":"tutorials/perl/","title":"Packaging a Perl (CPAM) package","text":"Packaging a Perl package is similar to packaging a Python package!
"},{"location":"tutorials/perl/#building-a-perl-package","title":"Building a Perl Package","text":""},{"location":"tutorials/perl/#a-perl-noarch-generic-package","title":"A perl noarch: generic
package","text":"The following recipe is for the Perl package Call::Context
. We use perl
in the host
requirements, and install the package using make
. The noarch: generic
is used to indicate that the package is architecture-independent - since this is a pure Perl package, it can be installed and run on any platform (noarch
).
recipe.yamlcontext:\n version: 0.03\n\npackage:\n name: perl-call-context\n version: ${{ version }}\n\nsource:\n url: https://cpan.metacpan.org/authors/id/F/FE/FELIPE/Call-Context-${{ version }}.tar.gz\n sha256: 0ee6bf46bc72755adb7a6b08e79d12e207de5f7809707b3c353b58cb2f0b5a26\n\nbuild:\n number: 0\n noarch: generic\n script:\n - perl Makefile.PL INSTALLDIRS=vendor NO_PERLLOCAL=1 NO_PACKLIST=1\n - make\n - make test\n - make install\n\nrequirements:\n build:\n - make\n host:\n - perl\n\ntests:\n - perl:\n uses:\n - Call::Context\n\nabout:\n license: GPL-1.0-or-later OR Artistic-1.0-Perl\n summary: Sanity-check calling context\n homepage: http://metacpan.org/pod/Call-Context\n
"},{"location":"tutorials/perl/#a-perl-package-with-a-c-extension","title":"A perl package with a C extension","text":"Some perl
packages have native code extensions. In this example, we will build a package for the Perl package Data::Dumper
using the C
compiler. The c
compiler and make
are required at build time in the build
requirements to compile the native code extension. We use perl
in the host
requirements, and install the package using make
.
recipe.yamlcontext:\n version: \"2.183\"\n\npackage:\n name: \"perl-data-dumper\"\n version: ${{ version }}\n\nsource:\n url: https://cpan.metacpan.org/authors/id/N/NW/NWCLARK/Data-Dumper-${{ version }}.tar.gz\n sha256: e42736890b7dae1b37818d9c5efa1f1fdc52dec04f446a33a4819bf1d4ab5ad3\n\nbuild:\n number: 0\n script:\n - perl Makefile.PL INSTALLDIRS=vendor NO_PERLLOCAL=1 NO_PACKLIST=1\n - make\n - make test\n - make install VERBINST=1\n\nrequirements:\n build:\n - ${{ compiler('c') }}\n - make\n host:\n - perl\n - perl-extutils-makemaker\n\ntests:\n - perl:\n uses:\n - Data::Dumper\n\nabout:\n homepage: https://metacpan.org/pod/Data::Dumper\n license: GPL-1.0-or-later OR Artistic-1.0-Perl\n summary: 'seeds germane, yet not germinated'\n
"},{"location":"tutorials/python/","title":"Writing a Python package","text":"Writing a Python package is fairly straightforward, especially for \"Python-only\" packages. In the second example we will build a package for numpy
which contains compiled code.
"},{"location":"tutorials/python/#a-python-only-package","title":"A Python-only package","text":"The following recipe uses the noarch: python
setting to build a noarch
package that can be installed on any platform without modification. This is very handy for packages that are pure Python and do not contain any compiled extensions.
Additionally, noarch: python
packages work with a range of Python versions (contrary to packages with compiled extensions that are tied to a specific Python version).
recipe.yamlcontext:\n version: \"8.1.2\"\n\npackage:\n name: ipywidgets\n version: ${{ version }}\n\nsource:\n url: https://pypi.io/packages/source/i/ipywidgets/ipywidgets-${{ version }}.tar.gz\n sha256: d0b9b41e49bae926a866e613a39b0f0097745d2b9f1f3dd406641b4a57ec42c9\n\nbuild:\n noarch: python # (1)!\n script: pip install . -v\n\nrequirements:\n # note that there is no build section\n host:\n - pip\n - python >=3.7\n - setuptools\n - wheel\n run:\n - comm >=0.1.3\n - ipython >=6.1.0\n - jupyterlab_widgets >=3.0.10,<3.1.0\n - python >=3.7\n - traitlets >=4.3.1\n - widgetsnbextension >=4.0.10,<4.1.0\n\ntests:\n - python:\n imports:\n - ipywidgets # (2)!\n\nabout:\n homepage: https://github.com/ipython/ipywidgets\n license: BSD-3-Clause\n license_file: LICENSE\n summary: Jupyter Interactive Widgets\n description: |\n ipywidgets are interactive HTML widgets for Jupyter notebooks and the IPython kernel.\n documentation: https://ipywidgets.readthedocs.io/en/latest/\n
The noarch: python
line tells rattler-build
that this package is pure Python and can be one-size-fits-all. noarch
packages can be installed on any platform without modification which is very handy. The imports
section in the tests is used to check that the package is installed correctly and can be imported. "},{"location":"tutorials/python/#running-the-recipe","title":"Running the recipe","text":"To build this recipe, simply run:
rattler-build build --recipe ./ipywidgets\n
"},{"location":"tutorials/python/#a-python-package-with-compiled-extensions","title":"A Python package with compiled extensions","text":"We will build a package for numpy
\u2013 which contains compiled code. Since compiled code is python
version-specific, we will need to specify the python
version explicitly. The best way to do this is with a \"variant_config.yaml\" file:
variants.yamlpython:\n - 3.11\n - 3.12\n
This will replace any python
found in the recipe with the versions specified in the variants.yaml
file.
recipe.yamlcontext:\n version: 2.0.1\n default_abi_level: 1.21\n\npackage:\n name: numpy\n version: ${{ version }}\n\nsource:\n - url: https://github.com/numpy/numpy/releases/download/v${{ version }}/numpy-${{ version }}.tar.gz\n sha256: 485b87235796410c3519a699cfe1faab097e509e90ebb05dcd098db2ae87e7b3\n\nbuild:\n python:\n entry_points:\n - f2py = numpy.f2py.f2py2e:main # [win]\n - numpy-config = numpy._configtool:main\n\nrequirements:\n build:\n - ${{ compiler('c') }}\n - ${{ compiler('cxx') }}\n # note: some `host` dependencies that run at build time (e.g., `cython`, `meson-python`)\n # should ideally be in `build` instead, this is because cross compilation of\n # Python packages in conda-forge uses `crossenv` rather than regular cross compilation.\n host:\n # note: variant is injected here!\n - python\n - pip\n - meson-python\n - pkg-config\n - python-build\n - cython\n - libblas\n - libcblas\n - liblapack\n run:\n - python\n run_exports:\n - numpy >=${{ default_abi_level }},<3.0.0a0\n\ntests:\n - python:\n imports:\n - numpy\n - numpy.fft\n - numpy.linalg\n - numpy.random\n - numpy.ctypeslib\n\n - script:\n - f2py -v\n - numpy-config --cflags\n\nabout:\n homepage: http://numpy.org/\n license: BSD-3-Clause\n license_file: LICENSE.txt\n summary: The fundamental package for scientific computing with Python.\n documentation: https://numpy.org/doc/stable/\n repository: https://github.com/numpy/numpy\n
The build script for Unix:
build.shmkdir builddir\n\n$PYTHON -m build -w -n -x \\\n -Cbuilddir=builddir \\\n -Csetup-args=-Dblas=blas \\\n -Csetup-args=-Dlapack=lapack\n\n$PYTHON -m pip install dist/numpy*.whl\n
The build script for Windows:
build.batmkdir builddir\n\n%PYTHON% -m build -w -n -x ^\n -Cbuilddir=builddir ^\n -Csetup-args=-Dblas=blas ^\n -Csetup-args=-Dlapack=lapack\nif %ERRORLEVEL% neq 0 exit 1\n\n:: `pip install dist\\numpy*.whl` does not work on windows,\n:: so use a loop; there's only one wheel in dist/ anyway\nfor /f %%f in ('dir /b /S .\\dist') do (\n pip install %%f\n if %ERRORLEVEL% neq 0 exit 1\n)\n
"},{"location":"tutorials/python/#running-the-recipe_1","title":"Running the recipe","text":"Running this recipe with the variant config file will build a total of 2 numpy
packages:
rattler-build build --recipe ./numpy\n
At the beginning of the build process, rattler-build
will print the following message to show you the variants it found:
Found variants:\n\nnumpy-1.26.4-py311h5f8ada8_0\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 Variant \u2506 Version \u2502\n\u255e\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2561\n\u2502 python \u2506 3.11 \u2502\n\u2502 target_platform \u2506 osx-arm64 \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\nnumpy-1.26.4-py312h440f24a_0\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 Variant \u2506 Version \u2502\n\u255e\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2561\n\u2502 python \u2506 3.12 \u2502\n\u2502 target_platform \u2506 osx-arm64 \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n
"},{"location":"tutorials/rust/","title":"Building a Rust package","text":"We're using rattler-build
to build a Rust package for the cargo-edit
utility. This utility manages Cargo dependencies from the command line.
recipe.yamlcontext:\n version: \"0.11.9\"\n\npackage:\n name: cargo-edit\n version: ${{ version }}\n\nsource:\n url: https://github.com/killercup/cargo-edit/archive/refs/tags/v${{ version }}.tar.gz\n sha256: 46670295e2323fc2f826750cdcfb2692fbdbea87122fe530a07c50c8dba1d3d7\n\nbuild:\n script:\n - cargo-bundle-licenses --format yaml --output ${SRC_DIR}/THIRDPARTY.yml # !(1)\n - $BUILD_PREFIX/bin/cargo install --locked --bins --root ${PREFIX} --path .\n\nrequirements:\n build:\n - ${{ compiler('rust') }}\n - cargo-bundle-licenses\n\ntests:\n - script:\n - cargo-upgrade --help # !(2)\n\nabout:\n homepage: https://github.com/killercup/cargo-edit\n license: MIT\n license_file:\n - LICENSE\n - THIRDPARTY.yml\n description: \"A utility for managing cargo dependencies from the command line.\"\n summary: \"A utility for managing cargo dependencies from the command line.\"\n
Note
The ${{ compiler(...) }}
functions are very useful in the context of cross-compilation. When the function is evaluated it will insert the correct compiler (as selected with the variant config) as well the target_platform
. The \"rendered\" compiler will look like rust_linux-64
when you are targeting the linux-64
platform.
You can read more about this in the cross-compilation section.
The cargo-bundle-licenses
utility is used to bundle all the licenses of the dependencies into a THIRDPARTY.yml
file. This file is then included in the package. You should always include this file in your package when you are redistributing it. Running scripts in bash
or cmd.exe
to test the package build well, expects an exit code of 0
to pass the test. To build this recipe, simply run:
rattler-build build \\\n --recipe ./cargo-edit/recipe.yaml\n
"}]}
\ No newline at end of file
diff --git a/dev/selectors/index.html b/dev/selectors/index.html
index 125273a9..bf694ccf 100644
--- a/dev/selectors/index.html
+++ b/dev/selectors/index.html
@@ -477,6 +477,27 @@
+
+
+
+
+
+ Perl
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/dev/sitemap.xml b/dev/sitemap.xml
index 8b4a3a63..1629741d 100644
--- a/dev/sitemap.xml
+++ b/dev/sitemap.xml
@@ -2,142 +2,147 @@
https://prefix-dev.github.io/rattler-build/dev/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/authentication_and_upload/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/automatic_linting/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/build_options/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/build_script/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/compilers/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/converting_from_conda_build/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/experimental_features/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/highlevel/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/internals/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/multiple_output_cache/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/package_spec/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/rebuild/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/recipe_generation/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/selectors/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/special_files/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/testing/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/tips_and_tricks/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/tui/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/variants/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/reference/cli/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/reference/jinja/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/reference/recipe_file/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/tutorials/cpp/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/tutorials/go/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/tutorials/javascript/
- 2024-11-27
+ 2024-12-04
+ daily
+
+
+ https://prefix-dev.github.io/rattler-build/dev/tutorials/perl/
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/tutorials/python/
- 2024-11-27
+ 2024-12-04
daily
https://prefix-dev.github.io/rattler-build/dev/tutorials/rust/
- 2024-11-27
+ 2024-12-04
daily
\ No newline at end of file
diff --git a/dev/sitemap.xml.gz b/dev/sitemap.xml.gz
index eaa9dde0..c6e3981f 100644
Binary files a/dev/sitemap.xml.gz and b/dev/sitemap.xml.gz differ
diff --git a/dev/special_files/index.html b/dev/special_files/index.html
index 6087caaa..d2a40a4d 100644
--- a/dev/special_files/index.html
+++ b/dev/special_files/index.html
@@ -477,6 +477,27 @@
+
+
+
+
+
+ Perl
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/dev/testing/index.html b/dev/testing/index.html
index cdeebe33..d3f1a697 100644
--- a/dev/testing/index.html
+++ b/dev/testing/index.html
@@ -477,6 +477,27 @@
+
+
+
+
+
+ Perl
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/dev/tips_and_tricks/index.html b/dev/tips_and_tricks/index.html
index 5bad4d97..69f2c4aa 100644
--- a/dev/tips_and_tricks/index.html
+++ b/dev/tips_and_tricks/index.html
@@ -477,6 +477,27 @@
+
+
+
+
+
+ Perl
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/dev/tui/index.html b/dev/tui/index.html
index 9a2d4d0d..e49ef590 100644
--- a/dev/tui/index.html
+++ b/dev/tui/index.html
@@ -477,6 +477,27 @@
+
+
+
+
+
+ Perl
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/dev/tutorials/cpp/index.html b/dev/tutorials/cpp/index.html
index 0ae9ae4e..fb5f0dd6 100644
--- a/dev/tutorials/cpp/index.html
+++ b/dev/tutorials/cpp/index.html
@@ -603,6 +603,27 @@
+
+
+
+
+
+ Perl
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/dev/tutorials/go/index.html b/dev/tutorials/go/index.html
index 59773ab0..241388a8 100644
--- a/dev/tutorials/go/index.html
+++ b/dev/tutorials/go/index.html
@@ -14,7 +14,7 @@
-
+
@@ -537,6 +537,27 @@
+
+
+
+
+
+ Perl
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
@@ -1442,16 +1463,16 @@ Example Go recipe