Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi env proposal documentation #584

Merged
merged 19 commits into from
Jan 2, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f78f562
docs: add multi env proposal
ruben-arts Dec 20, 2023
28718ea
Update docs/design_proposals/multi_environment_proposal.md
ruben-arts Dec 20, 2023
da9aef9
Update docs/design_proposals/multi_environment_proposal.md
ruben-arts Dec 20, 2023
2f50198
add links
ruben-arts Dec 20, 2023
781e05a
clearify the channels in features
ruben-arts Dec 20, 2023
7ac8a58
Update docs/design_proposals/multi_environment_proposal.md
ruben-arts Dec 20, 2023
2d46fd3
Add real world example polarify, folded on website
ruben-arts Dec 20, 2023
3b5360b
ci: avoid ci being run on docs
ruben-arts Dec 20, 2023
cbe8567
Update docs/design_proposals/multi_environment_proposal.md
ruben-arts Dec 20, 2023
c8b6b09
Update docs/design_proposals/multi_environment_proposal.md
ruben-arts Dec 20, 2023
dd9526b
Update docs/design_proposals/multi_environment_proposal.md
ruben-arts Dec 20, 2023
aea25ad
Update docs/design_proposals/multi_environment_proposal.md
ruben-arts Dec 20, 2023
42b5538
add prod vs test example
ruben-arts Dec 21, 2023
d7b059b
remove `default-features` field from the environments
ruben-arts Dec 21, 2023
4c80789
replace `environments` with `solve-group`
ruben-arts Dec 24, 2023
764c9a5
Update docs/design_proposals/multi_environment_proposal.md
ruben-arts Dec 26, 2023
414a5c8
add system-requirements to cuda example
ruben-arts Jan 2, 2024
5ea99c3
Merge branch 'main' into docs/add_multi_env_proposal
ruben-arts Jan 2, 2024
cee41f6
add `priority` to the channels to control the concatenation
ruben-arts Jan 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,14 @@ on:
- main
paths-ignore:
- "docs/**"
- "mkdocs.yml"
- "*.md"
workflow_dispatch:
pull_request:
paths-ignore:
- "docs/**"
- "mkdocs.yml"
- "*.md"

name: Rust

Expand Down
344 changes: 344 additions & 0 deletions docs/design_proposals/multi_environment_proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,344 @@
# Proposal Design: Multi Environment Support
## Objective
The aim is to introduce an environment set mechanism in the `pixi` package manager.
This mechanism will enable clear, conflict-free management of dependencies tailored to specific environments, while also maintaining the integrity of fixed lockfiles.


### Motivating Example
There are multiple scenarios where multiple environments are useful.

- **Testing of multiple package versions**, e.g. `py39` and `py310` or polars `0.12` and `0.13`.
- **Smaller single tool environments**, e.g. `lint` or `docs`.
- **Large developer environments**, that combine all the smaller environments, e.g. `dev`.
- **Strict supersets of environments**, e.g. `prod` and `test-prod` where `test-prod` is a strict superset of `prod`.
- **Multiple machines from one project**, e.g. a `cuda` environment and a `cpu` environment.
- **And many more.** (Feel free to edit this document in our GitHub and add your use case.)

This prepares `pixi` for use in large projects with multiple use-cases, multiple developers and different CI needs.

## Design Considerations
1. **User-friendliness**: Pixi is a user focussed tool that goes beyond developers. The feature should have good error reporting and helpful documentation from the start.
2. **Keep it simple**: Not understanding the multiple environments feature shouldn't limit a user to use pixi. The feature should be "invisible" to the non-multi env use-cases.
3. **No Automatic Combinatorial**: To ensure the dependency resolution process remains manageable, the solution should avoid a combinatorial explosion of dependency sets. By making the environments user defined and not automatically inferred by testing a matrix of the features.
4. **Single environment Activation**: The design should allow only one environment to be active at any given time, simplifying the resolution process and preventing conflicts.
5. **Fixed Lockfiles**: It's crucial to preserve fixed lockfiles for consistency and predictability. Solutions must ensure reliability not just for authors but also for end-users, particularly at the time of lockfile creation.

## Proposed Solution
!!! important
This is a proposal, not a final design. The proposal is open for discussion and will be updated based on the feedback.

### Feature & Environment Set Definitions
Introduce environment sets into the `pixi.toml` this describes environments based on `feature`'s. Introduce features into the `pixi.toml` that can describe parts of environments.
As an environment goes beyond just `dependencies` the `features` should be described including the following fields:

- `dependencies`: The conda package dependencies
ruben-arts marked this conversation as resolved.
Show resolved Hide resolved
- `pypi-dependencies`: The pypi package dependencies
- `system-requirements`: The system requirements of the environment
- `activation`: The activation information for the environment
- `platforms`: The platforms the environment can be run on.
- `channels`: The channels used to create the environment.
- `target`: All the above features but also separated by targets.
- `tasks`: Feature specific tasks, tasks in one environment are selected as default tasks for the environment.


```toml title="Default features" linenums="1"
[dependencies] # short for [feature.default.dependencies]
python = "*"
numpy = "==2.3"

[pypi-dependencies] # short for [feature.default.pypi-dependencies]
pandas = "*"

[system-requirements] # short for [feature.default.system-requirements]
libc = "2.33"

[activation] # short for [feature.default.activation]
scripts = ["activate.sh"]
```

```toml title="Different dependencies per feature" linenums="1"
[feature.py39.dependencies]
python = "~=3.9.0"
[feature.py310.dependencies]
python = "~=3.10.0"
[feature.test.dependencies]
pytest = "*"
```

```toml title="Full set of environment modification in one feature" linenums="1"
[feature.cuda]
dependencies = {cuda = "x.y.z", cudnn = "12.0"}
pypi-dependencies = {torch = "1.9.0"}
platforms = ["linux-64", "osx-arm64"]
activation = {scripts = ["cuda_activation.sh"]}
channels = ["nvidia"] # Would concat instead of overwrite, so the default channels are still used.
baszalmstra marked this conversation as resolved.
Show resolved Hide resolved
tasks = { warmup = "python warmup.py" }
target.osx-arm64 = {dependencies = {mlx = "x.y.z"}}
```


```toml title="Define tasks as defaults of an environment" linenums="1"
[feature.test.tasks]
test = "pytest"

[environments]
test = ["test"]

# `pixi run test` == `pixi run --environments test test`
```

The environment definition should contain the following fields:

- `features: Vec<Feature>`: The features that are included in the environment set, which is also the default field in the environments.
- `solve-group: String`: The solve group is used to group environments together at the solve stage.
This is useful for environments that need to have the same dependencies but might extend them with additional dependencies.
For instance when testing a production environment with additional test dependencies.

```toml title="Creating environments from features" linenums="1"
[environments]
# implicit: default = ["default"]
default = ["py39"] # implicit: default = ["py39", "default"]
baszalmstra marked this conversation as resolved.
Show resolved Hide resolved
py310 = ["py310"] # implicit: py310 = ["py310", "default"]
test = ["test"] # implicit: test = ["test", "default"]
test39 = ["test", "py39"] # implicit: test39 = ["test", "py39", "default"]
```

```toml title="Testing a production environment with additional dependencies" linenums="1"
[environments]
# Creating a `prod` environment which is the minimal set of dependencies used for production.
prod = {features = ["py39"], solve-group = "prod"}
# Creating a `test_prod` environment which is the `prod` environment plus the `test` feature.
test_prod = {features = ["py39", "test"], solve-group = "prod"}
# Using the `solve-group` to solve the `prod` and `test_prod` environments together
# Which makes sure the tested environment has the same version of the dependencies as the production environment.
```

```toml title="Creating environments without a default environment" linenums="1"
[dependencies]
# Keep empty or undefined to create an empty environment.

[feature.base.dependencies]
python = "*"

[feature.lint.dependencies]
pre-commit = "*"

[environments]
# Create a custom default
default = ["base"]
# Create a custom environment which only has the `lint` feature as the default feature is empty.
lint = ["lint"]
```

### Lockfile Structure
Within the `pixi.lock` file, a package may now include an additional `environments` field, specifying the environment to which it belongs.
To avoid duplication the packages `environments` field may contain multiple environments so the lockfile is of minimal size.
```yaml
- platform: linux-64
name: pre-commit
version: 3.3.3
category: main
environments:
- dev
- test
- lint
...:
- platform: linux-64
name: python
version: 3.9.3
category: main
environments:
- dev
- test
- lint
- py39
- default
...:
```


### User Interface Environment Activation
Users can manually activate the desired environment via command line or configuration.
This approach guarantees a conflict-free environment by allowing only one feature set to be active at a time.
For the user the cli would look like this:

```shell title="Default behavior"
pixi run python
# Runs python in the `default` environment
```

```shell title="Activating an specific environment"
pixi run -e test pytest
pixi run --environment test pytest
# Runs `pytest` in the `test` environment
```

```shell title="Activating a shell in an environment"
pixi shell -e cuda
pixi shell --environment cuda
# Starts a shell in the `cuda` environment
```
```shell title="Running any command in an environment"
pixi run -e test any_command
# Runs any_command in the `test` environment which doesn't require to be predefined as a task.
```

```shell title="Interactive selection of environments if task is in multiple environments"
# In the scenario where test is a task in multiple environments, interactive selection should be used.
pixi run test
# Which env?
# 1. test
# 2. test39
```

## Important links
- Initial writeup of the proposal: https://gist.github.com/0xbe7a/bbf8a323409be466fe1ad77aa6dd5428
- GitHub project: https://github.com/orgs/prefix-dev/projects/10

## Real world example use cases
??? tip "Polarify test setup"
In `polarify` they want to test multiple versions combined with multiple versions of polars.
This is currently done by using a matrix in GitHub actions.
This can be replaced by using multiple environments.

```toml title="pixi.toml"
[project]
name = "polarify"
# ...
channels = ["conda-forge"]
platforms = ["linux-64", "osx-arm64", "osx-64", "win-64"]

[tasks]
postinstall = "pip install --no-build-isolation --no-deps --disable-pip-version-check -e ."

[dependencies]
python = ">=3.9"
pip = "*"
polars = ">=0.14.24,<0.21"

[feature.py39.dependencies]
python = "3.9.*"
[feature.py310.dependencies]
python = "3.10.*"
[feature.py311.dependencies]
python = "3.11.*"
[feature.py312.dependencies]
python = "3.12.*"
[feature.pl017.dependencies]
polars = "0.17.*"
[feature.pl018.dependencies]
polars = "0.18.*"
[feature.pl019.dependencies]
polars = "0.19.*"
[feature.pl020.dependencies]
polars = "0.20.*"

[feature.test.dependencies]
pytest = "*"
pytest-md = "*"
pytest-emoji = "*"
hypothesis = "*"
[feature.test.tasks]
test = "pytest"

[feature.lint.dependencies]
pre-commit = "*"
[feature.lint.tasks]
lint = "pre-commit run --all"

[environments]
pl017 = ["pl017", "py39", "test"]
pl018 = ["pl018", "py39", "test"]
pl019 = ["pl019", "py39", "test"]
pl020 = ["pl020", "py39", "test"]
py39 = ["py39", "test"]
py310 = ["py310", "test"]
py311 = ["py311", "test"]
py312 = ["py312", "test"]
```

```yaml title=".github/workflows/test.yml"
jobs:
tests:
name: Test ${{ matrix.environment }}
runs-on: ubuntu-latest
strategy:
matrix:
environment:
- pl017
- pl018
- pl019
- pl020
- py39
- py310
- py311
- py312
steps:
- uses: actions/checkout@v4
- uses: prefix-dev/[email protected]
with:
# already installs the corresponding environment and caches it
environments: ${{ matrix.environment }}
- name: Install dependencies
run: |
pixi run --env ${{ matrix.environment }} postinstall
pixi run --env ${{ matrix.environment }} test
```
??? tip "Test vs Production example"
This is an example of a project that has a `test` feature and `prod` environment.
The `prod` environment is a production environment that contains the run dependencies.
The `test` feature is a set of dependencies and tasks that we want to put on top of the previously solved `prod` environment.
ruben-arts marked this conversation as resolved.
Show resolved Hide resolved
This is a common use case where we want to test the production environment with additional dependencies.

```toml title="pixi.toml"
[project]
name = "my-app"
# ...
channels = ["conda-forge"]
platforms = ["osx-arm64", "linux-64"]

[tasks]
postinstall-e = "pip install --no-build-isolation --no-deps --disable-pip-version-check -e ."
postinstall = "pip install --no-build-isolation --no-deps --disable-pip-version-check ."
dev = "uvicorn my_app.app:main --reload"
serve = "uvicorn my_app.app:main"

[dependencies]
python = ">=3.12"
pip = "*"
pydantic = ">=2"
fastapi = ">=0.105.0"
sqlalchemy = ">=2,<3"
uvicorn = "*"
aiofiles = "*"

[feature.test.dependencies]
pytest = "*"
pytest-md = "*"
pytest-asyncio = "*"
[feature.test.tasks]
test = "pytest --md=report.md"

[environments]
# both default and prod will have exactly the same dependency versions when they share a dependency
default = {features = ["test"], solve-group = "prod-group"}
prod = {features = [], solve-group = "prod-group"}
```
In ci you would run the following commands:
```shell
pixi run postinstall-e && pixi run test
```
Locally you would run the following command:
```shell
pixi run postinstall-e && pixi run dev
```

Then in a Dockerfile you would run the following command:
```dockerfile title="Dockerfile"
FROM ghcr.io/prefix-dev/pixi:latest # this doesn't exist yet
WORKDIR /app
COPY . .
RUN pixi run --env prod postinstall
EXPOSE 8080
CMD ["/usr/local/bin/pixi", "run", "--env", "prod", "serve"]
```
2 changes: 2 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ nav:
- C++/Cmake: examples/cpp-sdl.md
- OpenCV: examples/opencv.md
- ROS2: examples/ros2-nav2.md
- Design Proposals:
- Multi Env: design_proposals/multi_environment_proposal.md
- Community: Community.md
- FAQ: FAQ.md

Expand Down