Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to using poetry #374

Merged
merged 16 commits into from
May 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions .github/workflows/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,22 +8,28 @@ jobs:
check-code-style:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- uses: actions/checkout@v4

- name: Setup Poetry
uses: matrix-org/setup-python-poetry@v1
with:
install-project: "false"
python-version: "3.11"
- run: python -m pip install tox
- run: tox -e check_codestyle

- run: poetry run tox -e check_codestyle

check-types-mypy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- uses: actions/checkout@v4

- name: Setup Poetry
uses: matrix-org/setup-python-poetry@v1
with:
install-project: "false"
python-version: "3.11"
- run: python -m pip install tox
- run: tox -e check_types

- run: poetry run tox -e check_types

run-unit-tests:
name: Unit tests
Expand Down
67 changes: 42 additions & 25 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,35 +4,33 @@ Everyone is welcome to contribute code to Sygnal, provided you are willing to
license your contributions under the same license as the project itself. In
this case, the [Apache Software License v2](LICENSE).

### Create a virtualenv
### Installing dependencies

To contribute to Sygnal, ensure you have Python 3.8 or newer and then run:

```bash
python3 -m venv venv
./venv/bin/pip install -e '.[dev]'
```
Sygnal uses the [poetry](https://python-poetry.org/) project to manage its dependencies
and development environment. Once you have installed Python 3 and added the
source, you should install `poetry`.
Of their installation methods, we recommend
[installing `poetry` using `pipx`](https://python-poetry.org/docs/#installing-with-pipx),

This creates an isolated virtual Python environment ("virtualenv") just for
use with Sygnal, then installs Sygnal along with its dependencies, and lastly
installs a handful of useful tools
```shell
pip install --user pipx
pipx install poetry
```

If you get `ConnectTimeoutError`, this is caused by slow internet whereby
`pip` has a default time out of _15 sec_. You can specify a larger timeout
by passing `--timeout 120` to the `pip install` command above.
but see poetry's [installation instructions](https://python-poetry.org/docs/#installation)
for other installation methods.

Finally, activate the virtualenv by running:
Next, open a terminal and install dependencies as follows:

```bash
source ./venv/bin/activate
```sh
cd path/where/you/have/cloned/the/repository
poetry install
```

Be sure to do this _every time_ you open a new terminal window for working on
Sygnal. Activating the venv ensures that any Python commands you run (`pip`,
`python`, etc.) use the versions inside your venv, and not your system Python.

When you're done, you can close your terminal or run `deactivate` to disable
the virtualenv.
This will install the runtime and developer dependencies for the project. Be sure to check
that the `poetry install` step completed cleanly.

### Run the tests

Expand All @@ -56,6 +54,25 @@ ___________________________________ summary ___________________________________

Then all is well and you're ready to work!

You can also directly run the tests using poetry.

```sh
poetry run trial tests
```

You can run unit tests in parallel by specifying `-jX` argument to `trial` where `X` is the number of parallel runners you want. To use 4 cpu cores, you would run them like:

```sh
poetry run trial -j4 tests
```

If you wish to only run *some* unit tests, you may specify
another module instead of `tests` - or a test class or a method:

```sh
poetry run trial tests.test_apns.ApnsTestCase.test_expected
```

## How to contribute

The preferred and easiest way to contribute changes is to fork the relevant
Expand Down Expand Up @@ -90,13 +107,9 @@ Many of the conventions are enforced by scripts which are run as part of the
[continuous integration system](#continuous-integration-and-testing).

To help check and fix adherence to the code style, you can run `tox`
locally. You'll need Python 3.8 or later, and a virtual environment configured and
active:
locally. You'll need Python 3.8 or later:

```bash
# Activate the virtual environment
source ./venv/bin/activate

# Run the code style check
tox -e check_codestyle

Expand All @@ -114,6 +127,10 @@ Please ensure your changes match the cosmetic style of the existing project,
and **never** mix cosmetic and functional changes in the same commit, as it
makes it horribly hard to review otherwise.

## Further information on poetry

See the information provided in the [Synapse docs](https://github.com/element-hq/synapse/blob/master/docs/development/dependencies.md).

## Changelog

All changes, even minor ones, need a corresponding changelog / newsfragment
Expand Down
17 changes: 9 additions & 8 deletions RELEASING.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,36 @@
0. Consider whether this release will affect any customers, including those on
EMS, and warn them beforehand - in case they need to upgrade quickly.

1. Set a variable to the version number for convenience:
1. Update the version number in pyproject.toml.
2. Set a variable to the version number for convenience:
```sh
ver=x.y.z
```
1. Update the changelog:
3. Update the changelog:
```sh
towncrier --version=$ver
```
1. Push your changes:
4. Push your changes:
```sh
git add -u && git commit -m $ver && git push
```
1. Sanity-check the
5. Sanity-check the
[changelog](https://github.com/matrix-org/sygnal/blob/master/CHANGELOG.md)
and update if need be.
1. Create a signed tag for the relese:
6. Create a signed tag for the relese:
```sh
git tag -s v$ver
```
Base the tag message on the changelog.
1. Push the tag:
7. Push the tag:
```sh
git push origin tag v$ver
```
Pushing a tag on GitHub will automatically trigger a build in Docker Hub and
the resulting image will be published using the same tag as git.
1. Create release on GH project page:
8. Create release on GH project page:
```sh
xdg-open https://github.com/matrix-org/sygnal/releases/edit/v$ver
```
1. Notify #sygnal:matrix.org, #synapse-dev:matrix.org and EMS that a new
9. Notify #sygnal:matrix.org, #synapse-dev:matrix.org and EMS that a new
release has been published.
1 change: 1 addition & 0 deletions changelog.d/374.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Switch over to use poetry & add lock file to version control.
75 changes: 65 additions & 10 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,26 +6,81 @@
# docker build -f docker/Dockerfile .
#

ARG PYTHON_VERSION=3.11

###
### Stage 0: generate requirements.txt
###
# We hardcode the use of Debian bookworm here because this could change upstream.
FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm as requirements

# We install poetry in its own build stage to avoid its dependencies conflicting with
# sygnal's dependencies.
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --user "poetry==1.8.3"

WORKDIR /sygnal

# Copy just what we need to run `poetry export`...
COPY pyproject.toml poetry.lock /sygnal/

# If specified, we won't verify the hashes of dependencies.
# This is only needed if the hashes of dependencies cannot be checked for some
# reason, such as when a git repository is used directly as a dependency.
ARG TEST_ONLY_SKIP_DEP_HASH_VERIFICATION

# If specified, we won't use the Poetry lockfile.
# Instead, we'll just install what a regular `pip install` would from PyPI.
ARG TEST_ONLY_IGNORE_POETRY_LOCKFILE

# Export the dependencies, but only if we're actually going to use the Poetry lockfile.
# Otherwise, just create an empty requirements file so that the Dockerfile can
# proceed.
RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
/root/.local/bin/poetry export -o /sygnal/requirements.txt ${TEST_ONLY_SKIP_DEP_HASH_VERIFICATION:+--without-hashes}; \
else \
touch /sygnal/requirements.txt; \
fi

###
### Stage 0: builder
### Stage 1: builder
###
FROM python:3.11-slim as builder
FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm as builder

# Install git; Sygnal uses it to obtain the package version from the state of the
# git repository.
RUN apt-get update && apt-get install -y git
# To speed up rebuilds, install all of the dependencies before we copy over
# the whole sygnal project, so that this layer in the Docker cache can be
# used while you develop on the source.
#
# This is aiming at installing the `[tool.poetry.depdendencies]` from pyproject.toml.
COPY --from=requirements /sygnal/requirements.txt /sygnal/
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --prefix="/install" --no-deps --no-warn-script-location -r /sygnal/requirements.txt

# install sygnal and all of the python deps to /install.
# Copy over the rest of the sygnal source code.
COPY sygnal /sygnal/sygnal/
# ... and what we need to `pip install`.
COPY pyproject.toml README.md /sygnal/

COPY . /sygnal/
# Repeat of earlier build argument declaration, as this is a new build stage.
ARG TEST_ONLY_IGNORE_POETRY_LOCKFILE

RUN pip install --prefix="/install" --no-warn-script-location /sygnal
# Install the sygnal package itself.
# If we have populated requirements.txt, we don't install any dependencies
# as we should already have those from the previous `pip install` step.
RUN --mount=type=cache,target=/sygnal/target,sharing=locked \
--mount=type=cache,target=${CARGO_HOME}/registry,sharing=locked \
if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
pip install --prefix="/install" --no-deps --no-warn-script-location /sygnal; \
else \
pip install --prefix="/install" --no-warn-script-location /sygnal; \
fi

###
### Stage 1: runtime
### Stage 2: runtime
###

FROM python:3.11-slim
FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm

COPY --from=builder /install /usr/local

EXPOSE 5000/tcp
Expand Down
Loading
Loading