Skip to content

Commit

Permalink
feat: New GraphQL exporter in sync with master branch
Browse files Browse the repository at this point in the history
Signed-off-by: Daniel Alvarez-Coello <[email protected]>
  • Loading branch information
jdacoello committed Nov 25, 2024
2 parents ab93531 + a058c36 commit dcde474
Show file tree
Hide file tree
Showing 33 changed files with 1,570 additions and 1,262 deletions.
29 changes: 7 additions & 22 deletions .github/workflows/buildcheck.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,25 +27,12 @@ jobs:
uses: actions/checkout@v4
- name: Install apt dependencies
run: sudo apt install -y protobuf-compiler
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
cache: pip
- name: Install poetry
uses: abatilo/actions-poetry@v2
with:
poetry-version: "1.8.3"
- name: Poetry install
run: poetry install
- name: Install uv
uses: astral-sh/setup-uv@v3
- name: Mypy
run: poetry run mypy src
- name: flake8
run: |
poetry run flake8 src tests contrib
run: uv run mypy src
- name: Run tests
run: |
poetry run pytest --cov=vss_tools --cov-report=term-missing --cov-fail-under=88
run: uv run nox
- name: Test Binary Go Parser
run: |
cd binary/go_parser
Expand All @@ -62,13 +49,11 @@ jobs:
with:
python-version: ${{ env.PYTHON_VERSION }}
cache: pip
- name: Install poetry
uses: abatilo/actions-poetry@v2
with:
poetry-version: "1.8.3"
- name: Install uv
uses: astral-sh/setup-uv@v3
- name: Test Pypi packaging
run: |
poetry build
uv build
- name: Test that tools can be started
run: |
pip install dist/*.whl
Expand Down
94 changes: 50 additions & 44 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,13 @@ source .venv/bin/activate
```

pypi

```bash
pip install vss-tools
```

from source:

```bash
# default branch
pip install git+https://github.com/COVESA/vss-tools.git
Expand All @@ -37,7 +39,7 @@ See [usage](#usage) for how to start using it.

## Usage

General CLI help should be used for up to date info of how to use the tools.
General CLI help should be used for up-to-date info of how to use the tools.

```bash
# Help for toplevel options and lists sub commands
Expand All @@ -50,10 +52,9 @@ vspec export --help
vspec export json --help
```

Please check [here](./docs/vspec.md) for generic infos about exporters and their arguments
Please check [here](./docs/vspec.md) for generic info about exporters and their arguments
as well as [here](./docs/vspec_arch.md) for design decision, architecture and limitations.


## Compatibility with VSS

The [COVESA VSS project repository](https://github.com/COVESA/vehicle_signal_specification) includes vss-tools as a submodule.
Expand All @@ -62,90 +63,95 @@ It is not guaranteed that newer or older versions of vss-tools can successfully
The table below gives an overview of basic version support for e.g. `vspec export json`.
Other exporters may have stricter requirements.

VSS-tools version | Supported VSS catalog versions | Comments
-----------------|------------------------|----------------
`v3.0`| `v3.0` - `v3.1.1`
`v3.1`| `v3.0` -`v4.0`
`v4.0`| `v4.0`
`v4.1`| `v4.0` -
`4.2`| `v4.0` -
`5.0`| `v5.0` -
`<latest source>`| `v5.0` -
| VSS-tools version | Supported VSS catalog versions | Comments |
| ----------------- | ------------------------------ | -------- |
| `v3.0` | `v3.0` - `v3.1.1` |
| `v3.1` | `v3.0` -`v4.0` |
| `v4.0` | `v4.0` |
| `v4.1` | `v4.0` - |
| `4.2` | `v4.0` - |
| `5.0` | `v5.0` - |
| `<latest source>` | `v5.0` - |

### Changes affecting compatibility

Examples on changes affecting compatibility

* Somewhat stricter control in VSS-tools 5.0 onwards,
- Somewhat stricter control in VSS-tools 5.0 onwards,
minor changes required in VSS 4.X catalogs to make it pass the control.
* VSS version `v4.1` introduced a new syntax for the unit files that cannot be handled by `vss-tools < v4.1`
* From `v4.0` vss-tools expects unit file to be explicitly specified or provided in the same directory as the VSS input.
- VSS version `v4.1` introduced a new syntax for the unit files that cannot be handled by `vss-tools < v4.1`
- From `v4.0` vss-tools expects unit file to be explicitly specified or provided in the same directory as the VSS input.
VSS `v3.1` is the first VSS version including a unit file in the VSS repository.
This means vss-tools from `v4.0` onwards cannot handle VSS-versions prior to VSS `v3.1`
* VSS-tools `v3.1` only supported `default` for attributes, resulting in that newer VSS-versions is not supported.
* VSS-tools `v4.0` requires case-sensitive for type, resulting in that VSS versions `v3.1` and earlier is not supported.
- VSS-tools `v3.1` only supported `default` for attributes, resulting in that newer VSS-versions is not supported.
- VSS-tools `v4.0` requires case-sensitive for type, resulting in that VSS versions `v3.1` and earlier is not supported.

## Contributing

### Poetry
### UV

We are using [poetry](https://python-poetry.org/docs/) or a package and venv handling system.
Therefore a requirement to develop a tool is to install poetry on your host machine.
Check [here](https://python-poetry.org/docs/#installation) for official installation instructions. The recommended one is the following, however installing it through pip/pipx works aswell:
We are using [uv](https://docs.astral.sh/uv/) as a python package and project manager.
Therefore, a requirement to develop is to install `uv` on your host machine.
Check [here](https://docs.astral.sh/uv/#getting-started) for official installation instructions. The recommended one is the following, however installing it through pip/pipx works as well:

```bash
curl -sSL https://install.python-poetry.org | python3 -
# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
```

You can then use the following command in the root directory of `vss-tools` to install all dependencies:

```bash
poetry install
uv sync
```

Due to our [poetry.toml](./poetry.toml) the call will create a `.venv` directory but does not automatically activate it.

You can use that venv in several ways, with or without the help of `poetry`:
- `poetry shell`: Spawns a new subshell with the appropriate environment activated. After that you can directly call tools
- `poetry run <tool>`: Runs the given tool in the virtual environment but does return to the current environment after that
- `. $(poetry env info --path)/bin/activate`: Enables the virtual environment. Since we do store virtual envs directly with the tool the command can be simplified to `. .venv/bin/activate`. See [here](https://python-poetry.org/docs/basic-usage/#using-poetry-run) for more detailed information how to run poetry e.g. for Windows and how to exit/deactivate it.
- We also have an `.envrc` that automatically sources the `.venv` when available. Check [direnv](https://direnv.net/) for how to set it up.
It will create a `.venv` in the root of the project.
You can use the `venv` like you would without the usage of `uv`:

Examples (all from vss-tools root and after running `poetry install`):

venv activation:
```bash
source .venv/bin/activate
vspec --help
```
poetry shell:
```bash
poetry shell
vspec --help
```

poetry run:
Alternatively you can use `uv` to run the `vspec` tool:

```bash
poetry run vspec --help
uv run vspec --help
```

direnv:
If you have a working setup of [direnv](https://direnv.net/), navigating into the directory
activates the `venv` (after an initial `uv sync` and `direnv allow`)

```bash
vspec --help
```

`uv` can manage independent python versions.
If you would like to use a specific python version (e.g. `python3.12`):

```bash
# Python 3.12
uv sync -p python3.12
# From the activated env:
python --version # --> Python 3.12.7

# Python 3.13
uv sync -p python3.13
python --version # --> Python 3.13.0
```

### Pre-commit

This project uses [pre-commit](https://pre-commit.com/) which helps formatting the source code in a streamlined way.
This project uses [pre-commit](https://pre-commit.com/) which helps to format the source code in a streamlined way.
It is very recommended to use it.
To install hooks you can do `pre-commit install` from an [activated environment](#poetry).
To install hooks you can do `pre-commit install` from an [activated environment](#uv).
Hooks will then run every time you do a `git commit` on changed files.
If you would like to run all hooks on all files you can do `pre-commit run --all`.
Since `pre-commit` dependencies are installed in the virtual environment, it needs
to be activated to be able to run them on a commit.


### Installing additional tools

If you intend to run test cases related to `vspec` proto exporter you need to install the protobuf compiler.
Expand Down
57 changes: 29 additions & 28 deletions binary/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Binary Toolset
The binary toolset consists of a tool that translates the VSS YAML specification to the binary file format (see below),
and two libraries that provides methods that are likely to be needed by a server that manages the VSS tree, one written in C, and one in Go.<br>
and two libraries that provides methods that are likely to be needed by a server that manages the VSS tree, one written in C, and one in Go.

## Binary Parser Usage

Expand Down Expand Up @@ -70,35 +70,36 @@ and assuming a binary tree file has been created in the VSS parent directory:

## Encoding

The binary node file format is as follows:<br>
Name | Datatype | #bytes<br>
---------------------------------------<br>
NameLen | uint8 | 1<br>
Name | chararray | NameLen<br>
NodeTypeLen | uint8 | 1<br>
NodeType | chararray | NodeTypeLen<br>
UuidLen | uint8 | 1<br>
Uuid | chararray | UuidLen<br>
DescrLen | uint8 | 1<br>
Description | chararray | DescrLen<br>
DatatypeLen | uint8 | 1<br>
Datatype | chararray | DatatypeLen<br>
MinLen | uint8 | 1<br>
Min | chararray | MinLen<br>
MaxLen | uint8 | 1<br>
Max | chararray | MaxLen<br>
UnitLen | uint8 | 1<br>
Unit | chararray | UnitLen<br>
AllowedLen | uint8 | 1<br>
Allowed | chararray | AllowedLen<br>
DefaultLen | uint8 | 1<br>
Default | chararray | AllowedLen<br>
ValidateLen | uint8 | 1<br>
Validate | chararray | ValidateLen<br>
Children | uint8 | 1<br><br>
The binary node file format is as follows:

Name | Datatype | #bytes
------------|-----------|--------------
NameLen | uint8 | 1
Name | chararray | NameLen
NodeTypeLen | uint8 | 1
NodeType | chararray | NodeTypeLen
UuidLen | uint8 | 1
Uuid | chararray | UuidLen
DescrLen | uint8 | 1
Description | chararray | DescrLen
DatatypeLen | uint8 | 1
Datatype | chararray | DatatypeLen
MinLen | uint8 | 1
Min | chararray | MinLen
MaxLen | uint8 | 1
Max | chararray | MaxLen
UnitLen | uint8 | 1
Unit | chararray | UnitLen
AllowedLen | uint8 | 1
Allowed | chararray | AllowedLen
DefaultLen | uint8 | 1
Default | chararray | AllowedLen
ValidateLen | uint8 | 1
Validate | chararray | ValidateLen
Children | uint8 | 1

The Allowed string contains an array of allowed, each Allowed is preceeded by two characters holding the size of the Allowed sub-string.
The size is in hex format, with values from "01" to "FF". An example is "03abc0A012345678902cd" which contains the three Alloweds "abc", "0123456789", and "cd".<br><br>
The size is in hex format, with values from "01" to "FF". An example is "03abc0A012345678902cd" which contains the three Alloweds "abc", "0123456789", and "cd".

The nodes are written into the file in the order given by a recursive method as shown in the following pseudocode:

Expand Down
7 changes: 4 additions & 3 deletions docs/vspec.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,17 @@ vspec export json --vspec spec/VehicleSignalSpecification.vspec --output vss.jso
INFO Serializing compact JSON... json.py:148

```
## Exporter docs
## Exporter Specific Documentation

- [apigear](./apigear.md)
- [tree](./tree.md)
- [binary](../binary/README.md)
- [ddsidl](./ddsidl.md)
- [go](./go.md)
- [graphql](./graphql.md)
- [id](./id.md)
- [protobuf](./protobuf.md)
- [samm](./samm.md)
- [go](./go.md)
- [tree](./tree.md)

## Argument Explanations

Expand Down
6 changes: 3 additions & 3 deletions docs/vspec_arch.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,13 +134,13 @@ branch.
The command line interface is designed with [click](https://click.palletsprojects.com/).
We are making use of subcommands and evaluating them lazy.
That means that we can define the cli interface of subcommands in their own files.
The main cli entrypoint for `vspec` is defined in [vspec.py](../src/vss_tools/vspec/vspec.py).
The main cli entrypoint for `vspec` is defined in [cli.py](../src/vss_tools/cli.py).
There you can also see all the subcommands referenced.
When building a new exporter tool, this is the place that needs to be adapted as well
as creating a new file similar to the other exporters in [exporters](../src/vss_tools/vspec/exporters).
as creating a new file similar to the other exporters in [exporters](../src/vss_tools/exporters).

It is also possible to build a complete different script entrypoint and not hook into the current interface.
All cli options can be imported and used via [cli_options.py](../src/vss_tools/vspec/cli_options.py).
All cli options can be imported and used via [cli_options.py](../src/vss_tools/cli_options.py).
Only a few are left out which are defined on the toplevel command just as `--log-level`, `--log-file` and `--version`.

## Linters and Static Code Checkers
Expand Down
21 changes: 21 additions & 0 deletions noxfile.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import nox

PYTHON_VERSIONS = ["3.10", "3.13"]

nox.options.default_venv_backend = "uv"


@nox.session(python=PYTHON_VERSIONS)
def tests(session):
session.run_install(
"uv",
"sync",
env={"UV_PROJECT_ENVIRONMENT": session.virtualenv.location},
)
session.run(
"pytest",
"--cov=vss_tools",
"--cov-report=term-missing",
"--cov-fail-under=88",
*session.posargs,
)
Loading

0 comments on commit dcde474

Please sign in to comment.