Skip to content

Commit

Permalink
Tool restructuring
Browse files Browse the repository at this point in the history
Refactoring tool structure so that there is not a single entrypoint (vspec2x.py)
but rather individual ones. Makes it easier to add custom generators.

Signed-off-by: Erik Jaegervall <[email protected]>
  • Loading branch information
erikbosch committed Dec 22, 2023
1 parent 5343e9c commit ff3b513
Show file tree
Hide file tree
Showing 43 changed files with 886 additions and 798 deletions.
1 change: 0 additions & 1 deletion .github/workflows/buildcheck.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,6 @@ jobs:
mkdir /tmp/pypi_vss_test
cd /tmp/pypi_vss_test
# Just verify that we can start the tools
vspec2x.py --help
vspec2csv.py --help
vspec2json.py --help
vspec2yaml.py --help
Expand Down
2 changes: 1 addition & 1 deletion README-PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ If you just want the latest version this should be sufficient:
pip install vss-tools
```

When installed tools like `vspec2x.py` shall be available on your path.
When installed tools like `vspec2json.py` shall be available on your path.

For more information see the [VSS-Tools wiki](https://github.com/COVESA/vss-tools/wiki/PyPI-packing)

Expand Down
9 changes: 4 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,10 @@ Examples on tool usage can be found in the [VSS Makefile](https://github.com/COV

Tool | Description | Tool Category | Documentation |
| ------------------ | ----------- |-----------------------|-----------------------------------------------------------------------------------------------------------------------|
| [vspec2x.py](vspec2x.py) | Parses and expands VSS into different text based output formats. Currently supports `json`, `yaml`,`csv`,`idl` | Community Supported | Try `./vspec2x --help` or check [vspec2x documentation](docs/vspec2x.md) |
[vspec2csv.py](vspec2csv.py) | Shortcut for [vspec2x.py](vspec2x.py) generating CSV output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
[vspec2ddsidl.py](vspec2ddsidl.py) | Shortcut for [vspec2x.py](vspec2x.py) generating DDS-IDL output | Community Supported | [VSS2DDSIDL Documentation](docs/VSS2DDSIDL.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) |
[vspec2json.py](vspec2json.py) | Shortcut for [vspec2x.py](vspec2x.py) generating JSON output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
[vspec2yaml.py](vspec2yaml.py) | Shortcut for [vspec2x.py](vspec2x.py) generating flattened YAML output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
[vspec2csv.py](vspec2csv.py) | Generating CSV output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
[vspec2ddsidl.py](vspec2ddsidl.py) | Generating DDS-IDL output | Community Supported | [VSS2DDSIDL Documentation](docs/VSS2DDSIDL.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) |
[vspec2json.py](vspec2json.py) | Generating JSON output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
[vspec2yaml.py](vspec2yaml.py) | Generating flattened YAML output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
[vspec2binary.py](vspec2binary.py) | The binary toolset consists of a tool that translates the VSS YAML specification to the binary file format (see below), and two libraries that provides methods that are likely to be needed by a server that manages the VSS tree, one written in C, and one in Go | Community Supported | [vspec2binary Documentation](binary/README.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) |
[vspec2franca.py](vspec2franca.py) | Parses and expands a VSS and generates a Franca IDL specification | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
[vspec2c.py](obsolete/vspec2c.py) | The vspec2c tooling allows a vehicle signal specification to be translated from its source YAML file to native C code that has the entire specification encoded in it. | Obsolete (2022-11-01) | [Documentation](obsolete/vspec2c/README.md) |
Expand Down
81 changes: 35 additions & 46 deletions docs/vspec2x.md
Original file line number Diff line number Diff line change
@@ -1,58 +1,55 @@
# vspec2x converters
# Vspec2x-based generators

vspec2x is a family of VSS converters that share a common codebase.
[vspec2x](../vspec/vspec2x.py) is a generator framework that offers functionality to parse one or more vspec files and
transform that to an internal VSS model, which can be used as input for generators.

As a consequence it provides general commandline parameters guiding the parsing of vspec, as well as parameters specific to specific output formats.
Vspec2x-based generators have a number of common arguments.
It is partially configurable which arguments that shall be available for each generator,
so all arguments described in this documents are not available for all generators.
In addition to this generator-specific arguments may be supported

You can get a description of supported commandline parameters by running `vspec2x.py --help`.

This documentation will give some examples and elaborate more on specific parameters.
You can get a description of supported commandline arguments by running `<toolname> --help`, e.g. `vss2json.py --help`.
In this document `vspec2json.py` is generally used as example, but the same syntax is typically available also for other tools.

The supported arguments might look like this

```
usage: vspec2x.py [-h] [-I dir] [-e EXTENDED_ATTRIBUTES] [-s] [--abort-on-unknown-attribute] [--abort-on-name-style]
[--format format] [--uuid] [--no_expand] [-o overlays] [-u unit_file] [-q quantity_file]
[-vt vspec_types_file] [-ot <types_output_file>]
[--json-all-extended-attributes] [--json-pretty]
[--yaml-all-extended-attributes] [-v version] [--all-idl-features] [--gqlfield GQLFIELD GQLFIELD]
<vspec_file> <output_file>
```
$ vspec2json.py --help
usage: vspec2json.py [-h] [-I dir] [-e EXTENDED_ATTRIBUTES] [-s] [--abort-on-unknown-attribute] [--abort-on-name-style] [--uuid] [--no-expand]
[-o overlays] [-q quantity_file] [-u unit_file] [-vt vspec_types_file]
[-ot <types_output_file>] [--json-all-extended-attributes] [--json-pretty]
<vspec_file> <output_file>
```

An example command line to convert the VSS standard catalog into a JSON file is

```
% python vspec2x.py --format json -I ../spec -u ../spec/units.yaml ../spec/VehicleSignalSpecification.vspec vss.json
Output to json format
Known extended attributes:
Reading unit definitions from ../spec/units.yaml
Loading vspec from ../spec/VehicleSignalSpecification.vspec...
Calling exporter...
Generating JSON output...
Serializing compact JSON...
All done.
$ vspec2json.py -I ../spec -u ../vehicle_signal_specification/spec/units.yaml ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec vss.json
INFO Known extended attributes:
INFO Added 29 quantities from /home/erik/vehicle_signal_specification/spec/quantities.yaml
INFO Added 61 units from ../vehicle_signal_specification/spec/units.yaml
INFO Loading vspec from ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec...
INFO Calling exporter...
INFO Generating JSON output...
INFO Serializing compact JSON...
INFO All done.
```

This assumes you checked out the [COVESA Vehicle Signal Specification](https://github.com/covesa/vehicle_signal_specification) which contains vss-tools including vspec2x as a submodule.
This assumes you checked out the [COVESA Vehicle Signal Specification](https://github.com/covesa/vehicle_signal_specification) which contains vss-tools as a submodule.

The `-I` parameter adds a directory to search for includes referenced in you `.vspec` files. `-I` can be used multiple times to specify more include directories. The `-u` parameter specifies the unit file(s) to use. The `-q` parameter specifies quantity file(s) to use.

The first positional argument - `../spec/VehicleSignalSpecification.vspec` in the example - gives the (root) `.vspec` file to be converted. The second positional argument - `vss.json` in the example - is the output file.

The `--format` parameter determines the output format, `JSON` in our example. If format is omitted `vspec2x` tries to guess the correct output format based on the extension of the second positional argument. Alternatively vss-tools supports *shortcuts* for community supported exporters, e.g. `vspec2json.py` for generating JSON. The shortcuts really only add the `--format` parameter for you, so

```
python vspec2json.py -I ../spec -u ../spec/units.yaml ../spec/VehicleSignalSpecification.vspec vss.json
```

is equivalent to the example above.
It is the file `vspec2json.py` that specified which generator to use, i.e. which output to generate.

## General parameters

### --abort-on-unknown-attribute
Terminates parsing when an unknown attribute is encountered, that is an attribute that is not defined in the [VSS standard catalogue](https://covesa.github.io/vehicle_signal_specification/rule_set/), and not whitelisted using the extended attribute parameter `-e` (see below).

*Note*: Here an *attribute* refers to VSS signal metadata sich as "datatype", "min", "max", ... and not to the VSS signal type attribute
*Note*: Here an *attribute* refers to VSS signal metadata such as "datatype", "min", "max", ... and not to the VSS signal type attribute

### --abort-on-name-style
Terminates parsing, when the name of a signal does not follow [VSS recomendations](https://covesa.github.io/vehicle_signal_specification/rule_set/basics/#naming-conventions).
Expand Down Expand Up @@ -214,7 +211,7 @@ When deciding which quantities to use the tooling use the following logic:
As of today use of quantity files is optional, and tooling will only give a warning if a unit use a quantity not specified in a quantity file.

## Handling of overlays and extensions
`vspec2x` allows composition of several overlays on top of a base vspec, to extend the model or overwrite certain metadata. Check [VSS documentation](https://covesa.github.io/vehicle_signal_specification/introduction/) on the concept of overlays.
The generator framework allows composition of several overlays on top of a base vspec, to extend the model or overwrite certain metadata. Check [VSS documentation](https://covesa.github.io/vehicle_signal_specification/introduction/) on the concept of overlays.

Overlays are in general injected before the VSS tree is expanded. Expansion is the process where branches with instances are transformed into multiple branches.
An example is the `Vehicle.Cabin.Door` branch which during expansion get transformed into `Vehicle.Cabin.Door.Row1.Left`, `Vehicle.Cabin.Door.Row1.Right`, `Vehicle.Cabin.Door.Row2.Left`and `Vehicle.Cabin.Door.Row2.Right`.
Expand Down Expand Up @@ -357,23 +354,15 @@ Will also generate non-payload const attributes such as unit/datatype. Default i
Add additional fields to the nodes in the graphql schema. use: <field_name> <description>


## Writing your own exporter
This is easy. Put the code in file in the [vssexporters directory](../vspec/vssexporters/).
## Writing your own generator

Mandatory functions to be implemented are
```python
def add_arguments(parser: argparse.ArgumentParser):
```

and
This is done by creating a subclass of `class Vss2X`, see [vss2x.py](../vspec/vss2x.py).
You need at least to implement the abstract method `generate`.
In addition you may want to customize which generic arguments that shall be used and add
generator specific arguments, if needed.

```python
def export(config: argparse.Namespace, root: VSSNode):
```
See one of the existing exporters for an example.

Add your exporter module to the `Exporter` class in [vspec2x.py](../vspec2x.py).
A generic implementation example is available as a [test case](../tests/generators).

## Design Decisions and Architecture

Please see [vspec2x architecture document](vspec2x_arch.md).
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
url='https://github.com/COVESA/vss-tools',
license='Mozilla Public License 2.0',
packages=find_packages(exclude=('tests', 'contrib')),
scripts=['vspec2csv.py', 'vspec2x.py', 'vspec2franca.py', 'vspec2json.py', 'vspec2jsonschema.py',
scripts=['vspec2csv.py', 'vspec2franca.py', 'vspec2json.py', 'vspec2jsonschema.py',
'vspec2ddsidl.py', 'vspec2yaml.py', 'vspec2protobuf.py', 'vspec2graphql.py',
'vspec2id.py'],
python_requires='>=3.8',
Expand Down
66 changes: 66 additions & 0 deletions tests/generators/example_generator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
# Copyright (c) 2023 Contributors to COVESA
#
# This program and the accompanying materials are made available under the
# terms of the Mozilla Public License 2.0 which is available at
# https://www.mozilla.org/en-US/MPL/2.0/
#
# SPDX-License-Identifier: MPL-2.0

import sys
import logging
from vspec.model.vsstree import VSSNode
import argparse
from typing import Optional
from vspec.vss2x import Vss2X
from vspec.vspec2vss_config import Vspec2VssConfig
from vspec.vspec2x import Vspec2X


class ExampleGenerator(Vss2X):
"""
This is an example on how easy you can write your own generator
"""

def __init__(self, vspec2vss_config: Vspec2VssConfig, keyword: str):
# Change default configs
# By default Vspec2X requires that an output file is needed and specified
# That is not needed for the ExampleGenerator as it just use stdout
# By that reason we change default config to indicate that no output arguments is needed
# That also implies that there will be an error if an output argument is specified
vspec2vss_config.output_file_required = False

# A lot of other things we do not care about, this reduces number or arguments shown
# if you do "./example_generator.py --help"
vspec2vss_config.uuid_supported = False
vspec2vss_config.extended_attributes_supported = False
vspec2vss_config.type_tree_supported = False

# The rest here is generator-specific initializations
self.keyword = str.lower(keyword)
self.count = 0

def handle_node(self, node: VSSNode):
if self.keyword in str.lower(node.comment):
self.count += 1
for child in node.children:
self.handle_node(child)

def generate(self, config: argparse.Namespace, signal_root: VSSNode, vspec2vss_config: Vspec2VssConfig,
data_type_root: Optional[VSSNode] = None) -> None:
"""
It is required to implement the generate method
"""
self.handle_node(signal_root)

logging.info("Generating Example output...")
logging.info("I found %d comments with %s", self.count, self.keyword)


if __name__ == "__main__":
vspec2vss_config = Vspec2VssConfig()
# The generator shall know nothing about vspec processing or vspec2vss arguments!
# (Even if it may have some expectations on how the model look like)
generator = ExampleGenerator(vspec2vss_config, "VSS contributor")
vspec2x = Vspec2X(generator, vspec2vss_config)
vspec2x.main(sys.argv[1:])
34 changes: 34 additions & 0 deletions tests/generators/test.vspec
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Copyright (c) 2023 Contributors to COVESA
#
# This program and the accompanying materials are made available under the
# terms of the Mozilla Public License 2.0 which is available at
# https://www.mozilla.org/en-US/MPL/2.0/
#
# SPDX-License-Identifier: MPL-2.0

A:
type: branch
description: Branch A.

############ Testing Single Line Comments ##############

A.Erik:
datatype: float
type: sensor
unit: km
description: A sensor.
comment: A VSS contributor!

A.Sebastian:
datatype: float
type: sensor
unit: km
description: A sensor.
comment: A VSS contributor!

A.ChuckNorris:
datatype: float
type: sensor
unit: km
description: A sensor.
comment: An actor!
30 changes: 30 additions & 0 deletions tests/generators/test_generator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Copyright (c) 2023 Contributors to COVESA
#
# This program and the accompanying materials are made available under the
# terms of the Mozilla Public License 2.0 which is available at
# https://www.mozilla.org/en-US/MPL/2.0/
#
# SPDX-License-Identifier: MPL-2.0

import pytest
import os


@pytest.fixture
def change_test_dir(request, monkeypatch):
# To make sure we run from test directory
monkeypatch.chdir(request.fspath.dirname)


def test_generator(change_test_dir):

test_str = "./example_generator.py -u ../vspec/test_units.yaml test.vspec > out.txt 2>&1"
result = os.system(test_str)
assert os.WIFEXITED(result)
assert os.WEXITSTATUS(result) == 0

test_str = 'grep \"I found 2 comments with vss contributor\" out.txt > /dev/null'
result = os.system(test_str)
assert os.WIFEXITED(result)
assert os.WEXITSTATUS(result) == 0
os.system("rm -f out.txt")
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ def test_error(change_test_dir):
"""
Verify that you cannot have multiple type trees, ie. both TypesA and TypesB, there must be a single root
"""
test_str = "../../../vspec2x.py --format csv -vt struct1.vspec -vt struct2.vspec -u ../test_units.yaml " \
test_str = "../../../vspec2csv.py -vt struct1.vspec -vt struct2.vspec -u ../test_units.yaml " \
"test.vspec out.csv 1> out.txt 2>&1"
result = os.system(test_str)
assert os.WIFEXITED(result)
Expand Down
Loading

0 comments on commit ff3b513

Please sign in to comment.