Tool restructuring

.github/workflows/buildcheck.yml

@@ -76,7 +76,6 @@ jobs:
           mkdir /tmp/pypi_vss_test
           cd /tmp/pypi_vss_test
           # Just verify that we can start the tools
-          vspec2x.py --help
           vspec2csv.py --help
           vspec2json.py --help
           vspec2yaml.py --help

README-PYPI.md

@@ -24,7 +24,7 @@ If you just want the latest version this should be sufficient:
 pip install vss-tools
 ```
 
-When installed tools like `vspec2x.py` shall be available on your path.
+When installed tools like `vspec2json.py` shall be available on your path.
 
 For more information see the [VSS-Tools wiki](https://github.com/COVESA/vss-tools/wiki/PyPI-packing)
 

README.md

@@ -25,11 +25,10 @@ Examples on tool usage can be found in the [VSS Makefile](https://github.com/COV
 
  Tool | Description | Tool Category         | Documentation                                                                                                         |
 | ------------------ | ----------- |-----------------------|-----------------------------------------------------------------------------------------------------------------------|
-| [vspec2x.py](vspec2x.py) | Parses and expands VSS into different text based output formats. Currently supports `json`, `yaml`,`csv`,`idl`  | Community Supported   | Try `./vspec2x --help` or check [vspec2x documentation](docs/vspec2x.md)                                              |
-[vspec2csv.py](vspec2csv.py) | Shortcut for [vspec2x.py](vspec2x.py) generating CSV output | Community Supported   | Check [vspec2x documentation](docs/vspec2x.md)                                                                        |
-[vspec2ddsidl.py](vspec2ddsidl.py) | Shortcut for [vspec2x.py](vspec2x.py) generating DDS-IDL output | Community Supported   | [VSS2DDSIDL Documentation](docs/VSS2DDSIDL.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) |
-[vspec2json.py](vspec2json.py) |  Shortcut for [vspec2x.py](vspec2x.py) generating JSON output | Community Supported   | Check [vspec2x documentation](docs/vspec2x.md)                                                                        |
-[vspec2yaml.py](vspec2yaml.py) | Shortcut for [vspec2x.py](vspec2x.py) generating flattened YAML output | Community Supported   | Check [vspec2x documentation](docs/vspec2x.md)                                                                        |
+[vspec2csv.py](vspec2csv.py) | Generating CSV output | Community Supported   | Check [vspec2x documentation](docs/vspec2x.md)                                                                        |
+[vspec2ddsidl.py](vspec2ddsidl.py) | Generating DDS-IDL output | Community Supported   | [VSS2DDSIDL Documentation](docs/VSS2DDSIDL.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) |
+[vspec2json.py](vspec2json.py) |  Generating JSON output | Community Supported   | Check [vspec2x documentation](docs/vspec2x.md)                                                                        |
+[vspec2yaml.py](vspec2yaml.py) | Generating flattened YAML output | Community Supported   | Check [vspec2x documentation](docs/vspec2x.md)                                                                        |
 [vspec2binary.py](vspec2binary.py) | The binary toolset consists of a tool that translates the VSS YAML specification to the binary file format (see below), and two libraries that provides methods that are likely to be needed by a server that manages the VSS tree, one written in C, and one in Go | Community Supported   | [vspec2binary Documentation](binary/README.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) |
 [vspec2franca.py](vspec2franca.py) | Parses and expands a VSS and generates a Franca IDL specification | Community Supported   | Check [vspec2x documentation](docs/vspec2x.md)                                                                        |
 [vspec2c.py](obsolete/vspec2c.py) | The vspec2c tooling allows a vehicle signal specification to be translated from its source YAML file to native C code that has the entire specification encoded in it. | Obsolete (2022-11-01) | [Documentation](obsolete/vspec2c/README.md)                                                                           |

docs/vspec2x.md

@@ -1,58 +1,55 @@
-# vspec2x converters
+# Vspec2x-based generators
 
-vspec2x is a family of VSS converters that share a common codebase.
+[vspec2x](../vspec/vspec2x.py) is a generator framework that offers functionality to parse one or more vspec files and
+transform that to an internal VSS model, which can be used as input for generators.
 
-As a consequence it provides general commandline parameters guiding the parsing of vspec, as well as parameters specific to specific output formats.
+Vspec2x-based generators have a number of common arguments.
+It is partially configurable which arguments that shall be available for each generator,
+so all arguments described in this documents are not available for all generators.
+In addition to this generator-specific arguments may be supported
 
-You can get a description of supported commandline parameters by running `vspec2x.py --help`.
-
-This documentation will give some examples and elaborate more on specific parameters.
+You can get a description of supported commandline arguments by running `<toolname> --help`, e.g. `vss2json.py --help`.
+In this document `vspec2json.py` is generally used as example, but the same syntax is typically available also for other tools.
 
 The supported arguments might look like this
 
- ```
-usage: vspec2x.py [-h] [-I dir] [-e EXTENDED_ATTRIBUTES] [-s] [--abort-on-unknown-attribute] [--abort-on-name-style]
-                  [--format format] [--uuid] [--no_expand] [-o overlays] [-u unit_file] [-q quantity_file]
-                  [-vt vspec_types_file] [-ot <types_output_file>]
-                  [--json-all-extended-attributes] [--json-pretty]
-                  [--yaml-all-extended-attributes] [-v version] [--all-idl-features] [--gqlfield GQLFIELD GQLFIELD]
-                  <vspec_file> <output_file>
+```
+$ vspec2json.py --help
+usage: vspec2json.py [-h] [-I dir] [-e EXTENDED_ATTRIBUTES] [-s] [--abort-on-unknown-attribute] [--abort-on-name-style] [--uuid] [--no-expand]
+                     [-o overlays] [-q quantity_file] [-u unit_file] [-vt vspec_types_file]
+                     [-ot <types_output_file>] [--json-all-extended-attributes] [--json-pretty]
+                     <vspec_file> <output_file>
 ```
 
 An example command line to convert the VSS standard catalog into a JSON file is
 
 ```
-% python vspec2x.py --format json  -I ../spec -u ../spec/units.yaml ../spec/VehicleSignalSpecification.vspec vss.json
-Output to json format
-Known extended attributes:
-Reading unit definitions from ../spec/units.yaml
-Loading vspec from ../spec/VehicleSignalSpecification.vspec...
-Calling exporter...
-Generating JSON output...
-Serializing compact JSON...
-All done.
+$ vspec2json.py -I ../spec -u ../vehicle_signal_specification/spec/units.yaml ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec vss.json
+INFO     Known extended attributes:
+INFO     Added 29 quantities from /home/erik/vehicle_signal_specification/spec/quantities.yaml
+INFO     Added 61 units from ../vehicle_signal_specification/spec/units.yaml
+INFO     Loading vspec from ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec...
+INFO     Calling exporter...
+INFO     Generating JSON output...
+INFO     Serializing compact JSON...
+INFO     All done.
+
 ```
 
-This assumes you checked out the [COVESA Vehicle Signal Specification](https://github.com/covesa/vehicle_signal_specification) which contains vss-tools including vspec2x as a submodule.
+This assumes you checked out the [COVESA Vehicle Signal Specification](https://github.com/covesa/vehicle_signal_specification) which contains vss-tools as a submodule.
 
 The `-I` parameter adds a directory to search for includes referenced in you `.vspec` files. `-I` can be used multiple times to specify more include directories. The `-u` parameter specifies the unit file(s) to use. The `-q` parameter specifies quantity file(s) to use.
 
 The first positional argument - `../spec/VehicleSignalSpecification.vspec` in the example  - gives the (root) `.vspec` file to be converted. The second positional argument  - `vss.json` in the example - is the output file.
 
-The `--format` parameter determines the output format, `JSON` in our example. If format is omitted `vspec2x` tries to guess the correct output format based on the extension of the second positional argument. Alternatively vss-tools supports *shortcuts* for community supported exporters, e.g. `vspec2json.py` for generating JSON. The shortcuts really only add the `--format` parameter for you, so
-
-```
-python vspec2json.py  -I ../spec -u ../spec/units.yaml ../spec/VehicleSignalSpecification.vspec vss.json
-```
-
-is equivalent to the example above.
+It is the file `vspec2json.py` that specified which generator to use, i.e. which output to generate.
 
 ## General parameters
 
 ### --abort-on-unknown-attribute
 Terminates parsing when an unknown attribute is encountered, that is an attribute that is not defined in the [VSS standard catalogue](https://covesa.github.io/vehicle_signal_specification/rule_set/), and not whitelisted using the extended attribute parameter `-e` (see below).
 
-*Note*: Here an *attribute* refers to VSS signal metadata sich as "datatype", "min", "max", ... and not to the VSS signal type attribute
+*Note*: Here an *attribute* refers to VSS signal metadata such as "datatype", "min", "max", ... and not to the VSS signal type attribute
 
 ###  --abort-on-name-style
 Terminates parsing, when the name of a signal does not follow [VSS recomendations](https://covesa.github.io/vehicle_signal_specification/rule_set/basics/#naming-conventions).
@@ -214,7 +211,7 @@ When deciding which quantities to use the tooling use the following logic:
 As of today use of quantity files is optional, and tooling will only give a warning if a unit use a quantity not specified in a quantity file.
 
 ## Handling of overlays and extensions
-`vspec2x` allows composition of several overlays on top of a base vspec, to extend the model or overwrite certain metadata. Check [VSS documentation](https://covesa.github.io/vehicle_signal_specification/introduction/) on the concept of overlays.
+The generator framework allows composition of several overlays on top of a base vspec, to extend the model or overwrite certain metadata. Check [VSS documentation](https://covesa.github.io/vehicle_signal_specification/introduction/) on the concept of overlays.
 
 Overlays are in general injected before the VSS tree is expanded. Expansion is the process where branches with instances are transformed into multiple branches.
 An example is the `Vehicle.Cabin.Door` branch which during expansion get transformed into `Vehicle.Cabin.Door.Row1.Left`, `Vehicle.Cabin.Door.Row1.Right`, `Vehicle.Cabin.Door.Row2.Left`and `Vehicle.Cabin.Door.Row2.Right`.
@@ -357,23 +354,15 @@ Will also generate non-payload const attributes such as unit/datatype. Default i
 Add additional fields to the nodes in the graphql schema. use: <field_name> <description>
 
 
-## Writing your own exporter
-This is easy. Put the code in file in the [vssexporters directory](../vspec/vssexporters/).
+## Writing your own generator
 
-Mandatory functions to be implemented are
-```python
-def add_arguments(parser: argparse.ArgumentParser):
-```
-
-and
+This is done by creating a subclass of `class Vss2X`, see [vss2x.py](../vspec/vss2x.py).
+You need at least to implement the abstract method `generate`.
+In addition you may want to customize which generic arguments that shall be used and add
+generator specific arguments, if needed.
 
-```python
-def export(config: argparse.Namespace, root: VSSNode):
-```
-See one of the existing exporters for an example.
 
-Add your exporter module to the `Exporter` class in [vspec2x.py](../vspec2x.py).
+A generic implementation example is available as a [test case](../tests/generators).
 
-## Design Decisions and Architecture
 
 Please see [vspec2x architecture document](vspec2x_arch.md).

setup.py

@@ -35,7 +35,7 @@
     url='https://github.com/COVESA/vss-tools',
     license='Mozilla Public License 2.0',
     packages=find_packages(exclude=('tests', 'contrib')),
-    scripts=['vspec2csv.py', 'vspec2x.py', 'vspec2franca.py', 'vspec2json.py', 'vspec2jsonschema.py',
+    scripts=['vspec2csv.py', 'vspec2franca.py', 'vspec2json.py', 'vspec2jsonschema.py',
              'vspec2ddsidl.py', 'vspec2yaml.py', 'vspec2protobuf.py', 'vspec2graphql.py',
              'vspec2id.py'],
     python_requires='>=3.8',

tests/generators/example_generator.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+# Copyright (c) 2023 Contributors to COVESA
+#
+# This program and the accompanying materials are made available under the
+# terms of the Mozilla Public License 2.0 which is available at
+# https://www.mozilla.org/en-US/MPL/2.0/
+#
+# SPDX-License-Identifier: MPL-2.0
+
+import sys
+import logging
+from vspec.model.vsstree import VSSNode
+import argparse
+from typing import Optional
+from vspec.vss2x import Vss2X
+from vspec.vspec2vss_config import Vspec2VssConfig
+from vspec.vspec2x import Vspec2X
+
+
+class ExampleGenerator(Vss2X):
+    """
+    This is an example on how easy you can write your own generator
+    """
+
+    def __init__(self, vspec2vss_config: Vspec2VssConfig):
+        # Change default configs
+        # By default Vspec2X requires that an output file is needed and specified
+        # That is not needed for the ExampleGenerator as it just use stdout
+        # By that reason we change default config to indicate that no output arguments is needed
+        # That also implies that there will be an error if an output argument is specified
+        vspec2vss_config.output_file_required = False
+
+        # A lot of other things we do not care about, this reduces number or arguments shown
+        # if you do "./example_generator.py --help"
+        vspec2vss_config.uuid_supported = False
+        vspec2vss_config.extended_attributes_supported = False
+        vspec2vss_config.type_tree_supported = False
+
+        # The rest here is generator-specific initializations
+        self.count = 0
+
+    def add_arguments(self, parser: argparse.ArgumentParser):
+        """
+        Add generator specific arguments
+        """
+        parser.add_argument('-k', '--keyword',
+                            help="Phrase to search for", required=True)
+
+    def handle_node(self, node: VSSNode):
+        if self.keyword in str.lower(node.comment):
+            self.count += 1
+        for child in node.children:
+            self.handle_node(child)
+
+    def generate(self, config: argparse.Namespace, signal_root: VSSNode, vspec2vss_config: Vspec2VssConfig,
+                 data_type_root: Optional[VSSNode] = None) -> None:
+        """
+        It is required to implement the generate method
+        """
+        self.keyword = str.lower(config.keyword)
+        self.handle_node(signal_root)
+
+        logging.info("Generating Example output...")
+        logging.info("I found %d comments with %s", self.count, self.keyword)
+
+
+if __name__ == "__main__":
+    vspec2vss_config = Vspec2VssConfig()
+    # The generator shall know nothing about vspec processing or vspec2vss arguments!
+    # (Even if it may have some expectations on how the model look like)
+    generator = ExampleGenerator(vspec2vss_config)
+    vspec2x = Vspec2X(generator, vspec2vss_config)
+    vspec2x.main(sys.argv[1:])

tests/generators/test.vspec

@@ -0,0 +1,34 @@
+# Copyright (c) 2023 Contributors to COVESA
+#
+# This program and the accompanying materials are made available under the
+# terms of the Mozilla Public License 2.0 which is available at
+# https://www.mozilla.org/en-US/MPL/2.0/
+#
+# SPDX-License-Identifier: MPL-2.0
+
+A:
+  type: branch
+  description: Branch A.
+
+############ Testing Single Line Comments ##############
+
+A.Erik:
+  datatype: float
+  type: sensor
+  unit: km
+  description: A sensor.
+  comment: A VSS contributor!
+
+A.Sebastian:
+  datatype: float
+  type: sensor
+  unit: km
+  description: A sensor.
+  comment: A VSS contributor!
+
+A.ChuckNorris:
+  datatype: float
+  type: sensor
+  unit: km
+  description: A sensor.
+  comment: An actor!

tests/generators/test_generator.py

@@ -0,0 +1,30 @@
+# Copyright (c) 2023 Contributors to COVESA
+#
+# This program and the accompanying materials are made available under the
+# terms of the Mozilla Public License 2.0 which is available at
+# https://www.mozilla.org/en-US/MPL/2.0/
+#
+# SPDX-License-Identifier: MPL-2.0
+
+import pytest
+import os
+
+
+@pytest.fixture
+def change_test_dir(request, monkeypatch):
+    # To make sure we run from test directory
+    monkeypatch.chdir(request.fspath.dirname)
+
+
+def test_generator(change_test_dir):
+
+    test_str = "./example_generator.py -k \"VSS Contributor\" -u ../vspec/test_units.yaml test.vspec > out.txt 2>&1"
+    result = os.system(test_str)
+    assert os.WIFEXITED(result)
+    assert os.WEXITSTATUS(result) == 0
+
+    test_str = 'grep \"I found 2 comments with vss contributor\" out.txt > /dev/null'
+    result = os.system(test_str)
+    assert os.WIFEXITED(result)
+    assert os.WEXITSTATUS(result) == 0
+    os.system("rm -f  out.txt")

tests/vspec/test_multiple_type_trees/test_multiple_type_trees.py

@@ -22,7 +22,7 @@ def test_error(change_test_dir):
     """
     Verify that you cannot have multiple type trees, ie. both TypesA and TypesB, there must be a single root
     """
-    test_str = "../../../vspec2x.py --format csv -vt struct1.vspec -vt struct2.vspec -u ../test_units.yaml " \
+    test_str = "../../../vspec2csv.py -vt struct1.vspec -vt struct2.vspec -u ../test_units.yaml " \
                "test.vspec out.csv 1> out.txt 2>&1"
     result = os.system(test_str)
     assert os.WIFEXITED(result)