Overriding behaviour of rules files

[NikolausDemmel]

We are trying to determine what rules plugins should return during the update step. Closely related is the question on how the information of different rules files is merged. The following is a discussion on how this could be done and what are the requirements for some of the desired new functionality in xylem.

I am assuming that given the source files in sources.d, which have an order (alphabetical) and each have a ordered list of rules source entries (e.g. spec files), at some point we end up with an ordered list of rules definitions that need to be merged. I assume that each rules source entry generates one rules dictionary (or whatever data structure), with some a similar layout to what is described as the fully expanded format in https://github.com/catkin/xylem/blob/master/xylem/specs/rules.py#L236-L261. This means that the data structure for rules definitions is some form of nested dict with access pattern key -> os-name -> os-version -> package-manager, where os-name and os-version can have special values any_os and any_version.

Currently (see http://www.ros.org/reps/rep-0125.html), rosdep follows the simply rule that for any given key, the rule appearing first wins solely. There is no merging of multiple rules for the same key. Some level of control is given by using "tags" (e.g. fuerte, ubuntu, lucid), which can be specified for each rules source entry. An entry is only considered if all tags match the current operating system/version and ros distro. This is currently used in the default sources for the osx/homebrew specific rules and for the legacy gbpdistro entry specific to fuerte. I was under the impression, that these tags were meant to go away in xylem.

Question: Are these tags (os-name, os-version, rosdistrio) in the source files meant to go away?

In the following I assume that there are no such tags in the source files.

---

Merging different rules for specific os/version

Let us for now also ignore the rosdistro related entries and focus only on the situation with the default rules plugin of spec files. With the tags gone, there should be some way to merge rules for any specific key at least at the operating system level. I.e. to allow for homebrew definitions to be in a separate file, or in general, to allow independent installation of separate source files with definitions for additional (new) operating systems. The way this could be done is for a specific key, and a specific operating system, the top-most matching rule is used solely, so for example:

# first rules file
foo:
    ubuntu:
        apt: [foo]
bar:
    any_os:
        pip: [bar]

# second rules file
foo:
    any_os:
        pip: [foo]
bar:
    ubuntu:
        apt: [bar]

# merged definition
foo:
    any_os:
        pip: [foo]        
    ubuntu:
        apt: [foo]
bar:
    any_os:
        pip: [bar]

In a similar fashion, we might want to merge rules at the os-level for definitions of different os-versions:

# first rules file
foo:
    ubuntu:
        lucid: [foo-1]
bar:
    ubuntu: [bar-1]

# second rules file
foo:
    ubuntu: [foo-2]
bar:
    ubuntu:
        lucid: [bar-2]

# merged definition
foo:
    ubuntu:
        lucid: [foo-1]
        any_version: [foo-2]
bar:
    ubuntu:
        any_version: [bar-1]

This is relatively straight forward, however, there some cases which are maybe counter intuitive. For example:

# first rules file
foo:
    ubuntu:
        lucid: [foo-1]

# second rules file
foo:
    any_os: 
        pip: [foo-2]

# merged definition
foo:
    ubuntu:
        lucid: [foo-1]
    any_os: 
        pip: [foo-2]

with which foo does not resolve on any ubuntu version other than lucid. Here the question would be, if the any_os rule should kick in if there is a specific rule for the current os, but that rule has no clause matching the current os version. However, if we decide to allow this (i.e. foo resolves to foo-2 from pip on ubuntu precise with the above definitions), the following is still possibly confusing:

# first rules file
foo:
    ubuntu:
        lucid: [foo-1]
    any_os: 
        pip: [foo-1]

# second rules file
foo:
    ubuntu:
        any_version: [foo-2]

# merged definition
foo:
    ubuntu:
        lucid: [foo-1]
    any_os: 
        pip: [foo-1]

What might seem odd is that in the merged definition, foo resolves to the pip package manager on ubuntu precise even though there is a specific rule for apt on ubuntu in the second rules file that gets overwritten by the any_os rule in the first file even though there is an entry for ubuntu in the first file.

Still, I think we should choose that behavior IMO (but this is different from what is currently described for the new rules spec.

---

Multiple resolutions for a specific key for a specific os/version via multiple package managers

As another change in xylem over rosdep it was considered to resolve a specific key to multiple alternative package managers on the same os/version. Let us first consider this for a single rules file:

foo:
    osx:
        homebrew: [foo]
        macports: [foo]
bar: 
    osx:
        mavericks:
            homebrew: [bar]
        any_version:
            macports: [bar]
            fink: [bar]
baz:
    any_os:
        pip: [baz]
    ubuntu: 
        any_version: [baz]
        raring:
            my-installer: [baz]
        quantal: [baz-2]

Here, foo is resolved an all osx to both macports or homebrew,bar is available from macports or fink on all osx versions, and additionally from homebrew on Mavericks. Lastly, baz is resolved on pip everywhere, while ubuntu additionally has the resolution to baz from apt for all version but quantal, which has baz-2 from apt (and baz from pip); on raring finally baz is resolved to either baz from pip, baz from apt or baz from my-installer.

These set of rules might be obtained from the following separate files:

# first rules file (custom override)
baz:
    ubuntu: 
        raring:
            my-installer: [baz]
        quantal: [baz-2]

# second rules file (pip specific)
baz:
    any_os:
        pip: [baz]

# third rules file (homebrew specific)
foo:
    osx:
        homebrew: [foo]
bar: 
    osx:
        mavericks:
            homebrew: [bar]

# fourth rules file (osx specific)
foo:
    osx:
        macports: [foo]
bar: 
    osx:
        macports: [bar]
        fink: [bar]

# fifth rules file (base file)
baz:
    ubuntu: [baz]

Again, this is fairly straight forward but potentially has some confusing situations. Firstly, note that the order of these five files does not matter except that the first file come before the fifth (consider the apt rule on quantal).

The consequence of merging the rules at the package manager level is that rules definitions cannot easily be unset by a rule 'at the front', e.g. the homebrew rule for foo in the third file does not, as one could maybe expect hide the macports rule in the forth file. There needs to be a way to choose between the different possible package managers for a specific os/version. It has already been suggested, that for each package manager the os-plugins, pm-plugins and user configs can assign priorities. By default, for a given rule with multiple package managers on a specific os/version, the package manager with the highest priority is chosen.

This however is not sufficient, if the user for example wants make sure that for baz the pip installer is used on all os's. She could change the priority of the pip installer, but that would also affect other keys than baz. There are two possible remedies:

Firstly, there could be special sytax to unset specific package managers, e.g.

# second rules file (pip overrides apt for baz)
baz:
    any_os:
        pip: [baz]
        unset: [apt]
# or alternatively 
baz:
    any_os:
        pip: [baz]
        unset: all

Secondly, there could be syntax to assign high (or low) priority to a specific rule at the package manager level

# second rules file (pip overrides apt for baz)
baz:
    any_os:
        pip: 
            packages: [baz]
            priority: 100

assuming that the default priorities for apt and pip are 50 and 30, respectively.

Lastly, there could also be an additional command line option that overrides these priorities temporarily:

brew install baz --package-manager apt
# or
brew install baz --from apt

This specific example would not work togehter with the "unsetting" of apt above, since then the merged rules definition does not contain the resolution from apt any more.

---

I welcome opinions on this strategy for merging rules, at an os/version level, and possibly also at the package manager level. What might alternative strategies be in the light of allowing multiple package managers for a specific key/os/version triple?

[NikolausDemmel]

@wjwwood

[wjwwood]

Question: Are these tags (os-name, os-version, rosdistrio) in the source files meant to go away?

I am not 100% sure. I think that we cannot include things like the rosdistro specifically in the source files, but there may be room and need for such a mechanism in the general case. Let me think about this some and we should discuss it at the next meeting.

[NikolausDemmel]

One thing we have not considered at all so for, which I find unsatsfying, is the migration path for packages as they move from ROS packages to becoming system dependencies, e.g. as has happened to opencv: http://answers.ros.org/question/173773/depend-on-opencv-in-hydro/

Would it not be preferred to be able to retain the existing package name as a rosdep key? I'm not sure this can be done with the current proposal without filtering at the source-lists entry level? It might be doable if the rosdistro rules plugin can distinguish between rosdistros before rules files are merged (and if by default it comes before the other definitions), such that for hydro it could override the opencv2 rosdep key with the ros package, whereas for indigo and above it would not provide that rule and the ubuntu/homebrew/... provided packages are used.