A future proof package and import system

This is a collection of thoughts on the design of a reliable package and import system that is ready for future applications. At this stage, this page mostly represents my personal view (Christian Menard). I will also focus on the C++ target here as this is the target I know best. The C target is not a good example for these considerations as there is a fundamental design issue with the C target. Since the code generator places all code in a single generated .c file and does things like #include reactor.c to avoid the need for Makefiles, it circumvents many of the issues that come with imports that I will outline here. It simply ignores file scopes and namespaces altogether.

The status quo

The current import system is lean and simple. Write import Bar.lf in Foo.lf and every reactor defined in Bar.lf will be visible in the file scope Foo.lf. Bar.lf is looked up simply by scanning the directory Foo.lf is placed in. This works well for the simple programs and tests we have right now, but does not scale. I identify the following problems:

1. There is no notion of separate namespaces. Every reactor that Bar.lf defines becomes visible in Foo.lf. If both files define a Reactor Foo, there is a name clash and the import would be ill-formed. There should be a mechanism to distinguish the two definitions of Foo, such as using fully qualified names: Foo.Foo and Bar.Foo.

2. There is no concept for importing files from a directory structure. It is unclear how Foo.lf could import my/lib/Bar.lf.

3. There is no concept for packages or libraries that can be installed on the system. How could we import Reactors from a library that someone else provided?

These are the more obvious issues that we have talked about. However, there are more subtle ones that we haven't been discussed in depth (or at least not in the context of the import system design discussion). The open question is: What does importing a LF file actually mean? Obviously, an import should bring Reactors defined in another files into local scope. But what should happen with the other structures that are part of an LF file, namely target properties and preambles? That is not specified and our targets use a best practice approach. But this is far away from a good design that is scalable and future proof.

A quick dive into the C++ code generator

Before I discuss the problems with preambles and target properties, I would like to give you a quick overview of how the C++ code generator works. Consider the following LF program consisting of two files Foo.lf and Bar.lf:

// Bar.lf

reactor Bar {
 reaction(startup) {=
   // do something bar like
 =}
}

// Foo.lf

import Bar.lf

reactor Foo {
 bar = new Bar();
 reaction(startup) {=
   // do something foo like
 =}
}

Now let us have a look on what the C++ code generator does. It will produce a file structure like this:

CMakeLists.txt
main.cc
Bar/
  Bar.cc
  Bar.hh
Foo/
  Foo.cc
  Foo.hh

We can ignore CMakeLists.txt and main.cc for our discussion here. The former specifies how the whole program can be build and the latter contains the main() function and some code that is required to get the application up and running. For each processed <file>.lf file, the code generator creates a directory <file>. For each reactor <reactor> defined in <file>.lf, it will create <file>/<reactor>.cc and <file>/<reactor>.hh. The header file declares a class representing the reactor like this:

// Bar/Bar.hh

# pragma once

#include "reactor-cpp/reactor-cpp.hh"

class Bar : public reactor::Reacor {
 private:
  // default actions
  reactor::StartupAction startup {"startup", this};
  reactor::ShutdownAction shutdown {"shutdown", this};

 public:
  /* ... */

 private:
  // reaction bodies 
  r0_body();
};

The corresponding Bar/Bar.cc will look something like this:

#include "Bar/Bar.hh"

/* ... */

Bar::r0_body() {
  // do something bar like
}

Similarly, Foo.hh and Foo.cc will be generated. However, since Foo.lf imports Bar.lf and instantiated the reactor Bar it must be made visible. This is done by an include directive in the generated code like so:

// Foo/Foo.hh
### 
# pragma once

#include "reactor-cpp/reactor-cpp.hh"
#include "Foo/Foo.hh"

class Foo : public reactor::Reacor {
 private:
  // default actions
  reactor::StartupAction startup;
  reactor::ShutdownAction shutdown;

  // reactor instances
  Bar bar;
 public:
  /* ... */

 private:
  // reaction bodies 
  r0_body();
};

The problem with preambles

The problems with preamble in the context of imports were already discussed in a related issue, but I would like to summarize the problem here. While the examples above worked nicely even with imports, things get messy as soon as we introduce a preamble. Let's try this:

// Bar.lf

reactor Bar {
 preamble {=
   struct bar_t {
     int x;
     std::string y;
   };

   bar_t bar_func {
     return bar_t(42, "hello")
   }
 =}
 output out:bar_t;
 reaction(startup) -> out {=
   out.set(bar_fuc());
 =}
}

// Foo.lf

import Bar.lf

reactor Foo 
 bar = new Bar();
 reaction(bar.out) {=
   auto& value = bar.out.get();
   std::cout << "Received {" << value->x << ", " << value->y << "}\n";
 =}
}

This would be expected to print Received {32, hello}. However, before we can even compile this program, we need to talk about what should happen with the preamble during code generation and how the import affects it. So where should the preamble go? The first thing that comes to mind, is to embed it in the header file Bar.hh something like this:

// Bar/Bar.hh

# pragma once

#include "reactor-cpp/reactor-cpp.hh"

// preamble
struct bar_t {
  int x;
  std::string y;
};

bar_t bar_func {
  return bar_t(42, "hello")
}

class Bar : public reactor::Reacor {
 /* ... */
};

If we embed the preamble like this and compile the program ,then the compiler is actually happy and processes all *.cc files without any complaints. But, there is a huge problem while liking the binary. The linker sees multiple definitions of bar_func and has no idea which one to use. Why is that? Well, the definition of bar_func is contained in a header file. This should never be done in C/C++! Since includes translate to a plain text replacement by the preprocessor, Bar.cc will contain the full definition of bar_func. As Foo.cc imports Foo.hh which imports Bar.hh, also Foo.cc will contain the full definition. And since main.cc also has to include Foo.hh, main.cc will also contain the full definition of bar_func. So we have multiple definitions of the same function and the linker rightfully reports this as an error.

So what should we do? We could place the preamble in Bar.cc instead. This ensures that only Bar.cc sees the definition of bar_func.