modularity overview - secondary topic - anchors

[ColinKennedy]

Discussed in #1242

I wasn't sure if this would be better as an issue, discussion, or a thread in the same discussion as #1242. I can move / delete / etc this as suggested.

I'd like to discuss the idea of "anchors". In the modularity overview, anchors as a formal, automated process of combining {root}/python from a number of Rez packages in a resolve into a single /foo/python namespace. It makes our resolved PYTHONPATH shorter as a result. However I see additional opportunities with this kind of functionality which I'd like to raise to the larger group.

At companies I've worked, a Rez package may look like this:

package.py  # name = "my_package"
python/
    my_company/
        `__init__.py`  <-- important file - shared namespace
        my_package/
            `__init__.py`  <-- non-shared namespace

In the __init__.py, there is something like:

__import__('pkg_resources').declare_namespace(__name__)

And you import my_package with from my_company import my_package. This "my_company" namespace exists in hundreds of Rez packages. However if anchors existed, it could look like this

package.py  # name = "my_package"
python/
    `__init__.py`  <-- non-shared namespace

And then the anchor path could be similar to:

anchors:
  - subpath: python
    target: /my_company/python

Now there's no need for the shared namespace at all. The anchor, I imagined, would be a literal folder + file /my_company/__init__.py and the modules would still be importable via from my_company import my_package as expected.

Anchors as a shared namespace replacement

So the scenario outlined above is a complete Python shared namespace replacement. Which is great! Because shared namespaces can actually be very slow compared to regular imports (one TD showed me 30 second import on the farm due to a shared namespace related issue).

However anchors could also be used to place code directly into /my_company/__init__.py, which would improve bootstrapping company-wide code. For an example, see below.

Anchors for bootstrapping

One useful thing that shared namespaces are good at is adding code in the parent namespace to apply to all child namespaces. For example, bootstrapping logging.

Imagine a /my_company/__init__.py that contains this code:

import logging
import sys

LOGGER = logging.getLogger(__name__)
HANDLER = logging.StreamHandler(sys.stdout)

level = logging.INFO

if os.getenv("REZ_EPH_ENABLE_VERBOSE_LOGGING", "0") == "1":
    level = logging.DEBUG

HANDLER.setLevel(level)
FORMATTER = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
HANDLER.setFormatter(FORMATTER)
LOGGER.addHandler(HANDLER)
LOGGER.setLevel(level)

Now in "my_package", to make a logger and print to the terminal, they just have to add

import logging

LOGGER = logging.getLogger(__name__)
LOGGER.info("Some message")

Without the shared namespace bootstrap, Python would error saying "No handler found for my_company.my_package". But with the bootstrap code, it would print to the terminal as expected.

There's even more variants of this approach which would be useful. For example, say you want a Rez environment that has more verbose logging but only for modules under my_company. Then in the rez-env you could do rez-env foo bar my_package .enable_verbose_logging-1 and the ephemeral, once found, would change setLevel from logging.INFO to logging.DEBUG. Now without changing any of the foo bar my_package packages, extra logging configuration is generated and added to the __init__.py by the runtime engine during the resolve.

Conclusion

All this to say, I see anchoring as an opportunity for the runtime engine to inject runtime behavior. In the simplest case, collecting Rez packages PYTHONPATH into one place is one example runtime behavior. In this other case I've outlined, the runtime behavior is that + adding logging-related code directly into the /my_company/__init__.py of target: /my_company/python so that it can apply to the child packages within the my_company namespace.

I see this anchor as being useful for ...

• Any behavior that you may want to set on a parent and inherit to all child paths
• Any behavior where you want a child to selectively reference from the parent directory

It could be configuration details, logging, or anything.

[nerdvegas]

So first thing, wrt:

"""anchors as a formal, automated process of combining {root}/python from a number of Rez packages in a resolve into a single /foo/python namespace"""

That's not quite right. Anchors are a formal way of describing how part of a package's payload can be remapped into some form of overlaid storage. The subtle but important difference here is that there is nothing python specific about the statement I just made! Rez is not a language-centric package manager.

Whatever you want to achieve should be achievable via that mechanism. For example, you might have some mycompany package that provides that global __init__.py that you want, and remaps it into the right place via anchors.

One thing to note - I can see how you'd easily end up with packages that only work when they're remapped the way you want them to be. Imo that would be a bad idea - now you'd be tightly coupling your package implementation with the fact that it has to be consumed in some overlaid runtime, which kinda defeats the purpose of abstracting away the runtime. However, you could do this if you wanted and I don't think rez should stop you - but I certainly wouldn't want to see this sort of thing in any public package repo we may end up with.

ps - one thing this does make me realise though is that we'll need an optional way of defining overlay order, because there may be times when that is important.

[ColinKennedy]

Agreed on all points. I used logging is an example because including it or omitting it won't prevent tools from working but also is useful for say, one Rez environment you want to log debug messages. Maybe another Rez environment it'd be "warning + errors only go to a database" etc. And all under one nice tiny namespace. But this discussion thread isn't language specific, just showing that example.

One thing to note - I can see how you'd easily end up with packages that only work when they're remapped the way you want them to be. Imo that would be a bad idea ... I certainly wouldn't want to see this sort of thing in any public package repo we may end up with.

Yep, I wouldn't either. Anchoring IMO would need to be used very prescriptively to prevent misuse (and probably also only used by an individual company's non-open-source tools). But as long as we can keep the feature open for these sorts of non-essential use cases, I still think it'd be a big win.