modularity overview

[nerdvegas]

Overview

This is a general discussion of the core modules of rez, how they interrelate with one another, and what might be improved.

Current System

Consider this diagram showing the current state of rez (we refer to core functionality only, for eg we don't touch on anything build related here):

Note that:

• Blue denotes constant rez components (ie unchanging core functionality)
• Green denotes extensible (plugin-based) rez components
• Yellow denotes transitory data or objects

Here we see an overview of the components involved in generating a configured runtime in response to a user's request:

1. A user invokes rez-env, passing a list of package requests
2. The solver performs dependency resolution, using package definitions found from one or more package repositories, specified via a searchpath ($REZ_PACKAGES_PATH typically).
3. When the solver finishes, a "context" is created. This stores the list of resolved variants, along with metadata about the solve (the initial request, solve graph etc)
4. The context, package definitions, and a shell implementation (whichever is configured to use) are then used to create the configured runtime (this is currently all done in the ResolvedContext class).

Rez aims to be an extensible packaging system. However from that POV, there are some problems with the current system:

• Package payloads are assumed to be already available on disk
• Package payloads are stored in a location derived directly from the packages' repository (and in the case of the default filesystem repo type, they are stored within the package repo)
• The configured runtime can only reference existing payloads (it cannot for example, copy and merge them into one location). This results in issues such as long PYTHONPATH

Proposed System

It should be possible to use rez to mix and match various different sources of package data, and to express those as a configured runtime in different ways. For example, it should be possible to:

• Read package definitions from a REST service, download their associated payloads on demand from S3, and then consume them via an overlayfs- based runtime (thus package payloads overlap and what would otherwise be many separate PYTHONPATH entries could be collapsed into one)
• Read package definitions and their payloads from disk, and consume them in a 'standard' runtime (matching current rez behaviour)
• Read package definitions from postgres, download their payloads from an artifact server, and consume them in a runtime that constructs symlink-based payloads on demand (for similar end result to overlayfs, for eg)
• Read package definitions from disk, download their payloads as images from a container registry, and build them into a target runtime image on demand; the runtime in this case is then running an instance of the built image
• Any combination of the above (within reason)

A version of rez able to do these kinds of things might look like so:

As is done currently, the solver takes a user request, reads package definitions from a searchpath of package repositories, performs dependency resolution, and creates a context. However, we now introduce two new extensible objects to the system - an "artifact repository", and a "runtime engine".

Artifact Repository

The artifact repository is able to take package definitions from a package repository, and resolve them into some local consumable resource - often simply a directory on disk containing the package payload, for example. "Local" here is a little ambiguous, what I really mean is just a resource accessible to the user so that the runtime can be constructed. For example you may have an artifact repo that downloads package payloads from S3, and copies them into shared posix storage, where they're consumed by multiple users at your studio. A different artifact repo might unzip files from shared storage, into directories directly on users' local disk. It could even make sense to chain artifact repos together - perhaps your S3 repo downloads zipped artifacts to shared storage, then a "localization" repo unzips them onto local disk on demand.

It may often make sense to have different artifact repos associated with different package repos. For example, it's very convenient to install development packages to your standard ~/packages package repo, and to store your package payloads there too; however you may want your released packages to have their definitions written to postgres, and their payloads to S3. Rez should be able to support these two pairs of package/artifact repos at the same time - it shouldn't matter where a package was sourced from in your runtime (clearly there are limits - you wouldn't be able to mix disk- and container- based payloads within the one runtime, for example).

As a thought experiment, perhaps we might support defining pairs of package and artifact repos with a new syntax like so:

# syntax is PKGREPO,ARTREPO:[PKGREPO,ARTREPO]*
# note that we're implying here that there's a "filesystem" type pkg repo as well as a "filesystem" artifact repo, and when used
# in conjunction, result in a disk structure that matches current rez behaviour
#
]$ export REZ_PACKAGES_PATH=filesystem@~/packages,filesystem@~/packages:postgres@my-db-uri,s3@my-s3-uri

Runtime Engine

The runtime engine is responsible for taking the context, package and artifact repos, and a shell implementation, and using these to construct a configured runtime that's consumable by the user. Note that the shell is required, because part of the process of generating a runtime always involves taking packages' commands and converting them into equivalent shell code, which when sourced, configures the runtime.

Different implementations might include:

• A standard disk-based engine that mimics current behaviour. It simply expands references to package payloads (such as {root} in commands funcs) to the corresponding disk locations as resolved by the appropriate artifact repo.
• An overlayfs-based engine.
• A symlink-based engine that mimics an overlaid filesystem.
• A container-based engine.

You can see how, due to decoupling of artifact repo and runtime engine, we might have an overlayfs-based runtime that's using packages that have been downloaded from S3, or were already available on local disk, or were some combination of the two.

One interesting detail is how package payloads might be combined/overlaid in runtimes which support that. An idea I've had for some time is a new "anchor" feature, where packages can define generic named "anchor points" within their payloads. Two packages with anchor points of the same name, will have that part of their payloads overlaid in the same location. So then it would be common for python packages to define a "python" anchor at {root}/python. Thus the various package commands like env.PYTHONPATH.append('{root}/python') would collapse into a single path appended to PYTHONPATH. This approach would give us a way to express how packages should be combined in a generic way (not specific to a runtime engine implementation), whilst at the same time not imposing any kind of strict schema on a package's payload structure.

[instinct-vfx]

I like where this is going. I need a second read and some thinking before further commenting. Thanks for starting this!

[manuelkoester]

Thanks for this proposal, I also did couple of reading passes and I hope I understood most of it correctly. I still have to do quite a bit of additional readup on overlayfs and the proposed runtime engine. Some of which whooshed right over my head.
But I love the idea, it opens up lots of possibilities.

I think we had a discussion about a similar topic on the previous slack server; using perforce as a package repository and basically using it in the same way as the proposed artifact repository. I can't find the thread anymore unfortunately, but I think especially studios that do not or cannot rely on shared network drives would really benefit here, too.

While this proposal with its examples probably mostly focuses on cloud based animation/VFX workflows, this would also really help game studios, as usually their data storage is not on a filesystem, but rather on perforce. This proposal would help to then also store packages on perforce and not have to rely on additional systems and hardware that have to be maintained.

If I understood it correctly, another interesting factor for me would be the difference/similarity between cached packages and packages downloaded from the artifact repository.

[ColinKennedy]

Artifact Repository - could you clarify what contents are allowed within it?

I have some general questions about the idea of an artifact repository and its potential for hosting common Rez packages which others have already made. The most common criticisms I see about Rez so far is that there's no central place to see what packages others have made (whether they be a package for gcc or even just custom tools). As a result, any non-rez-pip-installable package needs to be made by hand, which is a high opportunity cost. Do you see the artifact repository as a way of bringing people together and making packages more discoverable?

From your description, I understood the artifact repository is a server which hosts installed Rez packages, no source/developer packages. So users would be able to see potentially versions of created gcc but if the package has some sort of bug that they need to fix, they'd not necessarily have access to the source/developer Rez package which generated it. Did I understand that correctly? Do you see the artifact repository as a solution for sources as well?

Can we contextualize this discussion thread to REP-002: Recipe Repositories?

In #673, there's a description of installing third-party packages and its dependencies locally, using a central "rez-recipes" git repository. While I realize REP-002 describes its work in an entirely "local-filesystem only" approach, I see some parallels in the "install a package and its dependencies from public rez package repositories." part of REP-002 and the "then a "localization" repo unzips them onto local disk on demand." portion of the artifact repository

Re: Anchors

One interesting detail is how package payloads might be combined/overlaid in runtimes which support that. An idea I've had for some time is a new "anchor" feature...Thus the various package commands like env.PYTHONPATH.append('{root}/python') would collapse into a single path appended to PYTHONPATH

Just leaving here this would be a big win :) . Is it possible that the anchor / runtime engine can be built as a parallel effort with or without the artifact repository? Or are the two features inseparable?

[Minkiu]

I'd like more insight on the artifact repo as well, currently where I work we have the packages repo:
/path/to/nfs/rez_packages

and the "artifacts" repo:
/path/to/nfs/rez_resources

where we store pre-built binaries for most non-in-house software, so then in the package.py we do:

# -*- coding: utf-8 -*-

name = 'blender'

version = '3.1.0'

description = 'Blender is a free and open-source 3D computer graphics software. https://www.blender.org'

variants = [['platform-linux']]

build_command = ''


def commands():
    env.PATH.append('/path/to/nfs/rez_resources/blender/blender-{0}-linux-x64'.format(this.version))

If I understood correctly, the artifact repository would allow us to not have to hardcode the /path/to/nfs/rez_resources/ since it would be able to, upon request, return the path to the payload or to the actual payload that then when building get's unpacked/unzipped?

[instinct-vfx]

Could you share why you took that approach? That eliminates use of payload caching, doesn't it?

[Minkiu]

It's what I inherited when joining, and I've grown fond of it, so don't have a real answer as to "why", and last I check we are not doing any payload caching (assuming you mean memcached server)... maybe that's why they wen this route? To speed things up?

[instinct-vfx]

memcached is "the other" caching. It caches package definitions, repository and resolves (to reduce file hits and increase performance).
Payload caching caches the payload of packages locally (so a package marked re-locatable is copied locally to a users machine on first resolve). That improves loading performance as things like Maya tend to be quite slow to load from the network). (See here for more info: https://github.com/nerdvegas/rez/wiki/Managing-Packages#package-caching)

[Minkiu]

I see, thanks for clarifying, yeah we are not doing that, and so far loading from /path/to/nfs server has been fast enough to not warrant local caching, but we might have to in the future... thanks for pointing out.

[JeanChristopheMorinPerso]

I took some time to read your proposal. I think it really well summarizes the big picture of the multiple discussions we've (the community in general) been having in our different communication channels and it's also more or less what I had in mind.

Here is some general notes/questions (in non specific other):

• In the "Current System" section, I would add a point that a package repository contains both package definitions and package payloads and that they have to live in the same package repository (they cannot be separated). That would help to describe the problem even more.
• In the "Proposed System" section, I would add a sub-section for to describe the changes planned for package repository, to make it clear that in the new architecture, the package repository would only contain package definitions (metadata).
• In the "Artifact Repository" section, I would clarify that it really is just a storage of artifacts resulting from a build.
• Which component would be responsible of fetching artifacts? If I understand your post, it would be the runtime enginre? Did I undertand this correctly? For filesystem artifact repositories it's not a big deal (since it's probably just going to be a no-op) but for anything other than filesystem, it's going to be kind of a big deal.

As for the runtime engine, I have a feeling that it's going to be the most contentious part... For example:

Note that the shell is required, because part of the process of generating a runtime always involves taking packages' commands and converting them into equivalent shell code, which when sourced, configures the runtime.

Someone might argue that if the runtime was a docker like solution, then shells might not necessarily make sense. Someone might want to have it's docker image completely baked (in the sense that when executing it, no scripts/entrypoint should be executed).

[nerdvegas]

I'm not sure about shells being optional. For that to be the case, you'd also have to include a caveat to the package author that their commands might be ignored, and that really seems to negate a fairly fundamental rez concept. But, maybe there will be edge cases. I think we're getting a bit too far into the weeds here :)

[nerdvegas]

[In the "Current System" section, I would add a point that a package repository contains both package definitions...]

This is kind of true but not quite, but I agree it deserves clarification. You can implement a package repo that stores the payloads elsewhere - but the filesystem impl happens not to. I'll add docs to this effect.

[Which component would be responsible of fetching artifacts]

Yes I think that the runtime would use the artifact repo (plugin) to fetch an artifact, given a unique variant key. This would be a standard function in the plugin interface, like copy_artifact_to_disk or something. That then raises the question though, where do they end up on disk? And that's why I was talking about the possibility of chaining artifact repos together. So for eg, you might have an S3 repo chained onto a disk repo that just persists all of the downloaded artifacts into some standard structure like <pkgname/version/variant-hash>; alternatively, or as well as (another link in the chain), you might have a repo that performs the local package caching (which would replace the builtin package caching mechanism we currently have). Filesystem-based repos might have a function "get_artifact_disk_location" which returns an artifact's location; non-disk-based repos would return None, which should prompt some chained repo to then perform the copy to its desired location, and it's probably gonna do this on demand and in respones to a call to its get_artifact_disk_location function.

Of course, all of the above is complicated by the idea that an artifact repo might serve up images rather than payloads to go to disk, and so we'll have to figure out how to structure the interface to take this into account. This is exactly the kind of conversation(s) I want to kick off!

[JeanChristopheMorinPerso]

In the "Artifact Repository", you say:

The artifact repository is able to take package definitions from a package repository, and resolve them into some local consumable resource

I think it would simplify things if the concept of repository/store and the concept of plugin are broken in two parts. For me, artifact repository means a place where artifacts are stored. An artifact repository plugin is what would interface between rez and the runtime engine. In other words, saying "The artifact repository is able to take package definitions from a package repository" implies that the store itself would know about rez. I'm pretty sure it's not what you intended, but I might be wrong...

Suggestion:

An artifact repository stores the artifacts produced by a build. Each repository type (filesystem, S3, GCS, Azure Blob, HDFS, etc) would come with a plugin that would take package definitions from a package repository, and resolve them into some local consumable resource.

Also, which component would take care of "packaging" (make a zip, tar, etc) package payloads? Would it be the artifact repository plugins?

[nerdvegas]

You're correct, but yes in this doc when I'm talking about these components I'm always talking about the plugins specifically. Note the green "artifact repository" in the diagram, and the note that "Green denotes extensible (plugin-based) rez components".

As far as which component packages up builds, this is deliberately not included here. I want to talk only about the core part of the system first - that is, everything to do with consuming package resources, not creating them. I think it generally helps focus the conversation. But to your point, I suspect that it would be the artifact repo plugin that would play a role in this.

Generally I would think of the artifact repo (plugin! always plugin) as the thing that stores the artifacts, but that is also able to resolve a package definition into its equivalent artifact repo location. So for example, a disk-based variant definition stored at (foo/1.2.3, with specific variant hash) would resolve to, say, an S3 bucket/object name in an S3-based artifact repo. The interesting part of that is whether the artifact repo (plugin!) needs to be aware of the type of package repo, in order to be able to resolve the artifact location. That might be painful, and it seems more tightly coupled than we'd like, so I suspect it may make sense to resolve artifact location based purely on the unique variant key (eg, package name + version + variant hash let's say). If we have different artifact repos that store the same version of the same package, that's fine - these repos would just be configured differently (for eg, perhaps the S3 artifact repo bases the bucket name on the "location" of the package repo, which for filesystem types, is the disk path of the repo).

[sharpencrag]

I am new to the concept of union mount systems like overlayfs, but based on my limited understanding, there really isn't equivalent functionality available in Windows (except maybe symbolic linking?). Is support for Windows something that's going to be an assumed goal for the project going forward? If so, how do we approach this modularity roadmap with Windows in mind? If not, does this fundamental change in structure preclude support for Windows?

[nerdvegas]

An overlayfs-based artifact repo is just one example of a plugin. Rez is
cross-platform, but I don't think there's a requirement that every plugin
be cross platform, only that there's reasonable amount of support across
all platforms to be able to use rez in a useful way.

So, if this meant that there's a time where an overlaid runtime is
available on linux but not on windows (but the standard non-overlaid
runtime is available on both), I think that's reasonable. We shouldn't make
cross-platform support the barrier to adding new functionality when it
comes to plugins, imo. On the flipside, we might also see Windows-specific
plugins (runtimes etc), and that would be fine also (in fact, there are
already windows-specific shell plugins, so in a sense we're already there).

hth
A