| Key | Value |
|---|---|
| Author(s) | Jordan.Brockopp |
| Reviewers | David.May, Emmanuel.Meinen, Kelly.Merrick, David.Vader, Matthew.Fevold |
| Date | August 25th, 2021 |
| Status | Reviewed |
Please provide a summary of the new feature, redesign or refactor:
Currently, Vela has an inconsistent structure of how the "core" repos are setup within the go-vela org.
In this context, a "core" repo includes any repos that require code or dependency changes in order to release the Vela product.
Today, we have the following "core" repos (11 total):
Before attempting to propose any changes to this structure, it's important to provide context and clarity on why this structure exists.
In the early beginnings of Vela, we thought that each individual capability/service/tool that Vela integrates with would get a unique repo.
The naming convention for these repos would have the syntax of pkg-<capability> to denote the code housed in that repo.
To provide an example with the current structure, we'll reference the pkg-runtime repo.
Vela has to integrate with different runtime services or tools in order to execute workloads.
So, the pkg-runtime repo contains all the code necessary to integrate with the different supported runtimes:
Likewise for the pkg-queue repo, Vela has to integrate with different queue services or tools in order to schedule workloads.
And the pkg-queue repo contains all the code necessary to integrate with the different supported queues:
However, this structure has some problems, which are detailed below, that leave some room for improvement.
Please briefly answer the following questions:
- Why is this required?
This is not required so there is a lot of room for discussion on this.
The intention or goal of this proposal is to accomplish the following:
- condense and simplify the "core" repo structure
- improve consistency among the "core" repos
- ease the burden necessary to contribute functionality to the product
- reduce the level of overhead when attempting to publish new releases
- If this is a redesign or refactor, what issues exist in the current implementation?
The first problem with this structure is ensuring consistency, both currently and going forward, among the repos in the go-vela org.
Currently, not all "core" repos follow the pkg-<capability> naming convention. i.e. compiler
Also, some "core" repos have the packages for different capabilities in the repo itself. i.e. server
If we plan to keep the pkg-<capability> structure, those should be pulled out into unique repos following that structure:
- server/database ->
pkg-database - server/secret ->
pkg-secret - server/source ->
pkg-source
The second problem with this structure is the increased burden, especially for new users, trying to contribute functionality.
Some, or most, of the "core" repos have dependencies on one or more repos.
This means that in order to make certain changes to the product, you have to introduce code changes to multiple repos.
To provide an example with the current structure, we'll reference a recent feature that allows you to customize the pipeline type for a repo.
This functionality was added as a part of go-vela/community#326.
To achieve this, a new pipeline_type field was added to the database which required code changes for multiple repos:
This also increases the complexity when attempting to functionally test code changes for new logic.
This is due to needing to modify each repo locally with the code changes for the functionality.
Once that is complete, the application must be rebuilt from the local code changes for each repo.
The third problem with this structure is level of overhead when attempting to publish new releases for the product.
The way the repos are setup today, there is a dependency tree that must be followed in order to properly release the product.
The way the repos must be released is in groups that use the following order:
After a release is completed for the first group, you must update the code dependencies for the second group before proceeding.
This pattern repeats, for each group, until you get to the last group which has dependencies on most of the preceding groups.
With this structure, we simply have to create more releases (one per repo) than we would if we were to condense the repos.
It also adds more complexity when releasing because you have to remember the order and ensure each repo has the latest dependencies.
- Are there any other workarounds, and if so, what are the drawbacks?
N/A
- Are there any related issues? Please provide them below if any exist.
N/A
Please describe your solution to the proposal. This includes, but is not limited to:
- new/updated endpoints or url paths
- new/updated configuration variables (environment, flags, files, etc.)
- performance and user experience tradeoffs
- security concerns or assumptions
- examples or (pseudo) code snippets
The purpose of this proposal is to accomplish the following:
- condense and simplify the "core" repo structure
- improve consistency among the "core" repos
- ease the burden necessary to contribute functionality to the product
- reduce the level of overhead when attempting to publish new releases
To accomplish this, the following changes are proposed:
- Move the compiler repo into the server repo as a nested package
- Accompanying this, would be the deprecation and archiving of the existing compiler repo
- Move part of the mock repo into the server repo as a nested package
- Accompanying this, would be the deprecation and archiving of the existing mock repo
- Move part of the pkg-queue repo into the server repo as a nested package
- Accompanying this, would be the deprecation and archiving of the existing pkg-queue repo
- Move part of the mock repo into the worker repo as a nested package
- Accompanying this, would be the deprecation and archiving of the existing mock repo
- Move the pkg-executor repo into the worker repo as a nested package
- Accompanying this, would be the deprecation and archiving of the existing pkg-executor repo
- Move part of the pkg-queue repo into the worker repo as a nested package
- Accompanying this, would be the deprecation and archiving of the existing pkg-queue repo
- Move the pkg-runtime repo into the worker repo as a nested package
- Accompanying this, would be the deprecation and archiving of the existing pkg-runtime repo
DISCLAIMER:
Consideration was given to moving the types repo into the server and/or worker.
However, it was determined that may involve too much overhead and complicate things.
The core concern for moving types is the chance for causing circular dependencies.
In the end, the new "core" repo structure for the go-vela org would look like (6 total):
1. cli
- compiler: moved -> server
- mock: moved -> server/worker
- pkg-executor: moved -> worker
- pkg-queue: moved -> server/worker
- pkg-runtime: moved -> worker
2. sdk-go
3. server
4. types
5. ui
6. workerThe final directory structure for the server would look like:
server
├── api
├── cmd
+├── compiler
├── database
+├── mock
+├── queue
├── random
├── router
├── secret
├── source
├── util
└── versionThe final directory structure for the worker would look like:
worker
├── api
├── api-spec
├── cmd
+├── executor
+├── mock
+├── queue
├── router
+├── runtime
└── versionPlease briefly answer the following questions:
- Is this something you plan to implement yourself?
Yes
- What's the estimated time to completion?
1 week
Please provide all tasks (gists, issues, pull requests, etc.) completed to implement the design:
Please list any questions you may have:
N/A