-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: integration of argoexec rock #10
Merged
i-chvets
merged 9 commits into
kf-4451-feat-workflows
from
kf-2351-feat-integrate-argo-exec-rock
Oct 4, 2023
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
606d9db
feat: integration of argoexec rock
ea929ff
fix: re-arranged test file
9703a1f
feat: added requirements files
d4baf6f
fix: removed integration env
4c948b2
fix: updated according to comments
998c54d
fix: added readme and cleanup tox.ini
6830859
fix: set proper version in rockcraft
5438e4b
fix: typo
49f0f7e
fix: removed not needed dependencies
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,20 @@ | ||
# argo-workflows-rocks | ||
ROCKs for Argoi Workflows | ||
# ROCKs for Argo Workflows | ||
|
||
This repository contains collection of ROCKs for Argo Workflows. | ||
|
||
- argocli/ contains Rockcraft project for Argo Workflows CLI (TO BE DEPRECATED?) | ||
- argoexec/ contains Rockcraft project for Argo Workflows executor. | ||
- workflow-controller/ contains Rockcraft project for Argo Workflows controller. | ||
|
||
To build ROCK execute the following command in correspomnding directory: | ||
|
||
``` | ||
rockcraft pack | ||
``` | ||
|
||
To run sanity tests on the ROCK execute the following command in correspomnding directory: | ||
|
||
``` | ||
tox -e sanity | ||
``` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
charmed_kubeflow_chisme | ||
pytest |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
# | ||
# This file is autogenerated by pip-compile with Python 3.8 | ||
# by the following command: | ||
# | ||
# pip-compile requirements-sanity.in | ||
# | ||
anyio==3.7.1 | ||
# via httpcore | ||
attrs==23.1.0 | ||
# via jsonschema | ||
certifi==2023.7.22 | ||
# via | ||
# httpcore | ||
# httpx | ||
# requests | ||
charmed-kubeflow-chisme==0.2.0 | ||
# via -r requirements-sanity.in | ||
charset-normalizer==3.2.0 | ||
# via requests | ||
deepdiff==6.2.1 | ||
# via charmed-kubeflow-chisme | ||
exceptiongroup==1.1.3 | ||
# via | ||
# anyio | ||
# pytest | ||
h11==0.14.0 | ||
# via httpcore | ||
httpcore==0.17.3 | ||
# via httpx | ||
httpx==0.24.1 | ||
# via lightkube | ||
idna==3.4 | ||
# via | ||
# anyio | ||
# httpx | ||
# requests | ||
importlib-resources==6.0.1 | ||
# via jsonschema | ||
iniconfig==2.0.0 | ||
# via pytest | ||
jinja2==3.1.2 | ||
# via charmed-kubeflow-chisme | ||
jsonschema==4.17.3 | ||
# via serialized-data-interface | ||
lightkube==0.14.0 | ||
# via charmed-kubeflow-chisme | ||
lightkube-models==1.27.1.4 | ||
# via lightkube | ||
markupsafe==2.1.3 | ||
# via jinja2 | ||
ops==2.5.1 | ||
# via | ||
# charmed-kubeflow-chisme | ||
# serialized-data-interface | ||
ordered-set==4.1.0 | ||
# via deepdiff | ||
packaging==23.1 | ||
# via pytest | ||
pkgutil-resolve-name==1.3.10 | ||
# via jsonschema | ||
pluggy==1.2.0 | ||
# via pytest | ||
pyrsistent==0.19.3 | ||
# via jsonschema | ||
pytest==7.4.0 | ||
# via -r requirements-sanity.in | ||
pyyaml==6.0.1 | ||
# via | ||
# lightkube | ||
# ops | ||
# serialized-data-interface | ||
requests==2.31.0 | ||
# via serialized-data-interface | ||
ruamel-yaml==0.17.32 | ||
# via charmed-kubeflow-chisme | ||
ruamel-yaml-clib==0.2.7 | ||
# via ruamel-yaml | ||
serialized-data-interface==0.7.0 | ||
# via charmed-kubeflow-chisme | ||
sniffio==1.3.0 | ||
# via | ||
# anyio | ||
# httpcore | ||
# httpx | ||
tenacity==8.2.3 | ||
# via charmed-kubeflow-chisme | ||
tomli==2.0.1 | ||
# via pytest | ||
urllib3==1.26.16 | ||
# via requests | ||
websocket-client==1.6.2 | ||
# via ops | ||
zipp==3.16.2 | ||
# via importlib-resources |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
# Copyright 2023 Canonical Ltd. | ||
# See LICENSE file for licensing details. | ||
# | ||
# Tests for required artifacts to be present in the ROCK image. | ||
# | ||
|
||
from charmed_kubeflow_chisme.rock import CheckRock | ||
from pathlib import Path | ||
|
||
import os | ||
import logging | ||
import random | ||
import pytest | ||
import string | ||
import subprocess | ||
import yaml | ||
from pytest_operator.plugin import OpsTest | ||
|
||
@pytest.fixture() | ||
def rock_test_env(): | ||
"""Yelds random docker container name, then cleans it up after.""" | ||
container_name = "".join([str(i) for i in random.choices(string.ascii_lowercase, k=8)]) | ||
yield container_name | ||
|
||
try: | ||
subprocess.run(["docker", "rm", container_name], check=True, capture_output=True) | ||
except Exception as error: | ||
print(f"Failed to remove Docker container {container_name} error: {str(error)}") | ||
|
||
|
||
@pytest.mark.abort_on_fail | ||
def test_rock_artefacts(ops_test: OpsTest, rock_test_env): | ||
"""Test that all artefacts are present in the ROCK: binaries.""" | ||
check_rock = CheckRock("rockcraft.yaml") | ||
container_name = rock_test_env | ||
LOCAL_ROCK_IMAGE = f"{check_rock.get_image_name()}:{check_rock.get_version()}" | ||
|
||
# verify that all artifacts are in correct locations | ||
try: | ||
subprocess.run( | ||
["docker", "run", "--name", container_name, LOCAL_ROCK_IMAGE, "exec", "pebble", "ls", "/bin/argoexec"], | ||
check=True | ||
) | ||
except subprocess.CalledProcessError as error: | ||
print(f"Failed to execute docker run for container {container_name} error: {str(error)}") | ||
assert 0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# Copyright 2023 Canonical Ltd. | ||
# See LICENSE file for licensing details. | ||
[tox] | ||
skipsdist = True | ||
skip_missing_interpreters = True | ||
|
||
[testenv] | ||
setenv = | ||
PYTHONPATH={toxinidir} | ||
PYTHONBREAKPOINT=ipdb.set_trace | ||
CHARM_REPO=https://github.com/canonical/argo-operators.git | ||
CHARM_BRANCH=main | ||
LOCAL_CHARM_DIR=charm_repo | ||
|
||
[testenv:update-requirements] | ||
allowlist_externals = | ||
pip-compile | ||
commands = | ||
pip-compile requirements-sanity.in | ||
deps = | ||
pip-tools | ||
description = Update requirements files by executing pip-compile on all requirements*.in files, including those in subdirs. | ||
|
||
[testenv:sanity] | ||
passenv = * | ||
allowlist_externals = | ||
bash | ||
deps = | ||
-r requirements-sanity.txt | ||
commands = | ||
bash -c 'NAME=$(yq eval .name rockcraft.yaml) && \ | ||
VERSION=$(yq eval .version rockcraft.yaml) && \ | ||
ARCH=$(yq eval ".platforms | keys" rockcraft.yaml | awk -F " " '\''{ print $2 }'\'') && \ | ||
ROCK="$\{NAME\}_$\{VERSION\}_$\{ARCH\}" && \ | ||
sudo skopeo --insecure-policy copy oci-archive:$ROCK.rock docker-daemon:$ROCK:$VERSION && \ | ||
docker save $ROCK > $ROCK.tar' | ||
# run rock tests | ||
pytest -v --tb native --show-capture=all --log-cli-level=INFO {posargs} {toxinidir}/tests |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a docker Python SDK, maybe you want to explore that option
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue is the size of the dependencies needed for testing. Last time we introduced Chisme for testing and it was rejected, because it was a large dependency. IMO the same rationale applies to this Python Docker SDK.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that rationale applies here, as this is in fact, a Python script, and not a command that runs inside a tox file. It's up to you if you want to use the SDK, it was a suggestion to make this script cleaner.