Design a vineyard csi driver for workloads based on kubernetes volumes. #1533

dashanji · 2023-08-22T02:29:58Z

What do these changes do?

Design the vineyard csi driver framework including the driver logic and deployment manifests.
Create the soft link of vineyard socket for every PersistentVolumeClaim.
Map every vineyard object to a PersistentVolumeClaim/PersistentVolume.
Introduce https://github.com/kubernetes-csi/csi-test to validate the idempotence of vineyard csi driver API.

Related issue number

Fixes parts of #1528

netlify · 2023-08-22T02:32:12Z

✅ Deploy Preview for v6d ready!

Name	Link
🔨 Latest commit	`7d829cf`
🔍 Latest deploy log	https://app.netlify.com/sites/v6d/deploys/650050a2a0a1c400087eb28b
😎 Deploy Preview	https://deploy-preview-1533--v6d.netlify.app/notes/references/crds
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

sighingnow

Is there any simple readme/docs/or even internal docs that can tell how to getting started with the csi-driver?

I would like to take a try.

sighingnow · 2023-09-04T06:32:28Z

k8s/Dockerfile

 # Use distroless as minimal base image to package the manager binary
 # Refer to https://github.com/GoogleContainerTools/distroless for more details
-FROM gcr.io/distroless/static:nonroot
+FROM ${BASE_IMAGE}


Do we need other base images?

Yes, the image gcr.io/distroless/static:nonroot doesn't have the mount command and the csi driver image need it.

Then you should use gcr.io/distroless/base:debug which has busybox installed and there would be a mount command available.

NEVER use your build environments (e.g., golang image) as the base to release your artifacts.

sighingnow · 2023-09-04T08:36:22Z

k8s/examples/vineyard-csidriver/preprocess/preprocess.py

+from sklearn.metrics import mean_squared_log_error, mean_absolute_error
+
+def path_to_vineyard_name(path):
+   return path.replace("/", "_")


No need for such hacks as name can have '/' since #1550.

Great patch!

sighingnow

Some first glance experiences:

The deployment of vineyardd and csi-driver should be separated, i.e., we first have vineyardd, the we want a volume view for those objects, we ask the vineyard-operator to launch/deploy csi agents on nodes (specify the namespace and instance name of Vineyardd CRD that the csi driver will manage).
the owner-reference of csi driver deployment should be the corresponding vineyard cluster
the owner-reference of those pv/pvc should be the csi-driver deployment

About the PUT/GET api, I suggest something like

vineyard.read('/path...managed...by...csi driver')
vineyard.write('/path..........', object)

where the corresponding vineyard IPC socket will be resolved from the abstract file inside the volume.

sighingnow

I have added some minor comments.

sighingnow · 2023-09-05T12:35:59Z

k8s/cmd/commands/csidriver/pkg/controller.go

@@ -0,0 +1,271 @@
+/*


Move the pkg/ directory to outside pkg/csi/.

sighingnow · 2023-09-05T12:37:05Z

k8s/cmd/commands/csidriver/pkg/driver.go

+			return s[0], s[1], nil
+		}
+	}
+	return "", "", fmt.Errorf("Invalid endpoint: %v", ep)


Avoid using fmt.Errorf, using github.com/pkg/errors.Errorf instead to track the traceback of errors for debugging in the future.

sighingnow · 2023-09-05T12:38:39Z

k8s/cmd/main.go

@@ -27,6 +27,7 @@ import (
 	gosdklog "github.com/v6d-io/v6d/go/vineyard/pkg/common/log"
 	"github.com/v6d-io/v6d/k8s/cmd/commands/client"
 	"github.com/v6d-io/v6d/k8s/cmd/commands/create"
+	"github.com/v6d-io/v6d/k8s/cmd/commands/csidriver"


I prefer csi over csidriver.

sighingnow · 2023-09-05T12:40:19Z

k8s/csidriver/deploy/csi-nodes-rbac.yaml

+roleRef:
+  kind: ClusterRole
+  name: vineyard-csi-nodes
+  apiGroup: rbac.authorization.k8s.io


Can we have a global service account vineyard (or maybe vineyard-operator) to cover all permissions we require and use it anywhere when a service account is needed?

for simplifying the management.

dashanji · 2023-09-06T02:44:21Z

Some first glance experiences:

The deployment of vineyardd and csi-driver should be separated, i.e., we first have vineyardd, the we want a volume view for those objects, we ask the vineyard-operator to launch/deploy csi agents on nodes (specify the namespace and instance name of Vineyardd CRD that the csi driver will manage).

the owner-reference of csi driver deployment should be the corresponding vineyard cluster

the owner-reference of those pv/pvc should be the csi-driver deployment

About the PUT/GET api, I suggest something like

vineyard.read('/path...managed...by...csi driver')

vineyard.write('/path..........', object)

where the corresponding vineyard IPC socket will be resolved from the abstract file inside the volume.

Thanks for the advice! It really makes sense to me.

sighingnow · 2023-09-15T09:38:09Z

k8s/examples/vineyard-csidriver/README.md

+
+### Write time
+
+| data_multiplier | without vineyard | with vineyard |


data_multipiler -> data scale, and use data size (e.g., xxx Mi) in this table.

sighingnow · 2023-09-15T09:39:17Z

k8s/examples/vineyard-csidriver/test/test.py

+import vineyard
+
+def test_model():
+    os.system('echo 3 > /proc/sys/vm/drop_caches')


If you want drop page cache, you first need a sync command, then echo to ....

sighingnow · 2023-09-15T09:39:30Z

k8s/examples/vineyard-csidriver/test/test.py

+
+def test_model():
+    os.system('echo 3 > /proc/sys/vm/drop_caches')
+    enable_vineyard = os.environ.get('ENABLE_VINEYARD', False)


ENABLE_VINEYARD -> WITH_VINEYARD.

sighingnow · 2023-09-15T09:51:10Z

k8s/Dockerfile

 # Use distroless as minimal base image to package the manager binary
 # Refer to https://github.com/GoogleContainerTools/distroless for more details
-FROM gcr.io/distroless/static:nonroot
+FROM ${BASE_IMAGE}


Then you should use gcr.io/distroless/base:debug which has busybox installed and there would be a mount command available.

NEVER use your build environments (e.g., golang image) as the base to release your artifacts.

sighingnow · 2023-09-18T03:12:13Z

k8s/Makefile

 		-t $(VINEYARD_CSI_IMAGE); \
 	else \
 		docker build -f k8s/Dockerfile . \
-		--build-arg BASE_IMAGE=golang:1.19-buster \
+		--build-arg BASE_IMAGE=gcr.io/distroless/base:debug \


You can directly replace the FROM statement in the dockerfile to avoid such a layer of complexity.

sighingnow · 2023-09-18T03:12:51Z

k8s/config/crd/bases/k8s.v6d.io_csidrivers.yaml

@@ -38,6 +38,9 @@ spec:
                      type: string
                  type: object
                type: array
+              enableDebugLog:
+                default: false
+                type: boolean


Can you let user set the log-level (as we previously discussed)?

Okay, it looks like users only can set verbose or non-verbose. Do you think EnableVerbose is more meaningful?

sighingnow · 2023-09-18T03:14:30Z

k8s/cmd/commands/csi/csi.go

+
+var csiExample = util.Examples(`
+	# start the csidriver with the specific endpoint and node id
+	vineyardctl csidriver --endpoint=unix:///csi/csi.sock --nodeid=csinode1`)


What the nodeid means?

sighingnow · 2023-09-18T03:16:12Z

k8s/csitest.Dockerfile

+
+USER root
+
+CMD ["go", "test", "-run", "./..."]


Why we need a separate docker file for go test?

sighingnow · 2023-09-18T03:25:24Z

k8s/examples/vineyard-csidriver/README.md

+
+Based on the above results, we can find that the read time of vineyard is nearly a constant, which is not affected by the data scale. The reason is that the data is stored in the shared memory of vineyard cluster, so it's actually a pointer copy operation.
+
+As a result, we can find that with vineyard, the argo workflow duration of the pipeline is reduced by 10%~20% and the actual execution time of the pipeline is reduced by about 30%.


Put the readme to the tutorial directory , same as what you have done for another "e xample" under the k8s/ directory:

https://github.com/v6d-io/v6d/tree/main/docs/tutorials/kubernetes

examples. * Add a new CRD CSIDriver to manage the deployment and status of the vineyard csi driver. * Add the CSIDriver benchmark test and relevant doc. * Introduce the kubernetes csi test to enhancing the robustness of the vineyard CSI Driver. * Generate the doc for CSIDriver CRD. Signed-off-by: Ye Cao <[email protected]>

* Update the golang-1.19 to gcr.io/distroless/base:debug as csidriver's base image. * Improve the csidriver example readme. Signed-off-by: Ye Cao <[email protected]>

* Add the deploy/delete csidriver API in the vineyardctl. * Update the CRD API doc and vineyardctl doc. * Update the helm chart. * Update the base image from gcr.io/distroless/static:nonroot to gcr.io/distroless/base:debug. Signed-off-by: Ye Cao <[email protected]>

dashanji · 2023-09-18T08:51:54Z

The new commit mainly contains 3 parts:

Improve the vineyard csi driver doc and move it to the kubernetes tutorials.
Add the deploy/delete csidriver API in the vineyardctl.
Update the helm chart.

@sighingnow Could you please take another look? Thanks.

sighingnow · 2023-09-18T11:03:04Z

docs/tutorials/kubernetes.rst

@@ -9,6 +9,7 @@ Vineyard on Kubernetes
   ./kubernetes/using-vineyard-operator.rst
   ./kubernetes/ml-pipeline-mars-pytorch.rst
   ./kubernetes/data-sharing-with-vineyard-on-kubernetes.rst
+   ./kubernetes/speed-up-pipelines-based-on-volumes-with-vineyard-csi-driver.rst


speed-up-pipelines-based-on-volumes-with-vineyard-csi-driver

Efficient data sharing in Kubeflow with Vineyard CSI driver.

sighingnow · 2023-09-18T11:04:26Z

k8s/Makefile

@@ -223,9 +224,11 @@ vendor:
 docker-build:
 	cd .. && \
 	if docker build --help | grep -q load; then \
-		docker build --load -f k8s/Dockerfile . -t $(IMG); \
+		docker build --load -f k8s/Dockerfile . \
+		-t $(IMG); \


Keep this part untouched. No need to add a newline here.

Signed-off-by: Ye Cao <[email protected]>

dashanji force-pushed the design-vineyard-csi-driver branch 2 times, most recently from ea420b5 to 6958569 Compare August 24, 2023 07:55

sighingnow reviewed Sep 4, 2023

View reviewed changes

dashanji force-pushed the design-vineyard-csi-driver branch 3 times, most recently from 2e1c8bc to a85597a Compare September 5, 2023 11:54

dashanji changed the title ~~[WIP] Design a vineyard csi driver for workloads based on kubernetes volumes.~~ Design a vineyard csi driver for workloads based on kubernetes volumes. Sep 5, 2023

sighingnow reviewed Sep 5, 2023

View reviewed changes

dashanji mentioned this pull request Sep 6, 2023

Python SDK: PUT/GET API for vineyard csi driver #1555

Closed

dashanji force-pushed the design-vineyard-csi-driver branch 7 times, most recently from 769a4fc to 2894f2c Compare September 15, 2023 09:24

sighingnow requested changes Sep 15, 2023

View reviewed changes

dashanji mentioned this pull request Sep 18, 2023

Add the deploy/delete csidriver API in the vineyardctl and update the relevant helm chart #1567

Closed

2 tasks

sighingnow approved these changes Sep 18, 2023

View reviewed changes

sighingnow reviewed Sep 18, 2023

View reviewed changes

k8s/csitest.Dockerfile Outdated

USER root

CMD ["go", "test", "-run", "./..."]

Copy link

Member

sighingnow Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we need a separate docker file for go test?

sighingnow reviewed Sep 18, 2023

View reviewed changes

sighingnow approved these changes Sep 18, 2023

View reviewed changes

dashanji added 2 commits September 18, 2023 16:38

* Move the csi/pkg to top-level pkg.

226f839

* Update the golang-1.19 to gcr.io/distroless/base:debug as csidriver's base image. * Improve the csidriver example readme. Signed-off-by: Ye Cao <[email protected]>

dashanji force-pushed the design-vineyard-csi-driver branch from 05991dd to 61f124d Compare September 18, 2023 08:39

dashanji force-pushed the design-vineyard-csi-driver branch from 61f124d to e6fd19c Compare September 18, 2023 08:51

sighingnow reviewed Sep 18, 2023

View reviewed changes

Update the title of the vineyard csi driver tutorial.

044c84a

Signed-off-by: Ye Cao <[email protected]>

dashanji force-pushed the design-vineyard-csi-driver branch from 904092a to 044c84a Compare September 18, 2023 11:23

sighingnow merged commit e5ca1c7 into v6d-io:main Sep 19, 2023
19 of 23 checks passed

dashanji deleted the design-vineyard-csi-driver branch September 19, 2023 05:47

dashanji mentioned this pull request Sep 20, 2023

Design the vineyard CSI driver for better integrating with kubeflow/argo examples #1528

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Design a vineyard csi driver for workloads based on kubernetes volumes. #1533

Design a vineyard csi driver for workloads based on kubernetes volumes. #1533

dashanji commented Aug 22, 2023 •

edited

Loading

netlify bot commented Aug 22, 2023 •

edited

Loading

sighingnow left a comment

sighingnow Sep 4, 2023

dashanji Sep 4, 2023

sighingnow Sep 15, 2023

sighingnow Sep 4, 2023

dashanji Sep 4, 2023

sighingnow left a comment •

edited

Loading

sighingnow left a comment

sighingnow Sep 5, 2023

sighingnow Sep 5, 2023

sighingnow Sep 5, 2023

sighingnow Sep 5, 2023

dashanji commented Sep 6, 2023

sighingnow Sep 15, 2023

dashanji Sep 18, 2023

sighingnow Sep 15, 2023

sighingnow Sep 15, 2023

dashanji Sep 18, 2023

sighingnow Sep 15, 2023

sighingnow Sep 18, 2023

sighingnow Sep 18, 2023

dashanji Sep 18, 2023

sighingnow Sep 18, 2023

sighingnow Sep 18, 2023

sighingnow Sep 18, 2023

dashanji commented Sep 18, 2023

sighingnow Sep 18, 2023

dashanji Sep 18, 2023

sighingnow Sep 18, 2023

dashanji Sep 18, 2023


		### Write time

		\| data_multiplier \| without vineyard \| with vineyard \|


		Based on the above results, we can find that the read time of vineyard is nearly a constant, which is not affected by the data scale. The reason is that the data is stored in the shared memory of vineyard cluster, so it's actually a pointer copy operation.

		As a result, we can find that with vineyard, the argo workflow duration of the pipeline is reduced by 10%~20% and the actual execution time of the pipeline is reduced by about 30%.

Design a vineyard csi driver for workloads based on kubernetes volumes. #1533

Design a vineyard csi driver for workloads based on kubernetes volumes. #1533

Conversation

dashanji commented Aug 22, 2023 • edited Loading

What do these changes do?

Related issue number

netlify bot commented Aug 22, 2023 • edited Loading

✅ Deploy Preview for v6d ready!

sighingnow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sighingnow left a comment • edited Loading

Choose a reason for hiding this comment

sighingnow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dashanji commented Sep 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dashanji commented Sep 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dashanji commented Aug 22, 2023 •

edited

Loading

netlify bot commented Aug 22, 2023 •

edited

Loading

sighingnow left a comment •

edited

Loading