Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mount the dedicated storage for each function #1408

Merged
merged 6 commits into from
Jul 26, 2024
Merged

mount the dedicated storage for each function #1408

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7c89c88
add mount /function_data space
akihikokuroda Jul 15, 2024
1e97109
add /function_data space
akihikokuroda Jul 15, 2024
fad981f
update file function with provider
akihikokuroda Jul 18, 2024
bcba7b7
client update
akihikokuroda Jul 18, 2024
c273ed8
adding test data
akihikokuroda Jul 18, 2024
4a80c3e
fix lint errors
akihikokuroda Jul 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions charts/qiskit-serverless/charts/gateway/templates/rayclustertemplate.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,9 @@ data:
- mountPath: /data
name: user-storage
subPath: {{`{{ user_id }}`}}
- mountPath: /function_data
name: user-storage
Tansito marked this conversation as resolved.
Show resolved Hide resolved
subPath: {{`{{ function_data }}`}}
env:
# Environment variables for Ray TLS authentication.
# See https://docs.ray.io/en/latest/ray-core/configure.html#tls-authentication for more details.
Expand Down Expand Up @@ -352,6 +355,9 @@ data:
- mountPath: /data
name: user-storage
subPath: {{`{{ user_id }}`}}
- mountPath: /function_data
name: user-storage
subPath: {{`{{ function_data }}`}}
Comment on lines +358 to +360
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to mount this with any kind of restricted permissions? In theory, and correct me if I'm wrong here, whoever is running the function (user or provider) would be able to access this location via the container...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that with the current custom image function design, the only the code in the custom image is executed in the Ray node. No user code is sent or executed in the Ray node so only the function code read/write to this directory.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Users could in theory find a way around that restriction (if we misimplement something, CVS / exploits, etc.) so it's a risk, we'd just need to determine if it's an acceptable one

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we just add the path when a function is from a provider? That could solve part of the problem. And correct me if I'm wrong here @akihikokuroda but all the providers will write here so they could potentially see files from other providers with this approach. What is the problem to use the name of provider as sub-path instead of function_data?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My idea was that in that way we could attach the specific provider path only when it's a function from the specific provider (but I don't know if there are limitations in this approach).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought function-data was the provider space and data was for users ? since users would already go to data on their own functions

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the provider, it may be useful to have both while they develop the function because it is the same environment when the function is executed by the user.

Copy link
Member

@Tansito Tansito Jul 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought function-data was the provider space and data was for users ?

yep

since users would already go to data on their own functions

basically that's my comment, for functions that come from the user we are adding /function_data too. It's true that /function_data and /data are going to point to user.username path so not a bid deal but probably we can just avoid add /function_data for user functions.

BTW, another answer that I would understand is: "well, we don't overcomplicate more the template this way" and I would agree. Yep.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Django template in helm template is very tricky :-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hahaha I agree, I agree. I'm just reviewing the rest of the code, Aki and I will update my review 👍

{{- if .Values.useCertManager }}
- mountPath: /tmp/tls
name: cert-tls
Expand Down
6 changes: 5 additions & 1 deletion gateway/api/ray.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ def submit_job(job: Job) -> Job:
return job


def create_ray_cluster(
def create_ray_cluster( # pylint: disable=too-many-branches
job: Job,
cluster_name: Optional[str] = None,
cluster_data: Optional[str] = None,
Expand Down Expand Up @@ -250,14 +250,18 @@ def create_ray_cluster(
node_image = settings.RAY_NODE_IMAGE

# if user specified image use specified image
function_data = user.username
if job.program.image is not None:
node_image = job.program.image
if job.program.provider.name:
function_data = job.program.provider.name

cluster = get_template("rayclustertemplate.yaml")
manifest = cluster.render(
{
"cluster_name": cluster_name,
"user_id": user.username,
"function_data": function_data,
"node_image": node_image,
"workers": job_config.workers,
"min_workers": job_config.min_workers,
Expand Down
Loading