IAM <-> Workspace integration for initial workspace provisioning

[achtsnits]

A request for a new workspace can come from different sources:

• One common way is through the operator’s website when a user purchases a subscription plan or resource hours.
• Another possibility is through an email from NoR granting sponsorship for a specific project.
• ...

The request will typically include details like the project name, the subscription/resources granted (such as storage quotas, compute/mem limits or overall hourse, gpu yes/no, ...) and the email address of the designated project owner.

Task-WS1) Extend with email of project owner

Once the platform operator receives such a request, they review and provision it. This is automated by the (configurable) EOEPCA provisioning pipeline, creating

1. the necessary storage resources, such as an S3 bucket,
2. resources on Kubernetes (e.g. with a dedicated Namespace and Resource-Quotas) and
3. the required (configured) deployments for the workspace (e.g. the Workspace UI)

As discussed, the pipeline should also create a new group in Keycloak using the project name and associate the user (whose email was provided in the request as owner) with this new Keycloak group. By convention the Keycloak group and Workspace name should match, the concrete Project Name should be an attribute on the Keycloak group.

Current Assumption: the user has already registered and already exists in the system (will be tracked in separate GitHub Issue TBD)

Task-IAM1) IAM BB to setup properly scoped client allowing the provisioning pipeline to execute these actions, i.e. with ability to list and create groups in Keycloak, associate users within a group, and check if a user already exists in the system
Task-WS2) Workspace BB to automate theses actions within the provisioning pipeline

Access to the workspace and its resources needs to be controlled. Only users who are part of the Keycloak group associated with the project should be allowed to access the deployments to the workspace. To achieve this, the pipeline can automatically create a properly configured Kubernetes Ingress Object (or custom CRD). If necessary the provisioning pipeline can also create additional "resources" (like a policy) on Keycloak side

Task-IAM2) IAM BB to specify what needs to be created
Task-WS3) Workspace BB to implement accordingly

[achtsnits]

We have concluded that Keycloak groups will represent teams and be used to manage the "user association" with workspaces. Note that the association is many-to-many (i.e., workspaces can be accessed by multiple users, and users can belong to multiple workspaces).

Sign-off:

1. The workspace owner’s email should be stored, meaning the email must be provided during provisioning and stored in Workspace Kubernetes manifest
2. A Keycloak group with the workspace name should be automatically created during provisioning.
3. The user (who we assume already exists for now) should be associated with this newly created Keycloak group.
4. Access to workspace API endpoints will only be allowed through the authorized route, with restricted access for others. (Note: This should be purely on IAM BB side via APX-Ingress)
5. Access to the workspace UI (deployed per team) will be restricted to users belonging to the respective team. (Note: The Ingress creation has to be automated in Workspace BB but needs help of IAM BB on what must be created in Keycloak and put into APX Ingress object)

[w-scho]

I generally agree with your comment above.
I assume that there is a 1:1:1 relationship between teams, groups and workspaces, i.e. each workspace has exactly one associated group whose members are the members of the associated team. Of course additional groups may exist for other purposes.

Regarding points 4 and 5:
I assume that it will be necessary to create a separate client for each workspace UI and preconfigure it appropriately. Depending on what turns out to be most suitable, the client could be created through OIDC dynamic client regstration, the Identity API or the Keycloak API. This client would only be used by the route for the authN/authZ process. Workspace instances would still access Keycloak via the workspace-bb client.
I assume that it may be necessary to configure a resource in Keycloak for each workspace API. This is at least the clean way to express separate protection domains, and I am not really sure if more fine-grained protection than on resource level is actually feasible in Keycloak.
If we want to avoid configuring too many resources in Keycloak or generally need more fine-grained authorization on the ingress level (which might be necessary for shared links), we may have to consider adopting the APISIX-OPA plugin (https://apisix.apache.org/docs/apisix/plugins/opa/).

[achtsnits]

thanks for your fast response, comments inline

I generally agree with your comment above.
I assume that there is a 1:1:1 relationship between teams, groups and workspaces, i.e. each workspace has exactly one associated group whose members are the members of the associated team. Of course additional groups may exist for other purposes.

Yes, matches my understanding

Regarding points 4 and 5:
I assume that it will be necessary to create a separate client for each workspace UI and preconfigure it appropriately. Depending on what turns out to be most suitable, the client could be created through OIDC dynamic client regstration, the Identity API or the Keycloak API. This client would only be used by the route for the authN/authZ process. Workspace instances would still access Keycloak via the workspace-bb client.
I assume that it may be necessary to configure a resource in Keycloak for each workspace API. This is at least the clean way to express separate protection domains, and I am not really sure if more fine-grained protection than on resource level is actually feasible in Keycloak.
If we want to avoid configuring too many resources in Keycloak or generally need more fine-grained authorization on the ingress level (which might be necessary for shared links), we may have to consider adopting the APISIX-OPA plugin (https://apisix.apache.org/docs/apisix/plugins/opa/).

I assume the need for multiple Keycloak clients arises from handling different domains, correct? Keycloak doesn’t seem to support subdomain whitelisting with something like *.develop.eoepca.org, does it?

If not, we could explore making these workspace UI deployments accessible under different subpaths within the same domain? That approach should work, right?

I understand the rationale behind point 5, but why do you see the need for dynamic clients also for point 4?

[w-scho]

A single Keycloak client can span multiple domains, but it is probably cleaner to add a client for each new domain than to update an existing client by adding URLs again and again. I tried to specify a base URL with wildcard or no base URL at all, but (as expected) this does not work. Keycloak allows wildcards for the path, but apparently not for the host part of the URL.
For subpaths, no separate clients are needed (though they would be possible). So a single client for all API(s) and another one for all UIs should be sufficient if all UIs share the same DNS name. This also implies that there is no need for dynamic client registration in this case.

[achtsnits]

so regardless which strategy we will follow, we need to not just authenticate the user but also check if he is allowed to access the url based on the user<->group association

lets assume we have userA and userB associated to team ws-team4, what must be setup upfront in keycloak and how does the K8s kind: ApisixRoute will then look like for https://ws-test4.develop.eoepca.org resp. https://develop.eoepca.org/ws-test4?

talking to the team I'm also afraid that the subroute path won't generally work as many applications will expect to be served at root path (for sure we can adapt for workspace-ui but there will be also other applications deployed in future so same problem then...)

can you prepare a setup with keycloak, ?OPA policy if needed?, ApisixRoute please and we will try to automate during workspace provisioning- do you agree and does that make sense to you?

[w-scho]

Let me first check if my understanding is roughly correct:
There is an existing (unprotected) workspace instance at https://ws-test4.develop.eoepca.org. For this instance, I'd create a corresponding ApisixRoute at https://ws-test4.apx.develop.eoepca.org and configure it so that it only allows users who are members of the corresponding Keycloak group. However, it need not enforce any more fine-grained restrictions, at least for now.
I assume that https://develop.eoepca.org/ws-test4 is just the path-pased alternative to the other URL (and not the WS API URL).
You wrote that the path-based approach would require adaptations to the workspace UI and may also cause problems with other applications. So this is obviously not the preferred approach and we should focus on the DNS-based one first.
I'll try to create a reasonable example setup for the DNS-based variant by tomorrow afternoon.

[achtsnits]

Let me first check if my understanding is roughly correct:
There is an existing (unprotected) workspace instance at https://ws-test4.develop.eoepca.org. For this instance, I'd create a corresponding ApisixRoute at https://ws-test4.apx.develop.eoepca.org and configure it so that it only allows users who are members of the corresponding Keycloak group. However, it need not enforce any more fine-grained restrictions, at least for now.

Thanks & yes, correct (https://ws-test4.develop.eoepca.org will just be a new Workspace-UI deployment specifically for team ws-test4)

Let's forget the path based routing here for the Workspace-UI.

Just to clarify - there will be one separate dedicated Workspace-UI Deployment for each team, but there will only be one central Workspace-API Deployment in the cluster - this Workspace-API Deployment is mainly used to retrieve credentials for the team, e.g. to access the buckets (e.g. user triggers Processing BB and this component will impersonate user to get credentials via HTTP GET workspace-api>/workspaces/ and uploads results there - my understanding at the moment at least)

[achtsnits]

to explore for the WorkspaceUI we manually applied

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: ws-test4
  namespace: ws-test4
spec:
  http:
    - name: ws-test4
      match:
        hosts:
          - ws-test4.apx.develop.eoepca.org
        paths:
          - /*
      backends:
        - serviceName: ws-test4
          servicePort: 8080

can you create a user in keycloak and associate it with group ws-test4, then only this user should have access

feel free to adapt accordingly directly in the cluster (it is for testing so not in ArgoCD as it would be reconciled otherwise)

similar to what @rconway configured for the /docs route on the ApisixRoute for the WorkapceAPI we also need to make some routes like https://ws-test4.apx.develop.eoepca.org/share/* (e.g. https://ws-test4.apx.develop.eoepca.org/share/MAfNY4Aw) publicly accessible for the WorkspaceUI

completely different topic: the WorkspaceUI uses WebSockets and they don't seem to work, can you enable that on the ingress controller (or does that need something on the route?), thanks

[w-scho]

According to WebSocket support, I assume that this is the intended solution: https://apisix.apache.org/docs/ingress-controller/concepts/apisix_route/#websocket-proxy
It is basically just setting websocket: true on the subroute level. I adjusted the ws-test4 route accordingly (and also disabled the proxy-rewrite plugin), but I am not able to test it, because I do not know where exactly websockets are used.

[achtsnits]

websockets are working now, thanks for fast mitigation!

[w-scho]

As you may already have noticed, I modified the route and added authentification and authorization through Keycloak.
I left /share/* open, i.e. there is no authN/authZ for this subtree. Nonetheless it does not fully work yet, probably because further paths (e.g. /static/*) need to be left unprotected. You can give me the list of paths to be left unprotected or simply add them yourself.

On Keycloak side, I added the following:

• A client called "ws-test4", with root URL https://ws-test4.apx.develop.eoepca.org and authN and authZ enabled. This client is referenced by the route and the client secret can be found there (but should probably be stored as a secret later)
• A client role called "ws_access" that represents the right to access the workspace UI associated with client "ws-test"
• A group called "ws-test4" with the role "ws_access" on client "ws-test4" assigned
• A user called "ws-test4-user" (with password "ws-test4") who is a member of group "ws-test4"
• A client policy called "Role ws_access Policy" that checks if the user has role "ws_access" on client "ws-test4". The policy is configured as a standard Keycloak policy of type "Role". It could also be implemented in a more generic way in OPA, but then it would still be necessary to configure a Keycloak policy of type "OPA" that references the actual OPA policy. So using the standard Keycloak policy type turned out to be a bit simpler for now.
• In the default permission of client "ws-test4", I replaced the Default Policy with the "Role ws_access Policy".

Altogether, this gives (only) user "ws-test4-user" (and other potential members of group "ws-test4") access to the "ws-test4" workspace. It would be possible to simplify this a bit by omitting the role and letting the policy check the group directly instead, but this would cause too much coupling between user and permission management IMO.

[achtsnits]

via https://iam-auth.develop.eoepca.org/admin/eoepca/console/ I added my own user to group ws-test4 and afterwards I could access https://ws-test4.apx.develop.eoepca.org/, cool thanks - good work, also like/agree on your permissions approach!

• can you please give my user more permissions so I can also inspect clients, roles and similar via https://iam-auth.develop.eoepca.org/admin/eoepca/console/

• so client role "ws_access" is created once (must be part of operator setup) and during workspace provisioning for team X with user A we can assume that the user A exists but we have to create

  • group X
  • user A <-> group X association
  • client X
  • client-policy (checking if role ws_access on client X exists for user)
  • default permission must be changed on client X to ws_access

via our IaC tooling (need to explore now if this all is already supported and possible)

• can you also please add all the necessary permissions to the client "workspace_pipeline" so we can finally automate

I put the current state of ws-test4 apisix route here as reference, we will build on top of that

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: ws-test4
  namespace: ws-test4
spec:
  http:
  - backends:
    - serviceName: ws-test4
      servicePort: 8080
    match:
      hosts:
      - ws-test4.apx.develop.eoepca.org
      paths:
      - /share/*
    name: ws-test4-open
  - backends:
    - serviceName: ws-test4
      servicePort: 8080
    match:
      hosts:
      - ws-test4.apx.develop.eoepca.org
      paths:
      - /*
    name: ws-test4
    plugins:
    - config:
        access_token_in_authorization_header: true
        client_id: ws-test4
        client_secret: Qmp5p...
        discovery: https://iam-auth.apx.develop.eoepca.org/realms/eoepca/.well-known/openid-configuration
      enable: true
      name: openid-connect
    - config:
        client_id: ws-test4
        client_secret: Qmp5p...
        discovery: https://iam-auth.apx.develop.eoepca.org/realms/eoepca/.well-known/uma2-configuration
        lazy_load_paths: true
        ssl_verify: false
      enable: true
      name: authz-keycloak
    websocket: true

[w-scho]

I added the "view-clients" and "query-clients" roles to your user, which should at least allow you to inspect the client. If anything is still missing, I can add further roles tomorrow. Also tomorrow I am going to amend the "workspace_pipeline" client.
Regarding the client role "ws_access" my understanding is that it is local to the client and thus needs to be recreated for each client (explicitly unless we can leverage some templating mechanism for this).

[achtsnits]

thanks for clarification

you need to help me - where do I find the client policies in this keycloak console UI tool?

[w-scho]

There is an Authorization tab with several subtabs in the client details view. However, you probably could not see it, because I had forgotten to give you the "view-authorization" role. I hope that you can see it now.

[achtsnits]

yes, thanks!

[w-scho]

I assigned the following roles to the workspace_pipeline client:

• create-client
• manage-clients (probably needed to configure the client after creation)
• manage-authorization (needed to configure authorization on client)
• manage-users (probably needed for creating groups as there is no separate create-group or manage-groups role)

I omitted the view-* and query-* roles for now, assuming that they are either not required for API access or implied in the manage-* roles. I can add some of them later if they turn out to be required.

[achtsnits]

while we have managed to automate the provisioning of Client, Role and ClientRolePolicy (see e.g. client ws-test5 - this should be mostly complete right?) we are still struggling to set the Default Permission to the Role ws_access Policy

we will investigate deeper tomorrow (need to dive into actual IaC code), but can you please explain the keycloak logic here and also check if there perhaps is another way without requiring replacing Default Policy with Role ws_access Policy? thanks

[w-scho]

Could you explain what exactly the issue is, so that I know what to look for? Is it a missing (or maybe well-hidden) API, or just weird behaviour of an API that should do the right thing, or anything else?
I remember that a colleague also had problems in conjunction with manipulating permissions via the API, but unfortunately he is not available today.
Alternative approaches I could imagine are to just add the new policy to the default permission (and also keep the default policy), to delete the default permission and create a new one or to delete the complete default resource and create a replacement for it. The idea behind manipulating the default permission was just to minimize effort.
The User Management demo notebook (https://github.com/EOEPCA/demo/blob/main/demoroot/notebooks/01%20User%20Management%20Keycloak.ipynb) includes code that creates resources, policies and permissions and associates them and maybe could serve as an example. The result is the Keycloak client called "demo". However, it does not manipulate or delete the default resource and default permission, but leaves them as they are, which is not an option for the workspace UI, because we need to change the protection of the whole server.

[achtsnits]

we are still struggling with the keycloak-provider of our (declarative) IaC tool to get that permission set properly, so we are still investigating a good way to achieve or workaround so we can automate the whole flow in workspace pipeline

thanks for your ideas and the notebook link, we will further explore - have a good weekend

[achtsnits]

we have the policy "Role ws_access Policy" in client "ws-test5"

https://iam-auth.develop.eoepca.org/admin/eoepca/console/#/eoepca/clients/38d2be6b-3e8e-4184-aa23-4436cb37dc9d/authorization/policy/f39dc340-e680-4272-bd47-445031cd4430/role

role id: '66d0ea00-e856-46a2-abfb-3276877dec00'
resource server id : '38d2be6b-3e8e-4184-aa23-4436cb37dc9d'

what would be the necessary API call now to get this policy added to the Authorizaton | Permissions for this client, can you help with that or shall we have a short call, e.g. 10am?

[w-scho]

I am not really familiar with Keycloak's authZ API, but I am trying to find out.
What I can say by now is:
The demo notebook creates a new permission with policies assigned this way:

• It calls assign_resources_permissions on the KeycloakClient from the old Identity Service (https://github.com/EOEPCA/um-identity-service/blob/develop/identityutils/keycloak_client.py)
• This in turn calls create_client_authz_resource_based_permission from the KeycloakAdmin client from the python-keycloak library (https://github.com/marcospereirampj/python-keycloak/blob/master/src/keycloak/keycloak_admin.py)
• Eventually this results in a POST on admin/realms/{realm-name}/clients/{client_id}/authz/resource-server/permission/resource?max=-1
  Example input from the demo notebook:

    {
        "name": "Premium permission",
        "type": "resource",
        "logic": "POSITIVE",
        "decisionStrategy": "UNANIMOUS",
        "resources": [
            "Premium resource"
        ],
        "policies": [
            "Only Premium User Policy"
        ]
    }

This is not exactly what you want to do, but it could be an alternative. I assume that there is a way to add policies to existing permissions, because the Keycloak UI allows doing that, but I do not know how.

[w-scho]

Maybe update_client_authz_resource_permission from the KeycloakAdmin client could be what we are looking for, though it looks a bit weird to me. It performs a PUT on admin/realms/{realm-name}/clients/{client_id}/authz/resource-server/permission/resource/{resource-id}.

[w-scho]

Another way to manipulate permissions may be the UMA-compliant Protection API provided by Keycloak: https://www.keycloak.org/docs/latest/authorization_services/index.html#_service_protection_api
However, its abstraction level seems to be a bit different in that it does not handle policies and permissions separately. So it is probably easier to keep using the Keycloak-specific API.