diff --git a/README.md b/README.md index 63fb6414..dc64d110 100644 --- a/README.md +++ b/README.md @@ -25,9 +25,6 @@ This project contains Ansible code that creates a baseline cluster in an existin - Manage SAS Viya Platform Deployments - Organize and persist configuration for any number of SAS Viya platform deployments across namespaces, clusters, or cloud providers. -- SAS Viya Multi-tenant Deployment - - Multi-tenancy in the SAS Viya platform creates and manages separate and secure cloud-computing units. See [SAS Viya Multi-Tenancy Documentation](./docs/user/Multi-Tenancy.md) for details. - - SAS Viya with SingleStore Deployment - SingleStore is a cloud-native database designed for data-intensive applications. See the [SAS Viya with SingleStore Documentation](./docs/user/SingleStore.md) for details. diff --git a/docs/CONFIG-VARS.md b/docs/CONFIG-VARS.md index e5baaf4a..32374f66 100644 --- a/docs/CONFIG-VARS.md +++ b/docs/CONFIG-VARS.md @@ -34,7 +34,6 @@ Supported configuration variables are listed in the table below. All variables - [Metrics Server](#metrics-server) - [NFS Client](#nfs-client) - [Postgres NFS Client](#postgres-nfs-client) - - [Multi-tenancy](#multi-tenancy) ## BASE @@ -452,56 +451,3 @@ The Postgres NFS client is currently supported by the nfs-subdir-external-provis | PG_NFS_CLIENT_CHART_NAME | nfs-subdir-external-provisioner Helm chart name | string | nfs-subdir-external-provisioner | false | | baseline | | PG_NFS_CLIENT_CHART_VERSION | nfs-subdir-external-provisioner Helm chart version | string | 4.0.18| false | | baseline | | PG_NFS_CLIENT_CONFIG | nfs-subdir-external-provisioner Helm values | string | See [this file](../roles/baseline/defaults/main.yml) for more information. | false | | baseline | - - -## Multi-tenancy - -| Name | Description | Type | Default | Required | Notes | Tasks | -| :--- | ---: | ---: | ---: | ---: | ---: | ---: | -| V4MT_ENABLE | Enables multi-tenancy in the SAS Viya platform deployment | bool | false | false || viya, multi-tenancy | -| V4MT_MODE | Set V4MT_MODE to either schema or database | string | schema | false | Two modes of data isolation (schemaPerApplicationTenant, databasePerTenant) are supported for tenant data. The default is schemaPerApplicationTenant. | viya, multi-tenancy | -| V4MT_TENANT_IDS | Maps to SAS_TENANT_IDS. One or more tenant IDs to onboard or offboard | string | | false | Example: Single tenant ID: "acme" or Multiple tenant IDs: "acme, cyberdyne, intech". Tenant IDs have a few naming restrictions, See the details [here](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=caltenants&docsetTarget=p0emzq13c0zbhxn1hktsdlmig934.htm#n1fptbibrh96r8n1jy317onpjd8r) | viya, multi-tenancy | -| V4MT_PROVIDER_PASSWORD | Optional: The password that is applied to the tenant administrator on each onboarded tenant | string | | false | Maps to SAS_PROVIDER_PASSWORD. When V4MT_PROVIDER_PASSWORD is specified, V4MT_PROVIDER_PASSWORD_{{TENANT-ID}} cannot be used. See details [here](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=caltenants&docsetTarget=p0emzq13c0zbhxn1hktsdlmig934.htm#p1ghvmezrb3cvxn1h7vg4uguqct6). | multi-tenancy | -| V4MT_PROVIDER_PASSWORD_{{TENANT-ID}} | Optional: Unique sasprovider password for each tenant being onboarded. {{TENANT-ID}} must be in uppercase. | string | | false | Maps to SAS_PROVIDER_PASSWORD_{{TENANT-ID}}. When V4MT_PROVIDER_PASSWORD_{{TENANT-ID}} is specified, V4MT_PROVIDER_PASSWORD cannot be used. See details [here](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=caltenants&docsetTarget=p0emzq13c0zbhxn1hktsdlmig934.htm#p1ghvmezrb3cvxn1h7vg4uguqct6). | multi-tenancy | -| V4MT_TENANT_CAS_CUSTOMIZATION | Map of objects with all tenant CAS customization variables. See the format below. | | | false | | multi-tenancy | - -### Tenant CAS Customization - -Some of the tenant CAS customizations can be defined with the V4MT_TENANT_CAS_CUSTOMIZATION variable, which is a map of objects. The variable has the following format: - -```bash -V4MT_TENANT_CAS_CUSTOMIZATION: - : - ... - : - ... - ... -``` -Below is the list of parameters each element can contain. - -| Name | Description | Type | Default | Required | Notes | Tasks | -| :--- | ---: | ---: | ---: | ---: | ---: | ---: | -| memory | Amount of RAM to allocate to per CAS node | string | | false | Numeric value followed by the units, such as 32Gi for 32 gibibytes. In Kubernetes, the unit is Gi. | multi-tenancy | -| cpus | Number of CPU cores to allocate per CAS node | string | | false | Either a whole number, representing that number of cores, or a number followed by m, indicating that number of milli-cores. | multi-tenancy | -| loadbalancer_enabled | Set up LoadBalancer to access CAS binary ports | bool | false | false | | multi-tenancy | -| loadbalancer_source_ranges | LoadBalancer source ranges specific to the tenant | list | false | false | | multi-tenancy | -| worker_count | The number of CAS worker nodes for tenants. Default is 0 (SMP) | int | 0 | false | | multi-tenancy | -| backup_controller_enabled | Enable backup CAS controller | bool | false | false | | multi-tenancy | - -Example: - -```bash -V4MT_TENANT_CAS_CUSTOMIZATION: - acme: - memory: 3Gi - cpus: 300m - loadbalancer_enabled: true - loadbalancer_source_ranges: ['0.0.0.0/0'] - worker_count: 0 - backup_controller_enabled: false - intech: - memory: 2Gi - cpus: 250m - worker_count: 1 - backup_controller_enabled: true -``` diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md index c7faf19a..e03995cf 100644 --- a/docs/Troubleshooting.md +++ b/docs/Troubleshooting.md @@ -10,7 +10,6 @@ - [Ansible Variables with Special Jinja2 Characters](#ansible-variables-with-special-jinja2-characters) - [Ingress-Nginx - use-forwarded-headers disabled](#ingress-nginx---use-forwarded-headers-disabled) - [Deploying with the SAS Orchestration Tool using a Provider Based Kubernetes Configuration File](#deploying-with-the-sas-orchestration-tool-using-a-provider-based-kubernetes-configuration-file) - - [SAS Risk Cirrus Solutions Multi-tenancy onboard failure](#sas-risk-cirrus-solutions-multi-tenancy-onboard-failure) - [Applying a New License for your SAS Viya Platform Deployment](#applying-a-new-license-for-your-sas-viya-platform-deployment) - [Tagging the AWS EC2 Load Balancers](#tagging-the-aws-ec2-load-balancers) @@ -346,21 +345,6 @@ You have a couple of options: * See [Kubernetes documentation](https://kubernetes.io/docs/concepts/security/service-accounts/) * Note: this is what the option above setting `create_static_kubeconfig=true` and running `terraform apply` would do for you automatically. -## SAS Risk Cirrus Solutions Multi-tenancy onboard failure -### Symptom: - -For SAS Viya Platform cadence v2023.07, Multi-tenancy onboard task fails for orders with SAS Risk Cirrus Solutions product. - -### Diagnosis: - -In the current multi-tenancy design of viya4-deployment the required tenant kubernetes resources or podtemplates are applied alongside cas-onboard task. However, SAS Risk Cirrus Solutions pods expect the podtemplates to be present at the time of onboarding. In SAS Viya Platform cadence v2023.07 the sas-tenant-job was modified to wait for all deployment services to start and return success or error code on exit for any services failures. This change causes the onboard task to fail as the SAS Risk Cirrus Solutions pods aren't able to start. - -### Solution: - -This is a known issue at the moment and will be fixed in future release of viya4-deployment. Meanwhile as a workaround users with SAS Risk Cirrus Solutions product please follow the manual steps to apply the required podtemplates and onboard tenants. - -After the initial provider deployment stabilizes follow the steps in `$deploy/sas-bases/examples/sas-tenant-job/README.md` to onboard tenants manually. - ## Applying a New License for your SAS Viya Platform Deployment ### Symptom: diff --git a/docs/user/Multi-Tenancy.md b/docs/user/Multi-Tenancy.md deleted file mode 100644 index f2fbe8b6..00000000 --- a/docs/user/Multi-Tenancy.md +++ /dev/null @@ -1,172 +0,0 @@ -# SAS Viya Multi-tenancy - -SAS Viya Multi-tenancy enables the onboarding and offboarding of tenants. The tenants share access to licensed SAS Viya applications, but tenants cannot access the data, workflows, users, and resources in other tenants. - -The onboarding and offboarding processes described here enable you to deploy tenants with specified users and groups as part of the deployment process. Standardized CAS Servers can be installed with each tenant, or customized CAS Servers can be installed after tenant onboarding. - -The tenant onboarding and offboarding processes can be run as needed after a successful provider deployment. Offboarding removes specified tenants and their CAS Servers. - -## Requirements for a Multi-tenant Environment - -1. CAS Server Resources Requirement. - - Each tenant requires a dedicated CAS server. See [CAS Server Resources](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=itopssr&docsetTarget=n0ampbltwqgkjkn1j3qogztsbbu0.htm#p1phbohacgeubcn0zgt2pdlqu0fu) for more details and [plan workload placement](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=dplyml0phy0dkr&docsetTarget=p0om33z572ycnan1c1ecfwqntf24.htm#p1ujrdxsdddpdpn1r3xavgwaa0tu) accordingly. - -2. PostgreSQL Requirement. - - SAS Viya Multi-tenancy requires either an internal PostgreSQL instance, which is the default option that is deployed automatically, or an external PostgreSQL instance that you configure and maintain. For external PostgreSQL, see [Requirements for External PostgreSQL](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=itopssr&docsetTarget=p05lfgkwib3zxbn1t6nyihexp12n.htm#p1wq8ouke3c6ixn1la636df9oa1u). Also for details see [PostgreSQL Requirements for a Multi-tenant Deployment](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=itopssr&docsetTarget=p05lfgkwib3zxbn1t6nyihexp12n.htm#p1r5u2f0yyiql5n11qb61lldcq1j). - - **Note:** Before deployment, when using internal PostgreSQL, be sure to size the total number of tenants that will be onboarded. The variable `V4MT_TENANT_IDS` must list all tenants planned to be onboarded. The list of tenants is used to calculate max_connections in PostgreSQL. After deployment, max_connections cannot be changed without redeploying the SAS Viya platform. - -3. TLS certificates. See [TLS Requirements](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=itopssr&docsetTarget=n18dxcft030ccfn1ws2mujww1fav.htm#p0bskqphz9np1ln14ejql9ush824). - -4. DNS configuration. See [DNS Requirements for Multi-tenancy](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=itopssr&docsetTarget=n18dxcft030ccfn1ws2mujww1fav.htm#n0mfva3uqvw78nn14s2deu1um3m1). - -5. LDAP or SCIM requirements. - - To configure LDAP see the steps in [OpenLDAP Customizations](#openldap-customizations). For more information on LDAP or SCIM requirements see [Additional LDAP Requirements for Multi-tenancy](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=itopssr&docsetTarget=n18dxcft030ccfn1ws2mujww1fav.htm#p1dr09lqs9w0w7n1iaklneorpy4r) or [Additional SCIM Requirements for Multi-tenancy](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=itopssr&docsetTarget=n18dxcft030ccfn1ws2mujww1fav.htm#n0snw477kspeqln1fmoeq3hu6c4m). - -## Limitations to Multi-tenancy Support -Multi-tenancy is not supported in every customer environment. For more information, see [Limitations to Multi-tenancy Support](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=itopssr&docsetTarget=n0jq6u1duu7sqnn13cwzecyt475u.htm#p11lcjg42kzdgjn1obgqb9zlaltw). - -## Actions - -| Name | Description | -| :--- | :--- | -| onboard | Adds and configures one or more tenants alongside the existing provider tenant | -| cas-onboard | Onboards a CAS server for an onboarded tenant | -| offboard | Removes one or more onboarded tenants | - -## Tasks - -| Name | Description | -| :--- | :--- | -| multi-tenancy | Enables you to onboard tenants, onboard CAS server for tenants, and offboard tenants. | - -## Preparation - -### Variable Definitions File (ansible-vars.yaml) - -Prepare your `ansible-vars.yaml` file, and set the variables as described in [Multi-tenancy](../CONFIG-VARS.md#multi-tenancy). The variables `V4MT_ENABLE` and `V4MT_MODE` must be set before deployment. Other variables can be set before the deployment or during the onboarding or offboarding procedures. See example [ansible-vars-multi-tenancy.yaml](../../examples/multi-tenancy/ansible-vars-multi-tenancy.yaml) - -**Note:** If your deployment has internal PostgreSQL then you must also set the variable `V4MT_TENANT_IDS` before deployment. You must include all the tenants that you plan to onboard in your deployment to calculate the `max_connections` correctly. Post deployment, when onboarding or offboarding, change the value of `V4MT_TENANT_IDS` to list the subset of involved tenants. - -For external PostgreSQL, `V4MT_TENANT_IDS` is optional for the deployment step. You can set it along with other variable during the onboarding or offboarding procedures. - -### OpenLDAP Customizations - -If the embedded OpenLDAP server is enabled, it is possible to change the users and groups that will be created. The required steps are similar to other customizations: - -1. Create the folder structure detailed in the [Customize Deployment Overlays](../../README.md#customize-deployment-overlays). -2. Copy the `./examples/multi-tenancy/openldap` folder into the `/site-config` folder. -3. Locate the openldap-modify-mt-users-groups.yaml file in the `/openldap` folder. -4. Modify the file to match the desired setup. The initial file contains example users and groups defined for tenant1 and tenant2. Make sure that you change the tenant IDs. - -**Note:** You must configure all the tenant-specific users and groups during the initial deployment as this method can only be used when you are first deploying the OpenLDAP server. Subsequently, you can either delete and redeploy the OpenLDAP server with a new configuration or add users using `ldapadd`. - -For example: - -- `/deployments` is the BASE_DIR -- The cluster is named demo-cluster -- The namespace will be named demo-ns -- Add overlay with custom LDAP setup - -```bash - /deployments <- parent directory - /demo-cluster <- folder per cluster - /demo-ns <- folder per namespace - /site-config <- location for all customizations - /openldap <- folder containing user defined customizations - /openldap-modify-mt-users-groups.yaml <- openldap overlay - ``` - -## Configure a Multi-tenant Deployment and Onboard Tenants - -Step 1. Configure a cluster with sufficient CAS node resources to support the number of tenants being onboarded. Deploy using your updated ansible-vars.yaml file. Run the following command to deploy SAS Viya Multi-tenancy: - - ```bash - ansible-playbook \ - -e BASE_DIR=$HOME/deployments \ - -e CONFIG=$HOME/deployments/dev-cluster/dev-namespace/ansible-vars.yaml \ - -e TFSTATE=$HOME/deployments/dev-cluster/terraform.tfstate \ - -e JUMP_SVR_PRIVATE_KEY=$HOME/.ssh/id_rsa \ - playbooks/playbook.yaml --tags "baseline,viya,install" - ``` - -Step 2. Make sure that the SAS Viya platform deployment is stable. Verify that you can log in to the provider tenant successfully. - -Step 3. Onboard tenants. Run the following command: - - ```bash - ansible-playbook \ - -e BASE_DIR=$HOME/deployments \ - -e KUBECONFIG=$HOME/deployments/.kube/config \ - -e CONFIG=$HOME/deployments/dev-cluster/dev-namespace/ansible-vars.yaml \ - -e TFSTATE=$HOME/deployments/dev-cluster/terraform.tfstate \ - -e JUMP_SVR_PRIVATE_KEY=$HOME/.ssh/id_rsa \ - playbooks/playbook.yaml --tags "multi-tenancy,onboard" - ``` -**Note:** As part of setup in the above `Onboard tenants` step, for every onboarded tenant, - -- A CAS server directory containing the configuration artifacts is created under the `/site-config` folder. -For example,if you have tenant with the ID `acme`, then a CAS server directory named `cas-acme-default` will be created. - -- Starting with SAS Viya Platform cadence 2023.03, each tenant will require their own copy of certain Kubernetes resources. Hence a new directory for each tenant containing all the `sas-programming-environment` files will be created under `$deploy/site-config/multi-tenant/`. For example, if you have a tenant with the ID `acme`, then a directory named `$deploy/site-config/multi-tenant/acme` will be created. - -Step 4. Add or update CAS customizations for tenants as needed and then run following command to onboard the CAS servers: - - ```bash - ansible-playbook \ - -e BASE_DIR=$HOME/deployments \ - -e KUBECONFIG=$HOME/deployments/.kube/config \ - -e CONFIG=$HOME/deployments/dev-cluster/dev-namespace/ansible-vars.yaml \ - -e TFSTATE=$HOME/deployments/dev-cluster/terraform.tfstate \ - -e JUMP_SVR_PRIVATE_KEY=$HOME/.ssh/id_rsa \ - playbooks/playbook.yaml --tags "multi-tenancy,cas-onboard" - ``` - -**Note:** -- If there are no additional CAS customizations required for tenants then run 'onboard' and 'cas-onboard' tags together in Step 3 and skip Step 4. -- The tenant CAS servers might take several mins to stabilize after the cas-onboard command above has completed successfully. - -## Log In and Validate an Onboarded Tenant -After the onboard and cas-onboard steps are complete see the steps [here](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=caltenants&docsetTarget=p0emzq13c0zbhxn1hktsdlmig934.htm#n05u0e3vmr5lcqn1l5xa2rhkdu6x) to login and validate an onboarded tenant. - -## Offboard Tenants and CAS Servers -Best practice before running offboard command: - -1. Perform a backup as a best-practice task. For more information, see [Backup and Restore: Perform an Ad Hoc Backup](https://documentation.sas.com/?cdcId=sasadmincdc&cdcVersion=default&docsetId=calbr&docsetTarget=p0cw7yuvwc83znn1igjc16zah2se.htm) in SAS Viya: Backup and Restore. - -2. Check for scheduled jobs and suspend all scheduled jobs to prevent them from automatically starting during or after offboarding. The jobs need to remain suspended at least until the offboarding of tenant CAS servers. If your deployment includes SAS Workflow Manager, use the [Workload Orchestrator](https://documentation.sas.com/?cdcId=sasadmincdc&cdcVersion=default&docsetId=evfun&docsetTarget=n15gjfza5o8i6hn1kr8f408c2et3.htm) page in SAS Environment Manager. Otherwise, use the [Jobs](https://documentation.sas.com/?cdcId=sasadmincdc&cdcVersion=default&docsetId=caljobs&docsetTarget=n0x3w4aokfoi1wn1q33jg4yrifge.htm) page in SAS Environment Manager. - -### Run the following command to Offboard Tenants and Offboard CAS Servers for tenants - - ```bash - ansible-playbook \ - -e BASE_DIR=$HOME/deployments \ - -e KUBECONFIG=$HOME/deployments/.kube/config \ - -e CONFIG=$HOME/deployments/dev-cluster/dev-namespace/ansible-vars.yaml \ - -e TFSTATE=$HOME/deployments/dev-cluster/terraform.tfstate \ - -e JUMP_SVR_PRIVATE_KEY=$HOME/.ssh/id_rsa \ - playbooks/playbook.yaml --tags "multi-tenancy,offboard" - ``` - -## Uninstall SAS Viya Multi-tenant Deployment -Before you run uninstall command make sure to run offboard command for any onboarded tenants. - -### Run the following command to uninstall - - ```bash - ansible-playbook \ - -e BASE_DIR=$HOME/deployments \ - -e KUBECONFIG=$HOME/deployments/.kube/config \ - -e CONFIG=$HOME/deployments/dev-cluster/dev-namespace/ansible-vars.yaml \ - -e TFSTATE=$HOME/deployments/dev-cluster/terraform.tfstate \ - -e JUMP_SVR_PRIVATE_KEY=$HOME/.ssh/id_rsa \ - playbooks/playbook.yaml --tags "baseline,viya,uninstall" - ``` - -## Troubleshooting - -1. Verify that all the pods are in `Running` or `Completed` state before offboarding the tenants. Otherwise, locks might have been added by SAS Viya platform services, and the offboarding job will exit without offboarding the tenants. -2. Do not attempt to offboard tenants immediately after onboarding. Most of the SAS Viya platform services are restarted during tenant onboarding. The tenant environment might be accessible during the time immediately following tenant onboarding, but there might be some services that have not yet stabilized. As a result, they can cause the issue that is described in the previous troubleshooting step.