From ae21d93de4a14747b0c3b95fdc49c00b23270a6a Mon Sep 17 00:00:00 2001 From: Venkata Jagannath <160357297+venkatajagannath@users.noreply.github.com> Date: Wed, 9 Oct 2024 08:55:31 -0400 Subject: [PATCH] Documentation changes -- Cloud Auth (#58) * doc changes * update --- codespell-ignore-words.txt | 2 + docs/getting_started/cloud_platforms_auth.rst | 68 +++++++++++++++++++ docs/getting_started/setup.rst | 11 ++- 3 files changed, 79 insertions(+), 2 deletions(-) create mode 100644 docs/getting_started/cloud_platforms_auth.rst diff --git a/codespell-ignore-words.txt b/codespell-ignore-words.txt index 3a7e83f..0d82c40 100644 --- a/codespell-ignore-words.txt +++ b/codespell-ignore-words.txt @@ -1 +1,3 @@ asend +AKS +aks diff --git a/docs/getting_started/cloud_platforms_auth.rst b/docs/getting_started/cloud_platforms_auth.rst new file mode 100644 index 0000000..7fd3aee --- /dev/null +++ b/docs/getting_started/cloud_platforms_auth.rst @@ -0,0 +1,68 @@ +Cloud Platform instructions +=========================== + + +Amazon Web Services (AWS) +------------------------- + +The below steps are useful if you have your Kubernetes cluster hosted on AWS EKS + +**1. Add AWS CLI to your Airflow worker** + +.. code-block:: sh + + pip install awscli + +Or add the package to your requirements.txt file. For other methods to install the CLI, please see `here `_ + +**2. Authenticate with AWS** + +.. code-block:: sh + + AWS_ACCESS_KEY_ID= + AWS_SECRET_ACCESS_KEY= + AWS_SESSION_TOKEN= + +The Airflow worker will need to access the Kubernetes cluster on AWS. Setup access using one of the methods listed `here1 `_ + +Microsoft Azure +--------------- + +If you have your Kubernetes cluster hosted on Azure Kubernetes Service (AKS), follow these steps: + +**1. Install Azure CLI on your Airflow worker** + +.. code-block:: sh + + pip install azure-cli + +For other installation methods, please refer to the `Azure CLI installation guide `_. + +**2. Authenticate with Azure** + +To authenticate the Airflow worker with Azure, get the below details from your k8 cluster's service principal: + +.. code-block:: sh + + AZURE_CLIENT_ID= + AZURE_CLIENT_SECRET= + AZURE_TENANT_ID= + +Replace ````, ````, and ```` with your specific Azure credentials. + +For more authentication options, see the `Azure authentication guide `_. + +Google Cloud Platform (GCP) +--------------------------- + +**1. Install the Google Cloud CLI on your Airflow worker** + +.. code-block:: sh + + pip install gcloud + +**2. Authenticate with GCP** + +.. code-block:: sh + + GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json" diff --git a/docs/getting_started/setup.rst b/docs/getting_started/setup.rst index 2d3d17b..4aea291 100644 --- a/docs/getting_started/setup.rst +++ b/docs/getting_started/setup.rst @@ -21,17 +21,24 @@ See the `installing Helm `_ page for other pip install astro-provider-ray +**3. (Optional) Set up Cloud Authentication** + +.. toctree:: + :maxdepth: 1 + + Amazon Web Services (AWS) + **3. Setting up the Airflow connection** - Setup/Teardown a Ray cluster on Kubernetes - **Connection ID**: e.g., "ray_k8s_conn" - **Connection Type**: "Ray" - - **Kube config path** OR **Kube config content (JSON format)**: Kubeconfig of the Kubernetes cluster where Ray cluster must be set up + - **Kube config path**: Fully qualified path to the kubeconfig file on your repo. For example, *"/usr/local/airflow/config/kubeconfig.yaml"* + - **Kube config content (JSON format)**: Kubeconfig of the Kubernetes cluster where Ray cluster must be set up in JSON format - **Namespace**: The K8s namespace where your cluster must be created. If not provided, "default" is used - **Optional fields**: Cluster context, Disable SSL, Disable TCP keepalive - - Use an existing Ray cluster - **Connection ID**: e.g., "ray_conn"