Skip to content

Commit

Permalink
Documentation changes -- Cloud Auth (#58)
Browse files Browse the repository at this point in the history
* doc changes

* update
  • Loading branch information
venkatajagannath authored Oct 9, 2024
1 parent a900d43 commit ae21d93
Show file tree
Hide file tree
Showing 3 changed files with 79 additions and 2 deletions.
2 changes: 2 additions & 0 deletions codespell-ignore-words.txt
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
asend
AKS
aks
68 changes: 68 additions & 0 deletions docs/getting_started/cloud_platforms_auth.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
Cloud Platform instructions
===========================


Amazon Web Services (AWS)
-------------------------

The below steps are useful if you have your Kubernetes cluster hosted on AWS EKS

**1. Add AWS CLI to your Airflow worker**

.. code-block:: sh
pip install awscli
Or add the package to your requirements.txt file. For other methods to install the CLI, please see `here <https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html>`_

**2. Authenticate with AWS**

.. code-block:: sh
AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
AWS_SESSION_TOKEN=<your-session-token>
The Airflow worker will need to access the Kubernetes cluster on AWS. Setup access using one of the methods listed `here1 <https://docs.aws.amazon.com/cli/v1/userguide/cli-chap-authentication.html>`_

Microsoft Azure
---------------

If you have your Kubernetes cluster hosted on Azure Kubernetes Service (AKS), follow these steps:

**1. Install Azure CLI on your Airflow worker**

.. code-block:: sh
pip install azure-cli
For other installation methods, please refer to the `Azure CLI installation guide <https://docs.microsoft.com/en-us/cli/azure/install-azure-cli>`_.

**2. Authenticate with Azure**

To authenticate the Airflow worker with Azure, get the below details from your k8 cluster's service principal:

.. code-block:: sh
AZURE_CLIENT_ID=<your-client-id>
AZURE_CLIENT_SECRET=<your-client-secret>
AZURE_TENANT_ID=<your-tenant-id>
Replace ``<app-id>``, ``<password-or-cert>``, and ``<tenant>`` with your specific Azure credentials.

For more authentication options, see the `Azure authentication guide <https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli>`_.

Google Cloud Platform (GCP)
---------------------------

**1. Install the Google Cloud CLI on your Airflow worker**

.. code-block:: sh
pip install gcloud
**2. Authenticate with GCP**

.. code-block:: sh
GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json"
11 changes: 9 additions & 2 deletions docs/getting_started/setup.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,17 +21,24 @@ See the `installing Helm <https://helm.sh/docs/intro/install/>`_ page for other
pip install astro-provider-ray
**3. (Optional) Set up Cloud Authentication**

.. toctree::
:maxdepth: 1

Amazon Web Services (AWS) <cloud_platforms_auth>

**3. Setting up the Airflow connection**

- Setup/Teardown a Ray cluster on Kubernetes

- **Connection ID**: e.g., "ray_k8s_conn"
- **Connection Type**: "Ray"
- **Kube config path** OR **Kube config content (JSON format)**: Kubeconfig of the Kubernetes cluster where Ray cluster must be set up
- **Kube config path**: Fully qualified path to the kubeconfig file on your repo. For example, *"/usr/local/airflow/config/kubeconfig.yaml"*
- **Kube config content (JSON format)**: Kubeconfig of the Kubernetes cluster where Ray cluster must be set up in JSON format
- **Namespace**: The K8s namespace where your cluster must be created. If not provided, "default" is used
- **Optional fields**: Cluster context, Disable SSL, Disable TCP keepalive


- Use an existing Ray cluster

- **Connection ID**: e.g., "ray_conn"
Expand Down

0 comments on commit ae21d93

Please sign in to comment.