Updates to deployment guides #3994

davidmirror-ops · 2023-08-28T20:25:30Z

Tracking issue

Closes #3973

Describe your changes

Add references to tutorials in multiple places
Updates the Multicluster setup guide
Removes or rewrites sentences to better match current status

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Screenshots

Note to reviewers

Signed-off-by: davidmirror-ops <[email protected]>

gdabisias · 2023-08-29T14:44:04Z

rsts/deployment/deployment/cloud_production.rst

-   This currently assumes that you have nginx ingress. We'll be updating these
-   in the near future to use the ALB ingress controller instead.
+   This section assumes that you're using the NGINX Ingress controller. Instructions and annotations for the ALB controller
+   are covered in the `Flyte The Hard Way <https://github.com/davidmirror-ops/flyte-the-hard-way/blob/main/docs/06-intro-to-ingress.md#setting-up-amazons-load-balancer-alb-ingress-controller>`__ tutorial.


FYI these were the final annotations that made things work correctly for me

{ "alb.ingress.kubernetes.io/certificate-arn": helm.values.userSettings.certificateArn "alb.ingress.kubernetes.io/group.name": "flyte" "alb.ingress.kubernetes.io/listen-ports": '[{"HTTPS":443}]' "alb.ingress.kubernetes.io/scheme": "internet-facing" "alb.ingress.kubernetes.io/ssl-redirect": "443" "alb.ingress.kubernetes.io/target-type": "ip" "kubernetes.io/ingress.class": "alb" "alb.ingress.kubernetes.io/inbound-cidrs": "xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx" }```

gdabisias · 2023-08-29T14:45:24Z

rsts/deployment/deployment/index.rst

 may interact with other parts of your stack.

-In addition to searching and posting on the `Flyte Slack community <https://flyte-org.slack.com/archives/C01P3B761A6>`__,
+Considering the above, we recommend checking out the `"Flyte The Hard Way" <https://github.com/davidmirror-ops/flyte-the-hard-way/tree/main#flyte-the-hard-way>`__ set of community-maintained tutorials that can guide you through the process of preparing the infrastructure and


In general I would avoid pointing to an external guide, give also that the guide you are referring to is mostly referring to the single binary deployment, which might be confusing. I think all documentation should be in one place

Agree. A second iteration of updates to these guides should be pointed to extend the reach and make it more actionable. Also expanding the tutorial in the https://github.com/unionai-oss/deploy-flyte repo to cover flyte-core

what if we forked your guide, @davidmirror-ops , and moved it to flyteorg?

@eapolinario I was thinking of something similar. Moving forward, the idea is that TF is the preferred approach but a manual guide is always a good resource

gdabisias · 2023-08-29T14:47:52Z

rsts/deployment/deployment/multicluster.rst

-To feed the credentials to FlyteAdmin, you must retrieve them from your new data plane cluster and upload them to admin (for example, within Lyft, `Confidant <https://github.com/lyft/confidant>`__ is used).
+   .. prompt:: bash $
+
+     kubectl create token <service-account-name> -n <namespace>


I'd expand on what service-account-name should be here.
If I understand correctly this is a separate service account, created in the data plane, which will be assume by the flyteadmin service account in the control plane to generate all the required resources in the data plane, right?

gdabisias · 2023-08-29T14:48:31Z

rsts/deployment/deployment/multicluster.rst

+
+     kubectl create token <service-account-name> -n <namespace>
+
+To feed the credentials to FlyteAdmin, you must retrieve them from your new data plane cluster and upload them to ``FlyteAmin``.


I would not use the word "upload" which might be misleading. I'd say that you need update your control plane configuration with the new secrets

gdabisias · 2023-08-29T14:49:20Z

rsts/deployment/deployment/multicluster.rst

-     cluster_2_cacert: {{ cluster 2 cacert here }}
-     cluster_3_token: {{ cluster 3 token here }}
-     cluster_3_cacert: {{ cluster 3 cacert here }}
+     cluster_1_token: "cluster-1-token-here"


Are we assuming that we have 3 dataplanes or 2? Since in the first case we only need 2 entries. I'd also specify how to change the names of the clusters

Signed-off-by: davidmirror-ops <[email protected]>

gdabisias · 2023-09-20T13:37:55Z

rsts/deployment/deployment/index.rst

@@ -108,7 +85,7 @@ There are three different paths for deploying a Flyte cluster:
   This option is appropriate if all your compute can `fit on one EKS cluster <https://docs.aws.amazon.com/eks/latest/userguide/service-quotas.html>`__ .
   As of this writing, a single Flyte cluster can handle more than 13,000 nodes.

-   Whatever path you choose, note that ``FlytePropeller`` itself can be sharded as well, though typically it's not required.
+   Regardless of using single or multiple Kubernetes clusters for Flyte, note that ``FlytePropeller`` -tha main data plane component- can be sharded as well, if scale demands require it.


typo: tha -> the

Signed-off-by: davidmirror-ops <[email protected]>

gdabisias · 2023-10-03T15:10:18Z

rsts/deployment/deployment/cloud_production.rst

+         :language: yaml
+         :lines: 127-135
+
+   .. group-tab:: ``flyte-binary`` on EKS using ALB 


Isn't this also the one used for flyte-core using ALB? I'd add that too

gdabisias · 2023-10-03T15:10:52Z

rsts/deployment/deployment/index.rst

@@ -49,29 +49,6 @@ deployment comes with a containerized `Minio <https://min.io/>`__, which offers
   - **GCP**: `GCS <https://cloud.google.com/storage/>`__
   - **Azure**: `Azure Blob Storage <https://azure.microsoft.com/en-us/products/storage/blobs>`__

-
-Cluster Configuration


Why are we removing this section?

I'm just not sure how it fits here. Additionally is not clear what specific K8s resources it refers to (besides namespaces for projects, which is only an example). So this relationship between Flyte objects and K8s resources is worth documenting, but I'm not sure it fits in this sort of pre-requisites section.

gdabisias · 2023-10-03T15:11:28Z

rsts/deployment/deployment/index.rst

@@ -108,7 +85,7 @@ There are three different paths for deploying a Flyte cluster:
   This option is appropriate if all your compute can `fit on one EKS cluster <https://docs.aws.amazon.com/eks/latest/userguide/service-quotas.html>`__ .
   As of this writing, a single Flyte cluster can handle more than 13,000 nodes.

-   Whatever path you choose, note that ``FlytePropeller`` itself can be sharded as well, though typically it's not required.
+   Regardless of using single or multiple Kubernetes clusters for Flyte, note that ``FlytePropeller`` -the main data plane component- can be sharded as well, if scale demands require it.


It is not clear what the word "shared" means here.

Added a reference to the sharding mechanism

gdabisias · 2023-10-03T15:12:30Z

rsts/deployment/deployment/multicluster.rst

+
+.. tabbed:: AWS
+
+   1. An IAM Policy that defines the permissions needed for Flyte. A minimum set of permissions include:


You actually need 2 sets of policies, one for the control plane and one for the data plane

Are you defining different permissions for control and data plane? The set of perms indicated here work for both

ideally they should be different, but initially they may be the same

gdabisias · 2023-10-03T15:12:42Z

rsts/deployment/deployment/multicluster.rst

+   .. code-block:: json
+
+
+   2. At least three IAM Roles configured: one for the controlplane components, another for the dataplane


controlplane -> control plane

gdabisias · 2023-10-03T15:19:38Z

rsts/deployment/deployment/multicluster.rst

+     namespace: flyte
+   type: Opaque
+   stringData:
+     dataplane_1_token: eyJhbGciOiJSUzI1NiIsImtpZCI6IlM0WlhfMm1Yb1U4Z1V4R0t6STZDdkhGTVVvVDBZcDAxbjdVbDc1Y1VxR28ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJmbHl0ZSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXRhcGxhbmUxLXRva2VuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImZseXRlYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkNTdhNjMwZi00ZTZmLTQzNTgtYjQwOS00M2UyMTlhYjg4NTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Zmx5dGU6Zmx5dGVhZG1pbiJ9.Fbn5qJjWP1wyJ08PgZXnrrUdKEhLRYqUzG9Vff1maFO3yBKkv_EBuYc2hjGeW5_ORCrT9qKcFAd3AE_tM3P8AQ-dRoA6K-RcJ2qinxabWmk9RYbtKFr1zujswU6dm-iB7JkjY7yYyBRewbw_m4QRacgG8K11c8bYZ9SZoV86EqGmsNdeCPuv5GiPBiJ0p3hgta4kZ1knCNf8qLBUQVZ-9G5vabYM0lyD6dvGOqlOs1bMzgLeijvpQN471dTLmIZ71anOG2gkuJW_AusnWDF_0rJ3yfISf3dRkhXkLswyq-awgtKbz6ZYjPaJ1eA8dNvSlbDoNrMXOGNlx7p7KhOY-w


No need to put an actual secret here

gdabisias · 2023-10-03T15:20:08Z

rsts/deployment/deployment/multicluster.rst

-     cluster_3_cacert: {{ cluster 3 cacert here }}
-
-Create cluster credentials secret in the control plane cluster.
+   stringData:


I'd suggest to use directly encoded data here since the new lines are error prone

Right.
So you mean also modifying the previous command to avoid the -D and also using a data section, right?

gdabisias · 2023-10-03T15:20:15Z

rsts/deployment/deployment/multicluster.rst

-
-Create cluster credentials secret in the control plane cluster.
+   stringData:
+     dataplane_1_token: eyJhbGciOiJSUzI1NiIsImtpZCI6IlM0WlhfMm1Yb1U4Z1V4R0t6STZDdkhGTVVvVDBZcDAxbjdVbDc1Y1VxR28ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJmbHl0ZSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXRhcGxhbmUxLXRva2VuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImZseXRlYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkNTdhNjMwZi00ZTZmLTQzNTgtYjQwOS00M2UyMTlhYjg4NTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Zmx5dGU6Zmx5dGVhZG1pbiJ9.Fbn5qJjWP1wyJ08PgZXnrrUdKEhLRYqUzG9Vff1maFO3yBKkv_EBuYc2hjGeW5_ORCrT9qKcFAd3AE_tM3P8AQ-dRoA6K-RcJ2qinxabWmk9RYbtKFr1zujswU6dm-iB7JkjY7yYyBRewbw_m4QRacgG8K11c8bYZ9SZoV86EqGmsNdeCPuv5GiPBiJ0p3hgta4kZ1knCNf8qLBUQVZ-9G5vabYM0lyD6dvGOqlOs1bMzgLeijvpQN471dTLmIZ71anOG2gkuJW_AusnWDF_0rJ3yfISf3dRkhXkLswyq-awgtKbz6ZYjPaJ1eA8dNvSlbDoNrMXOGNlx7p7KhOY-w


No need for an actual token

gdabisias · 2023-10-03T15:20:22Z

rsts/deployment/deployment/multicluster.rst

+   stringData:
+     dataplane_1_token: eyJhbGciOiJSUzI1NiIsImtpZCI6IlM0WlhfMm1Yb1U4Z1V4R0t6STZDdkhGTVVvVDBZcDAxbjdVbDc1Y1VxR28ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJmbHl0ZSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXRhcGxhbmUxLXRva2VuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImZseXRlYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkNTdhNjMwZi00ZTZmLTQzNTgtYjQwOS00M2UyMTlhYjg4NTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Zmx5dGU6Zmx5dGVhZG1pbiJ9.Fbn5qJjWP1wyJ08PgZXnrrUdKEhLRYqUzG9Vff1maFO3yBKkv_EBuYc2hjGeW5_ORCrT9qKcFAd3AE_tM3P8AQ-dRoA6K-RcJ2qinxabWmk9RYbtKFr1zujswU6dm-iB7JkjY7yYyBRewbw_m4QRacgG8K11c8bYZ9SZoV86EqGmsNdeCPuv5GiPBiJ0p3hgta4kZ1knCNf8qLBUQVZ-9G5vabYM0lyD6dvGOqlOs1bMzgLeijvpQN471dTLmIZ71anOG2gkuJW_AusnWDF_0rJ3yfISf3dRkhXkLswyq-awgtKbz6ZYjPaJ1eA8dNvSlbDoNrMXOGNlx7p7KhOY-w
+     dataplane_1_cacert:  |
+         -----BEGIN CERTIFICATE-----


No need for an actual cert

gdabisias · 2023-10-03T15:20:51Z

rsts/deployment/deployment/multicluster.rst

-        team1:
-        - id: cluster_1
+        label1:
+        - id: dataplane_1
          weight: 1


Make sure to explain here what these weight values are

Signed-off-by: davidmirror-ops <[email protected]>

davidmirror-ops · 2023-10-04T22:12:39Z

@gdabisias Thanks for your careful reviews!

gdabisias · 2023-10-05T12:14:09Z

rsts/deployment/deployment/multicluster.rst

+
+   .. code-block:: yaml
+
+      configmap:


I think you also need the configuration for datacatalog; in my case I had to also add the endpoint for that.
Note that this should be moved to the default charts and not left to the user

gdabisias · 2023-10-05T12:16:05Z

rsts/deployment/deployment/multicluster.rst

+
+                    "oidc.eks.<AWS-REGION-CODE>.amazonaws.com/id/<DATAPLANE1-OIDC-PROVIDER>:sub": [
+                    "system:serviceaccount:flyte:flytepropeller",
+                    "system:serviceaccount:*:default"


I'd explain here that ideally we would add only the namespaces needed, i.e. project-env, so that users can be more specific if they want

Signed-off-by: davidmirror-ops <[email protected]>

eapolinario

Looking good, left just a couple of comments.

eapolinario · 2023-10-05T21:43:12Z

charts/flyte-binary/eks-production.yaml

@@ -132,7 +132,7 @@ ingress:
    nginx.ingress.kubernetes.io/app-root: /console
  grpcAnnotations:
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
-  host: development.uniondemo.run
+  host: development.uniondemo.run # change for the URL you'll use to connect to Flyte


Can we make this an inexistent url? Maybe rename this to '<your.flyte.url>' ?

eapolinario · 2023-10-05T21:47:53Z

rsts/deployment/deployment/sandbox.rst

@@ -41,7 +41,7 @@ Requirements
 - Install `docker <https://docs.docker.com/engine/install/>`__ or any other OCI-compatible tool, like Podman or LXD.
 - Install `flytectl <https://github.com/flyteorg/flytectl>`__, the official CLI for Flyte.

-While Flyte can run any OCI-compatible task image, using the default Kubernetes container runtime (cri-o), the Flyte
+While Flyte can run any OCI-compatible task image using the default Kubernetes container runtime (cri-o), the Flyte


Isn't the default k8s container runtime containerd?

eapolinario · 2023-10-05T21:48:43Z

rsts/deployment/deployment/index.rst

 may interact with other parts of your stack.

-In addition to searching and posting on the `Flyte Slack community <https://flyte-org.slack.com/archives/C01P3B761A6>`__,
+Considering the above, we recommend checking out the `"Flyte The Hard Way" <https://github.com/davidmirror-ops/flyte-the-hard-way/tree/main#flyte-the-hard-way>`__ set of community-maintained tutorials that can guide you through the process of preparing the infrastructure and


what if we forked your guide, @davidmirror-ops , and moved it to flyteorg?

Signed-off-by: davidmirror-ops <[email protected]>

gdabisias · 2023-10-06T13:04:57Z

Looks all good :)
Please remember to open issues for the points that should not be part of the guide, but should be directly part of the deployment charts, so that we don't forget

Updates to deployment guides

db159e1

Signed-off-by: davidmirror-ops <[email protected]>

davidmirror-ops requested review from wild-endeavor and eapolinario August 28, 2023 21:07

gdabisias reviewed Aug 29, 2023

View reviewed changes

Update multicluster docs round 2

4063745

Signed-off-by: davidmirror-ops <[email protected]>

davidmirror-ops marked this pull request as draft September 20, 2023 10:07

davidmirror-ops requested a review from gdabisias September 20, 2023 10:08

gdabisias reviewed Sep 20, 2023

View reviewed changes

davidmirror-ops added 3 commits September 28, 2023 12:47

Updates instructions from last run

d36e9dc

Signed-off-by: davidmirror-ops <[email protected]>

Add instructions to add clusters

08692a7

Signed-off-by: davidmirror-ops <[email protected]>

Fix typos

be2abfd

Signed-off-by: davidmirror-ops <[email protected]>

davidmirror-ops marked this pull request as ready for review September 29, 2023 15:58

davidmirror-ops added 8 commits September 29, 2023 11:01

Fix JSON indentation in example

4c51dc2

Signed-off-by: davidmirror-ops <[email protected]>

Fix JSON indentation in example 2nd try

70737b7

Signed-off-by: davidmirror-ops <[email protected]>

Fix JSON missing blank line

bb25ab6

Signed-off-by: davidmirror-ops <[email protected]>

Fix JSON missing blank line 3rd try

cf77de9

Signed-off-by: davidmirror-ops <[email protected]>

Fix JSON missing blank line 4th try

56cfed8

Signed-off-by: davidmirror-ops <[email protected]>

Fix JSON syntax

828359b

Signed-off-by: davidmirror-ops <[email protected]>

Fix JSON syntax 6th try

c7f7e3e

Signed-off-by: davidmirror-ops <[email protected]>

Remove JSON block

18f0fc8

Signed-off-by: davidmirror-ops <[email protected]>

gdabisias reviewed Oct 3, 2023

View reviewed changes

davidmirror-ops added 8 commits October 3, 2023 12:15

Fix error in line 57

e5cea21

Signed-off-by: davidmirror-ops <[email protected]>

Fix spelling

e9b685b

Signed-off-by: davidmirror-ops <[email protected]>

Apply feedback from review

b257695

Signed-off-by: davidmirror-ops <[email protected]>

Fix hyperlink

ee5c001

Signed-off-by: davidmirror-ops <[email protected]>

Fix blank space

7db1b7f

Signed-off-by: davidmirror-ops <[email protected]>

Incorporate review

0e2d871

Signed-off-by: davidmirror-ops <[email protected]>

Incorporate 2nd round of review

9199f41

Signed-off-by: davidmirror-ops <[email protected]>

Instructions using 2 IAM Roles

db07bab

Signed-off-by: davidmirror-ops <[email protected]>

gdabisias reviewed Oct 5, 2023

View reviewed changes

Incorporate 3rd round of feedback

83d8c35

Signed-off-by: davidmirror-ops <[email protected]>

eapolinario reviewed Oct 5, 2023

View reviewed changes

davidmirror-ops added 2 commits October 5, 2023 17:18

Add instructions to enable controlplane wf execution

e784efd

Signed-off-by: davidmirror-ops <[email protected]>

Incorporate 4th round of reviews

fb0bb64

Signed-off-by: davidmirror-ops <[email protected]>

davidmirror-ops requested a review from eapolinario October 6, 2023 18:03

eapolinario approved these changes Oct 6, 2023

View reviewed changes

davidmirror-ops merged commit 8d66d05 into flyteorg:master Oct 7, 2023
16 checks passed

davidmirror-ops deleted the deployment-guide-updates branch October 7, 2023 12:41

davidmirror-ops restored the deployment-guide-updates branch October 7, 2023 12:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates to deployment guides #3994

Updates to deployment guides #3994

davidmirror-ops commented Aug 28, 2023 •

edited

Loading

gdabisias Aug 29, 2023

gdabisias Aug 29, 2023

davidmirror-ops Oct 3, 2023

eapolinario Oct 5, 2023

davidmirror-ops Oct 5, 2023

gdabisias Aug 29, 2023 •

edited

Loading

gdabisias Aug 29, 2023

gdabisias Aug 29, 2023

gdabisias Sep 20, 2023

gdabisias Oct 3, 2023

gdabisias Oct 3, 2023

davidmirror-ops Oct 3, 2023

gdabisias Oct 3, 2023

davidmirror-ops Oct 4, 2023

gdabisias Oct 3, 2023

davidmirror-ops Oct 3, 2023

gdabisias Oct 4, 2023

gdabisias Oct 3, 2023

gdabisias Oct 3, 2023

gdabisias Oct 3, 2023

davidmirror-ops Oct 3, 2023

gdabisias Oct 4, 2023

gdabisias Oct 3, 2023

gdabisias Oct 3, 2023

gdabisias Oct 3, 2023

davidmirror-ops commented Oct 4, 2023

gdabisias Oct 5, 2023

gdabisias Oct 5, 2023

eapolinario left a comment

eapolinario Oct 5, 2023

eapolinario Oct 5, 2023

eapolinario Oct 5, 2023

gdabisias commented Oct 6, 2023


		kubectl create token <service-account-name> -n <namespace>

		To feed the credentials to FlyteAdmin, you must retrieve them from your new data plane cluster and upload them to ``FlyteAmin``.


		.. tabbed:: AWS

		1. An IAM Policy that defines the permissions needed for Flyte. A minimum set of permissions include:

		.. code-block:: json


		2. At least three IAM Roles configured: one for the controlplane components, another for the dataplane

Updates to deployment guides #3994

Updates to deployment guides #3994

Conversation

davidmirror-ops commented Aug 28, 2023 • edited Loading

Tracking issue

Describe your changes

Check all the applicable boxes

Screenshots

Note to reviewers

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gdabisias Aug 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidmirror-ops commented Oct 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eapolinario left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gdabisias commented Oct 6, 2023

davidmirror-ops commented Aug 28, 2023 •

edited

Loading

gdabisias Aug 29, 2023 •

edited

Loading