[helm] Cannot start replication pods when using a service account not named airbyte-admin (Airbyte OSS 1.3.1)
#49938
Comments
AcidFlow
added
area/platform
|
To add more insights, it seems that the replication pod first starts with the custom service account name, and might be changed to This then caused other issues (for instance accessing AWS secret manager from the pod) since we rely on IRSA to let the pod service account assume a specific AWS role to access the secrets (the airbyte-admin was not allowed to assume the appropriate role because we assumed we only had our own service account name). |
|
Hello @AcidFlow, Thank you for bringing this to our attention. It appears that despite specifying a different service account in your This behavior may be due to hardcoded references within the workload launcher component. Do you sharing the permissions or the configs you used to create the custom service account? |
marcosmarxm
changed the title
airbyte-admin (Airbyte OSS 1.3.1)airbyte-admin (Airbyte OSS 1.3.1)
marcosmarxm
added
team/deployments
and removed
needs-triage
team/compose
team/platform-move
autoteam
labels
|
Hello @marcosmarxm ! Thanks for the quick reply. We copied the service account definition from the official helm chart and added an annotation to let it assume our AWS roles through EKS IRSA. Here is the service account and bindings definition redacted: ---
apiVersion: v1
kind: ServiceAccount
automountServiceAccountToken: true
metadata:
annotations:
helm.sh/hook: pre-install
helm.sh/hook-weight: "-10"
eks.amazonaws.com/role-arn: arn:aws:iam::<REDACTED_AWS_ACCOUNT>:role/<REDACTED_ROLE_NAME>
name: "airbyte"
namespace: "<REDACTED>"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: airbyte-role
namespace: "<REDACTED>"
annotations:
helm.sh/hook: pre-install
helm.sh/hook-weight: "-5"
rules:
- apiGroups: ["*"]
resources: ["jobs", "pods", "pods/log", "pods/exec", "pods/attach", "secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] # over-permission for now
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: airbyte-binding
namespace: "<REDACTED>"
annotations:
helm.sh/hook: pre-install
helm.sh/hook-weight: "-3"
roleRef:
apiGroup: ""
kind: Role
name: airbyte-role
subjects:
- kind: ServiceAccount
name: airbyte |
|
We were facing the same issue, and turns out what worked for us, is if the chart is already installed and running, then change "create": true to "create": false then restart the airbyte-workload-launcher pods (https://artifacthub.io/packages/helm/airbyte/airbyte?modal=values): |
Helm Chart Version
1.3.1
What step the error happened?
During the Sync
Relevant information
Summary
After upgrading Airbyte to 1.3.1 (OSS), when workload-launcher starts a replication pod, it does a PATCH call to pods it is trying to launch, however the PATCH call seems to try to use
airbyte-adminas its ServiceAccount even though a different service account name is specified in thevalues.yaml(the pod being started has uses the correct service account and has theJOB_KUBE_SERVICEACCOUNTfrom the values file).Step to reproduce
airbyte-adminbut has the necessary role bindingsworkload-launcherpod logs:Let me know if you need any further information, I'll be happy to help.
Additional information
Note
The issue is not happening on 1.2.0.
Workaround
As a workaround I created another service account named
airbyte-adminhaving the same role binding as the role we normally use.Then workload-launcher will be able to start the replication pod.
Relevant log output
The text was updated successfully, but these errors were encountered: