Add postgres init container to resolve permissions for some k3s deployments: init container commands fail with permission denied

[kcjones91]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  admin_user: admin
  service_type: nodeport
  postgres_security_context_settings:
    fsGroup: 26
  postgres_data_volume_init: true
  postgres_init_container_commands: |
    chown 26:0 /var/lib/pgsql/data
    chmod 700 /var/lib/pgsql/data

I am running into this on my end. Probably an easy fix?

kubectl logs awx-demo-postgres-15-0 -n awx-dev -c init
chown: changing ownership of '/var/lib/pgsql/data': Permission denied
chmod: changing permissions of '/var/lib/pgsql/data': Permission denied

swipe@swipe-worker-1:/mnt$ ls -la /mnt/data/postgres/
total 0
drwxrwxrwx. 2 swipe swipe  6 Dec  9 15:48 .
drwxr-xr-x. 3 swipe swipe 22 Dec  9 13:17 ..

AWX Operator version

2.19.1

AWX version

24

Kubernetes platform

kubernetes

Kubernetes/Platform version

K8s

Modifications

no
Running on Fedora 41

Steps to reproduce

Fresh install of AWX following the helm chart deployment guide. I followed this issue: #1805, tracking the issue for the fresh deployment. I attempted deploy AWX with my current configuration:

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  admin_user: admin
  service_type: nodeport
  postgres_security_context_settings:
    fsGroup: 26
  postgres_data_volume_init: true
  postgres_init_container_commands: |
    chown 26:0 /var/lib/pgsql/data
    chmod 700 /var/lib/pgsql/data

apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /mnt/data/postgres 

When I apply the configuration I recieve the following error:

kubectl logs awx-demo-postgres-15-0 -n awx-dev -c init
chown: changing ownership of '/var/lib/pgsql/data': Permission denied
chmod: changing permissions of '/var/lib/pgsql/data': Permission denied

Expected results

AWX deployed with a working persistent database.

Actual results

kubectl logs awx-demo-postgres-15-0 -n awx-dev -c init
chown: changing ownership of '/var/lib/pgsql/data': Permission denied
chmod: changing permissions of '/var/lib/pgsql/data': Permission denied

Additional information

No response

Operator Logs

No response

Tasks

Beta Give feedback

1. fixed typo in docs #2000
   community +1

Tasks

Beta Give feedback

No tasks being tracked yet.

[YaronL16]

It should work, might be related to the indentation on the fsGroup: 26, which should be further indented by 2 spaces.

Also try it without:

  postgres_security_context_settings:
    fsGroup: 26

And see if that makes any different for the init container

[craph]

Hi @kcjones91 ,

It works.

You have a wrong indentation in your file for fsGroup.

The instruction in #1805 solve the issue for me and I have the same version of AWX.

Best Regards,

[kcjones91]

Hi @kcjones91 ,

It works.

You have a wrong indentation in your file for fsGroup.

The instruction in #1805 solve the issue for me and I have the same version of AWX.

Best Regards,

This was my mistake. I did fix this issue before asking the question and removed that value:

helm get values awx-dev -n awx-dev
USER-SUPPLIED VALUES:
AWX:
  enabled: true
  name: awx
  spec:
    admin_user: admin
    postgres_data_volume_init: true
    postgres_init_container_commands: |
      chown 26:0 /var/lib/pgsql/data
      chmod 700 /var/lib/pgsql/data
    service_type: NodePort

[kcjones91]

It should work, might be related to the indentation on the fsGroup: 26, which should be further indented by 2 spaces.

Also try it without:

  postgres_security_context_settings:
    fsGroup: 26

And see if that makes any different for the init container

helm get values awx-dev -n awx-dev
USER-SUPPLIED VALUES:
AWX:
  enabled: true
  name: awx
  spec:
    admin_user: admin
    postgres_data_volume_init: true
    postgres_init_container_commands: |
      chown 26:0 /var/lib/pgsql/data
      chmod 700 /var/lib/pgsql/data
    service_type: NodePort

helm get values awx-dev -n awx-dev
USER-SUPPLIED VALUES:
AWX:
  enabled: true
  name: awx
  spec:
    admin_user: admin
    postgres_data_volume_init: true
    postgres_init_container_commands: |
      chown 26:0 /var/lib/pgsql/data
      chmod 700 /var/lib/pgsql/data
    postgres_security_context_settings:
      fsGroup: 26
    service_type: nodeport

kubectl logs awx-postgres-15-0 -n awx-dev -c init
chown: changing ownership of '/var/lib/pgsql/data': Permission denied
chmod: changing permissions of '/var/lib/pgsql/data': Permission denied

Sorry this was my typo. I fixed this before testing. It initially would not apply my configuration due to indention. But I did have that same error.

[kcjones91]

The issue was Selinux:

sudo semanage fcontext -a -t container_file_t "/mnt/data/postgres(/.*)?"
sudo restorecon -Rv /mnt/data/postgres

I am using vanilla k8s on Fedora. I tried to update above, but this solved the issue.