Container image v24.9.0: uid 1000 is missing from /etc/passwd

[marshallford]

Hello 👋🏼,

I maintain a Terraform provider for Ansible which executes ansible-navigator run commands using the ansible-dev-tools container image as the default EEI. Anyways, while running the project's acceptance tests locally using version 24.9.0 of the container image I bumped into what I suspect might be a regression involving this change: #364. See below for the bug report. Lastly, one more detail which confuses me: I opened a PR against said Terraform provider project to run the acceptance tests in CI with the container image change. To my surprise, the tests all pass (including those which fail on my local dev machine as described below). Perhaps it has to do with the user id or container engine that GitHub runners have setup?

Thank you!

Host OS: Linux
Host user (id -u): 1000
Ansible navigator version: 24.9.0
Container engine version (Docker): 27.2.1
Container image: ghcr.io/ansible/community-ansible-dev-tools:v24.9.0

Error:

EP_WARN: uid 1000 is missing from /etc/passwd, which is not writable; this error is likely fatal

PLAY [all] ********************************************************************* 
TASK [Gathering Facts] *********************************************************
fatal: [test]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: No user exists for uid 1000", "unreachable": true}

PLAY RECAP *********************************************************************
test                       : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   

Inventory:

"all":
  "hosts":
    "test":
      "ansible_host": "some.host.name"
      "ansible_user": "ubuntu"

Config:

ansible-navigator:
    ansible-runner:
        timeout: 600
    execution-environment:
        container-engine: auto
        enabled: true
        image: ghcr.io/ansible/community-ansible-dev-tools:v24.9.0
    logging:
        level: debug
    mode: stdout
    playbook-artifact:
        enable: true
    time-zone: UTC

Command:

ansible-navigator run playbook.yaml --inventory inventory --playbook-artifact-save-as playbook-artifact.json --log-file ansible-navigator.log

[ssbarnea]

This report needs to polished a little bit because in its current form is misleading. AFAIK, by design docker mounts /etc/passwd from host and never uses the file that is inside the container.

I am not sure if podman does the same, but the idea is that seeing such a message is to be expected.

[marshallford]

1. I attempted to reproduce this report with a vanilla install of podman (v5.2.4) on the same host and was not able to reproduce the error. This likely explains why CI for the PR linked above did not fail -- the standard Ubuntu GH runner includes podman, which ansible-navigator "prefers" over docker when container-engine: auto is set.

2. Perhaps more importantly, I think this error can be more broadly attributed to the issue discussed here WRT running SSH from within a container which does not have the UID used by the host. In addition, I suspect podman's default of --userns=host may explain why this works for podman users, but I'm not positive. Perhaps the entrypoint script for this container could look for env vars and create a user with a certain UID/GID on the fly to support docker users that don't wish to resolve this with userns-remap configuration steps?

This report needs to polished a little bit because in its current form is misleading

Let me know what I can add/change.

AFAIK, by design docker mounts /etc/passwd from host and never uses the file that is inside the container

Could you expand on that? I compared the /etc/passwd file on my host to the one in the container and they are different.

[jcox10]

I have never seen docker mount /etc/passwd from the host, unless it is explicitly done via a volume mount.

I have a feeling this issue is due to #364 where the podman user was removed from the container, it had uid 1000. I believe it was a bit short-sighted to remove the non-root user.

[cidrblock]

I think you are right here..... navigator sets the user to root if the engine is podman: https://github.com/ansible/ansible-navigator/blob/4b9165c69fe726c52c525600700c6cd4608a6f84/src/ansible_navigator/runner/base.py#L95 but not so for docker. While the devcontainer file specifies root for both docker and podman it's not the case for navigator.

It does look like a miss...

[cidrblock]

There's actually a little more to this....... ansible-builder injects an entrypoint script that should be building user 1000 at invocation of the container

https://github.com/ansible/ansible-builder/blob/16a5dce5e9164b1d57a101d6d623831399d3bd58/src/ansible_builder/_target_scripts/entrypoint#L79

that said, /etc/passwd isn't writeable, we might change that

[ssbarnea]

This issue happens only with docker and only on newer docker versions, like the one shipping with ubuntu-24.04 or with docker-desktop on macos. They changed something that no longer allows the dark magic entrypoint.py used by ansible-builder to dynamically add the missing user to the container.

I am not sure what we need to do to address it,... it might make more sense to play a little bit with simple containers only only try to make /etc/passwd editable.