Unable to connect after run & Amazon heath checks fail

[four43]

Describe the Issue

After running the playbook I restart the instance and access it. If I take an AMI of the instance and try and run it again however, it won't start properly.

After running:

sudo ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook \
    -i localhost, --connection=local \
    site.yml \
    -e os_gpg_key_pubkey_name=gpg-pubkey-d832c631-6515c85e \
    -e amzn2023cis_syslog_service=journald \
    --tags level1-server \
    --skip-tags rule_1.2.4,rule_4.6.6 | tee cis-ansible-harden.log

I can pull logs from the instance that is failing:

Boot Log

�[2J�[01;01H�[=3h�[2J�[01;01H�[2J�[01;01H�[=3h�[2J�[01;01H�[2J�[01;01H�[=3h�[2J�[01;01H�[0m�[35m�[40m�[2J�[01;01H�[=3h�[2J�[01;01H�[0m�[37m�[40m  Booting `Amazon Linux (6.1.92-99.174.amzn2023.x86_64) 2023'


[    0.071111] RETBleed: WARNING: Spectre v2 mitigation leaves CPU vulnerable to RETBleed attacks, data leaks possible!
[   11.677731] kauditd_printk_skb: 37 callbacks suppressed
[   11.677733] audit: type=1305 audit(1718920166.950:71): op=set audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 res=1
[   11.679489] audit: type=1300 audit(1718920166.950:71): arch=c000003e syscall=46 success=yes exit=60 a0=3 a1=7ffdb7634340 a2=4000 a3=7ffdb76343cc items=0 ppid=1 pid=833 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-journal" exe="/usr/lib/systemd/systemd-journald" subj=system_u:system_r:syslogd_t:s0 key=(null)
[   11.683252] audit: type=1327 audit(1718920166.950:71): proctitle="/usr/lib/systemd/systemd-journald"
[   11.688457] systemd[1]: Started systemd-journald.service - Journal Service.
[   11.691369] audit: type=1130 audit(1718920166.960:72): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   11.746154] systemd-journald[833]: Received client request to flush runtime journal.
[   11.796820] audit: type=1130 audit(1718920167.070:73): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.015400] audit: type=1130 audit(1718920167.290:74): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-sysusers comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.025959] systemd-journald[833]: Data hash table of /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal has a fill level at 78.5 (1785 of 2275 items, 1310720 file size, 734 bytes per hash table item), suggesting rotation.
[   12.028004] systemd-journald[833]: /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal: Journal header limits reached or header out-of-date, rotating.
[   12.105780] audit: type=1130 audit(1718920167.380:75): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journal-flush comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.147423] audit: type=1130 audit(1718920167.420:76): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.176941] audit: type=1130 audit(1718920167.450:77): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dracut-shutdown comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.423940] audit: type=1130 audit(1718920167.700:78): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-machine-id-commit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   13.071902] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[   13.112759] ACPI: button: Power Button [PWRF]
[   13.113313] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1
[   13.114258] ACPI: button: Sleep Button [SLPF]
[   13.140928] cryptd: max_cpu_qlen set to 1000
[   13.142801] pps_core: LinuxPPS API ver. 1 registered
[   13.143429] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[   13.146413] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[   13.147349] i8042: Warning: Keylock active
[   13.148826] PTP clock support registered
[   13.159199] serio: i8042 KBD port at 0x60,0x64 irq 1
[   13.159739] serio: i8042 AUX port at 0x60,0x64 irq 12
[   13.230539] AVX2 version of gcm_enc/dec engaged.
[   13.231253] AES CTR mode by8 optimization enabled
[   13.231933] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.12.0g
[   13.242607] ena 0000:00:05.0: ENA device version: 0.10
[   13.243190] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[   13.322668] ena 0000:00:05.0: LLQ is not supported Fallback to host mode policy.
[   13.334765] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0400000, mac addr 16:ff:ef:c7:8f:25
[   13.432768] ena 0000:00:05.0 ens5: renamed from eth0
[   13.685470] zram_generator::config[1496]: zram0: system has too much memory (7811MB), limit is 800MB, ignoring.
[   15.322988] ena 0000:00:05.0 ens5: Local page cache is disabled for less than 16 channels
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mdbus-broke…ce�[0m - D-Bus System Message Bus.
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39msystemd-ho…d.service�[0m - Home Area Manager.
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39msystemd-lo…rvice�[0m - User Login Management.
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mdbus-broke…ce�[0m - D-Bus System Message Bus.
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mpolicy-rou…m - Set up policy routes for ens5.
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39msystemd-ne…Wait for Network to be Configured.
[  135.957908] cloud-init[3385]: Cloud-init v. 22.2.2 running 'init' at Thu, 20 Jun 2024 21:51:31 +0000. Up 135.92 seconds.
[  136.047154] cloud-init[3385]: ci-info: ++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++
[  136.048922] cloud-init[3385]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[  136.050499] cloud-init[3385]: ci-info: | Device |  Up  |           Address            |    Mask   | Scope |     Hw-Address    |
[  136.052055] cloud-init[3385]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[  136.053562] cloud-init[3385]: ci-info: |  ens5  | True | fe80::14ff:efff:fec7:8f25/64 |     .     |  link | 16:ff:ef:c7:8f:25 |
[  136.055157] cloud-init[3385]: ci-info: |   lo   | True |          127.0.0.1           | 255.0.0.0 |  host |         .         |
[  136.056710] cloud-init[3385]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[  136.058206] cloud-init[3385]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[  136.059320] cloud-init[3385]: ci-info: +-------+-------------+---------+-----------+-------+
[  136.060500] cloud-init[3385]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[  136.061734] cloud-init[3385]: ci-info: +-------+-------------+---------+-----------+-------+
[  136.062895] cloud-init[3385]: ci-info: |   0   |  fe80::/64  |    ::   |    ens5   |   U   |
[  136.064000] cloud-init[3385]: ci-info: |   1   |    local    |    ::   |    ens5   |   U   |
[  136.065106] cloud-init[3385]: ci-info: |   2   |  multicast  |    ::   |    ens5   |   U   |
[  136.066211] cloud-init[3385]: ci-info: +-------+-------------+---------+-----------+-------+
[  136.221822] cloud-init[3385]: 2024-06-20 21:51:31,506 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ea5fa0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8eba9a0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  136.232889] cloud-init[3385]: 2024-06-20 21:51:31,506 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8eba9a0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  136.633570] cloud-init[3385]: 2024-06-20 21:51:31,917 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed0d60>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed9700>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  136.642945] cloud-init[3385]: 2024-06-20 21:51:31,918 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed9700>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  137.037221] cloud-init[3385]: 2024-06-20 21:51:32,321 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed0fa0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed0430>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  137.046367] cloud-init[3385]: 2024-06-20 21:51:32,321 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed0430>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  137.439805] cloud-init[3385]: 2024-06-20 21:51:32,724 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ee3ac0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e6d460>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  137.448930] cloud-init[3385]: 2024-06-20 21:51:32,724 - url_helper.py[WARNING]: Calling 'None' failed [1/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e6d460>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  137.851360] cloud-init[3385]: 2024-06-20 21:51:33,135 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ee3100>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e6d9a0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  137.860560] cloud-init[3385]: 2024-06-20 21:51:33,135 - url_helper.py[WARNING]: Calling 'None' failed [1/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e6d9a0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  138.254861] cloud-init[3385]: 2024-06-20 21:51:33,538 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e72820>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e811c0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  138.263932] cloud-init[3385]: 2024-06-20 21:51:33,539 - url_helper.py[WARNING]: Calling 'None' failed [2/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e811c0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  139.659081] cloud-init[3385]: 2024-06-20 21:51:34,942 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88580>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88ee0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  139.668717] cloud-init[3385]: 2024-06-20 21:51:34,943 - url_helper.py[WARNING]: Calling 'None' failed [3/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88ee0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  142.062814] cloud-init[3385]: 2024-06-20 21:51:37,346 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88fa0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88760>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  142.072069] cloud-init[3385]: 2024-06-20 21:51:37,346 - url_helper.py[WARNING]: Calling 'None' failed [5/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88760>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  145.469382] cloud-init[3385]: 2024-06-20 21:51:40,753 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e9b2e0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e9bc40>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  145.478307] cloud-init[3385]: 2024-06-20 21:51:40,753 - url_helper.py[WARNING]: Calling 'None' failed [9/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e9bc40>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  149.874896] cloud-init[3385]: 2024-06-20 21:51:45,158 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e9b070>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e94af0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  149.886574] cloud-init[3385]: 2024-06-20 21:51:45,159 - url_helper.py[WARNING]: Calling 'None' failed [13/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e94af0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  155.282642] cloud-init[3385]: 2024-06-20 21:51:50,566 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88df0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e81dc0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  155.293011] cloud-init[3385]: 2024-06-20 21:51:50,566 - url_helper.py[WARNING]: Calling 'None' failed [19/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e81dc0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  162.692613] cloud-init[3385]: 2024-06-20 21:51:57,976 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88e50>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88e20>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  162.702151] cloud-init[3385]: 2024-06-20 21:51:57,976 - url_helper.py[WARNING]: Calling 'None' failed [26/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88e20>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  172.103163] cloud-init[3385]: 2024-06-20 21:52:07,387 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e72c70>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e6d9d0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  172.112474] cloud-init[3385]: 2024-06-20 21:52:07,387 - url_helper.py[WARNING]: Calling 'None' failed [36/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e6d9d0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  183.512976] cloud-init[3385]: 2024-06-20 21:52:18,796 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e9b2b0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e72ee0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  183.522415] cloud-init[3385]: 2024-06-20 21:52:18,797 - url_helper.py[WARNING]: Calling 'None' failed [47/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e72ee0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39msystemd-lo…rvice�[0m - User Login Management.
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mdbus-broke…ce�[0m - D-Bus System Message Bus.
[  195.745473] systemd-journald[833]: Data hash table of /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal has a fill level at 75.0 (1707 of 2275 items, 1310720 file size, 767 bytes per hash table item), suggesting rotation.
[  195.748522] systemd-journald[833]: /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal: Journal header limits reached or header out-of-date, rotating.
[  196.929195] cloud-init[3385]: 2024-06-20 21:52:32,213 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ee35b0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed9430>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  196.938671] cloud-init[3385]: 2024-06-20 21:52:32,213 - url_helper.py[WARNING]: Calling 'None' failed [60/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed9430>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  212.342925] cloud-init[3385]: 2024-06-20 21:52:47,626 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e88580>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ee3fa0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  212.352846] cloud-init[3385]: 2024-06-20 21:52:47,627 - url_helper.py[WARNING]: Calling 'None' failed [76/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ee3fa0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  230.763157] cloud-init[3385]: 2024-06-20 21:53:06,047 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed0d30>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed0a30>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  230.772492] cloud-init[3385]: 2024-06-20 21:53:06,047 - url_helper.py[WARNING]: Calling 'None' failed [94/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed0a30>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[  252.181711] cloud-init[3385]: 2024-06-20 21:53:27,465 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8e72a30>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed0280>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception
[  252.190589] cloud-init[3385]: 2024-06-20 21:53:27,465 - url_helper.py[WARNING]: Calling 'None' failed [116/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77e8ed0280>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))]
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mrefresh-po… - Refresh policy routes for ens5.
[  276.457125] cloud-init[3385]: 2024-06-20 21:53:51,741 - url_helper.py[WARNING]: Timed out waiting for addresses: http://169.254.169.254:80/latest/api/token http://[fd00:ec2::254]:80/latest/api/token, exception(s) raised while waiting:
[  276.460149] cloud-init[3385]: 2024-06-20 21:53:51,741 - url_helper.py[WARNING]: Calling 'None' failed [140/120s]: unexpected error ['NoneType' object has no attribute 'contents']
[  276.462266] cloud-init[3385]: 2024-06-20 21:53:51,741 - url_helper.py[ERROR]: Timed out, no response from urls: ['http://169.254.169.254:80/latest/api/token', 'http://[fd00:ec2::254]:80/latest/api/token']
[  276.464809] cloud-init[3385]: 2024-06-20 21:53:51,746 - DataSourceEc2.py[WARNING]: IMDS's HTTP endpoint is probably disabled
[  276.672411] cloud-init[3385]: 2024-06-20 21:53:51,956 - cc_write_metadata.py[WARNING]: there is no identity dataset
[  276.674120] cloud-init[3385]: 2024-06-20 21:53:51,956 - cc_write_metadata.py[WARNING]: using path services/domain against metadata failed: KeyError: 'services'
[  276.754607] cloud-init[3385]: 2024-06-20 21:53:52,038 - util.py[WARNING]: Failed to set the hostname to localhost (localhost)
[  276.761129] cloud-init[3385]: 2024-06-20 21:53:52,045 - util.py[WARNING]: Running module set_hostname (<module 'cloudinit.config.cc_set_hostname' from '/usr/lib/python3.9/site-packages/cloudinit/config/cc_set_hostname.py'>) failed
[  276.860316] cloud-init[3385]: Generating public/private ed25519 key pair.
[  276.861437] cloud-init[3385]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key
[  276.862742] cloud-init[3385]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub
[  276.863892] cloud-init[3385]: The key fingerprint is:
[  276.864601] cloud-init[3385]: SHA256:KSUtM749HGhNcaxaihffWqpT0SjNo3qHAe9XIQ0CrD8 root@localhost
[  276.865747] cloud-init[3385]: The key's randomart image is:
[  276.866492] cloud-init[3385]: +--[ED25519 256]--+
[  276.867225] cloud-init[3385]: |   ...  ...      |
[  276.868073] cloud-init[3385]: |    . ...o.      |
[  276.869823] cloud-init[3385]: |   .  =++*       |
[  276.870506] cloud-init[3385]: |  . ..oXX.+      |
[  276.871162] cloud-init[3385]: |   . +=OS= .     |
[  276.871871] cloud-init[3385]: |    E.B=o.+      |
[  276.872700] cloud-init[3385]: |     =.++=       |
[  276.873385] cloud-init[3385]: |    . = =.       |
[  276.874034] cloud-init[3385]: |     ..=         |
[  276.874685] cloud-init[3385]: +----[SHA256]-----+
[  276.875330] cloud-init[3385]: Generating public/private ecdsa key pair.
[  276.876282] cloud-init[3385]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
[  276.877605] cloud-init[3385]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
[  276.879196] cloud-init[3385]: The key fingerprint is:
[  276.881608] cloud-init[3385]: SHA256:Hu0fsLaeoN8TcPhmxvJWUsFSXnOkE/gFdetjijLVcIA root@localhost
[  276.882799] cloud-init[3385]: The key's randomart image is:
[  276.883589] cloud-init[3385]: +---[ECDSA 256]---+
[  276.884430] cloud-init[3385]: |          .+..=+=|
[  276.885387] cloud-init[3385]: |         E..=. =+|
[  276.886109] cloud-init[3385]: |         . o.+oo |
[  276.886793] cloud-init[3385]: |        o.. = o. |
[  276.887546] cloud-init[3385]: |        S=oo . + |
[  276.888336] cloud-init[3385]: |       ..oXoo o .|
[  276.889038] cloud-init[3385]: |        oO+=..   |
[  276.889733] cloud-init[3385]: |       . +*+ .   |
[  276.890904] cloud-init[3385]: |      ...o=..    |
[  276.891653] cloud-init[3385]: +----[SHA256]-----+
[�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mcloud-init…it job (metadata service crawler).
[  277.397792] cloud-init[5851]: Cloud-init v. 22.2.2 running 'modules:config' at Thu, 20 Jun 2024 21:53:52 +0000. Up 277.34 seconds.
[  277.839259] cloud-init[5856]: Cloud-init v. 22.2.2 running 'modules:final' at Thu, 20 Jun 2024 21:53:53 +0000. Up 277.78 seconds.
ci-info: no authorized SSH keys fingerprints found for user ec2-user.
<14>Jun 20 21:53:53 cloud-init: #############################################################
<14>Jun 20 21:53:53 cloud-init: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Jun 20 21:53:53 cloud-init: 256 SHA256:Hu0fsLaeoN8TcPhmxvJWUsFSXnOkE/gFdetjijLVcIA root@localhost (ECDSA)
<14>Jun 20 21:53:53 cloud-init: 256 SHA256:KSUtM749HGhNcaxaihffWqpT0SjNo3qHAe9XIQ0CrD8 root@localhost (ED25519)
<14>Jun 20 21:53:53 cloud-init: -----END SSH HOST KEY FINGERPRINTS-----
<14>Jun 20 21:53:53 cloud-init: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNAX7PY1EB+vmSy3xCeOMZtkbRmy4xr+9kfgQ4q7YDQlkhcP6WAcViRKa1hohdk9y6SziYifO3owtpX74pOoJso= root@localhost
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDqtskN6MD+/+WAicLEraMPfSkqqHQXYSIFs6rUKIWkm root@localhost
-----END SSH HOST KEY KEYS-----
[  277.965751] cloud-init[5856]: Cloud-init v. 22.2.2 finished at Thu, 20 Jun 2024 21:53:53 +0000. Datasource DataSourceNone.  Up 277.96 seconds
[  277.968961] cloud-init[5856]: 2024-06-20 21:53:53,253 - cc_final_message.py[WARNING]: Used fallback datasource


Authorized uses only. All activity may be monitored and reported.

Expected Behavior

Instance fully boots without failures

Actual Behavior

See log above in repro steps

Control(s) Affected
What controls are being affected by the issue

I have no idea! I was hoping someone here might have an idea of what it nuking those systemd units.

Environment (please complete the following information):

• branch being used: release 1.1.0
  ansible 2.10.17
  python version = 3.9.17 (main, Jun 13 2023, 16:05:09) [GCC 8.3.0]

Additional Notes
Thanks for any insight or ideas!

Possible Solution
Unknown

[uk-bolly]

hi @four43

The original AMI works as expected i am assuming. Just when you take a copy to a new AMI?
I believe this could be something to do with cloud-init, it doesnt appear to be anything to do with this role specifically. Hopefully someone may have seen similar.
If you do find the root cause that comes from the role please let us know and we can see if we can add something for you.

Many thanks

uk-bolly

[vireshsolanki]

Yes, it is issue with AMI's

[herman-wong-cf]

I've spotted this issue in my own build.

@four43, in the role => tasks => section_4 => cis_4.6.x.yml => "4.6.5 | PATCH | Ensure default user umask is 027 or more restrictive | Set umask for /etc/login.defs pam_umask settings", try comment out "/etc/bashrc" from the "loop".

@uk-bolly I don't know what the ultimate root cause is, but excluding that file from the loop allowed me to launch the instance normally. I'm guessing something runs in cloud-init that depends on a loose umask in /etc/bashrc.

This is specifically an issue where there are no problems/errors with the Packer AMI build, but launching an instance from the AMI leads to basic systemd service failures.

[uk-bolly]

hi @herman-wong-cf @four43

Thank you both for the feedback, Having a quick read up it is indeed cloud-init. In order to be compliant that will need to be adjusted, to either skip as you have mentioned or set the permissions back in cloud init and once completed fix it to be compliant.

Its not something that we would change as part of the role. It would be a great article once resolved on how to fix it.

Kindest

uk-bolly

[vireshsolanki]

hi @uk-bolly , your script changes the /etc/fstab entry so when creating the AMI it got crashes because of that and @herman-wong-cf the role => tasks => section_4 => cis_4.6.x.yml => "4.6.5 | PATCH | Ensure default user umask is 027 or more restrictive | Set umask for /etc/login.defs pam_umask settings", try comment out "/etc/bashrc" from the "loop". this one is issue as well, after implementing it cloud-init runs successfully.