testcontainers.k3s failed creating kube-system containers due to ttrpc uknown error

[asafm]

Environmental Info:
K3s Version: 1.31.2

Node(s) CPU architecture, OS, and Version:

Darwin ip-192-168-1-19.ec2.internal 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30 PDT 2024; root:xnu-
10063.141.2~1/RELEASE_ARM64_T6030 arm64

Cluster Configuration:
1 node, both master and control plane

Describe the bug:
I'm using testcontainers to start k3s, and I see that the 3 kube-system containers are stuck in ContainerCreating state.
The events I see on one of them:

Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               13m                    default-scheduler  Successfully assigned kube-system/coredns-56f6fc8fd7-9vgrw to 4b0004577137
  Warning  FailedCreatePodSandBox  12m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "d8f28deeabadf6896fa7892392fe237965b6c39982e0ff201aacffd1b3f6b2ff": ttrpc: closed: unknown
  Warning  FailedCreatePodSandBox  12m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "5138391774e9e4271952d089772d4f0b1bdf55c82e5ef6c750fc6bbd9aaa0663": ttrpc: closed: unknown
  Warning  FailedCreatePodSandBox  12m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "225f0c7cb0ff43176f70c7e9bad8f4aa63aa1c37f6f99f9eb7b5640e0d29896e": ttrpc: closed: unknown
  Warning  FailedCreatePodSandBox  12m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "a619b5787675f5b65cbd556f6a93e78066e3889c2bdf4fa2641f942c69f3eb2f": ttrpc: closed: unknown
  Warning  FailedCreatePodSandBox  12m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "f180ea0040b847cb14189ad8b362341777886e334d84fc9e5517534e677db3b6": ttrpc: closed: unknown
  Warning  FailedCreatePodSandBox  12m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "b923c825664e2e5f2b9f731a5e25fdfc0b1cdb45416f00df7b9b2e99ca3f2a68": ttrpc: closed: unknown
  Warning  FailedCreatePodSandBox  12m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "086c02e430f6de1a8d32a0d7c766ac16af33f8ee64da620110dfc0645c51a2da": ttrpc: closed: unknown
  Warning  FailedCreatePodSandBox  12m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "feaa20fa4798c00457362af2c5b3bf6e0d3388cb423b24c58df583478bc58ad6": ttrpc: closed: unknown
  Warning  FailedCreatePodSandBox  12m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "3a468651b1c6c4652af3c827741285f7fc8fb6889d8991107edde4eb5c3a3c0f": ttrpc: closed: unknown
  Warning  FailedCreatePodSandBox  12m (x4 over 12m)      kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container task "6cf8b5c29f7f91b0fb615678879cb1cac3d7474681db3503ab954b2bb7b31ff0": ttrpc: closed: unknown
  Normal   SandboxChanged          2m57s (x582 over 12m)  kubelet            Pod sandbox changed, it will be killed and re-created.

Steps To Reproduce:
You can run this pytest

import os
import tempfile
from time import sleep

import kubernetes
import pytest
from kubernetes import client
from testcontainers.k3s import K3SContainer

TEST_NAMESPACE="airflow-test"

@pytest.fixture(scope="session")
def k3s_cluster():
    """Fixture to start and stop a k3s cluster using Testcontainers."""
    with K3SContainer() as k3s:
        # Retrieve kubeconfig YAML directly from k3s_cluster.config_yaml()
        kubeconfig_content = k3s.config_yaml()

        # Write kubeconfig to a temporary file
        with tempfile.NamedTemporaryFile(delete=False) as temp_config_file:
            temp_config_file.write(kubeconfig_content.encode())
            kubeconfig_path = temp_config_file.name

        kubernetes.config.load_kube_config(config_file=kubeconfig_path)
        namespace_metadata = client.V1ObjectMeta(name=TEST_NAMESPACE)
        namespace = client.V1Namespace(metadata=namespace_metadata)
        api_instance = client.CoreV1Api()
        api_instance.create_namespace(body=namespace)
        print(f"Namespace '{TEST_NAMESPACE}' created successfully.")
        print(f"Kube config file: {kubeconfig_path}")

        # Yield control back to the test
        yield kubeconfig_path

        # Clean up temporary kubeconfig file
        os.remove(kubeconfig_path)


def test_bla(k3s_cluster):
    sleep(1200)

Grab the config file from the system and use k9s to connect k9s --kubeconfig {the-file-path}
You can see per pod description the events.

Expected behavior:
kube-system container up and running

Actual behavior:
All 3 kube-system containers (core-dns, metrics server and localpath provisioner) are stuck in ContainerCreating

Additional context / logs:
Running from my Macbook Pro M3

[dereknola]

We have nothing to do with testcontainers. Its some third-party publishing the K3s modules. I'm not 100% how testcontainers works, but K3s is designed for linux. You said

Running from my Macbook Pro M3

does testcontainers run this in some kind of linux VM? because otherwise you will not get K3s working on a mac.