Bug: Debugging Python Lambda with VScode's debugger doesn't work with latest AWS Lambda Python docker image

[blieusong]

Description:

Debugging a lambda function locally with sam on VScode fails.

Steps to reproduce:

Run the debugger locally on VScode on any lambda function. I use the following launch.json

{
    "configurations": [
        {
            "type": "aws-sam",
            "request": "direct-invoke",
            "name": "xxxxx (python3.11)",
            "invokeTarget": {
                "target": "code",
                "projectRoot": "${workspaceFolder}/lambda/iam_key_rotation/src",
                "lambdaHandler": "index.lambda_handler",
                "architecture": "arm64",
            },
            "lambda": {
                "runtime": "python3.11",
                "payload": {
                    "json": {
                        "account_type": "xxxxx",
                        "username": "xxxxx"
                    },
                },
                "environmentVariables": {
                    "XXXXX": "90",
                }
            }
        }
}

This happens only with the latest public.ecr.aws/lambda/python image. Forcing previous version of the images doesn't yield this issue

Observed result:

2023-12-06 09:25:35 [INFO]: Preparing to debug locally: Lambda "index.lambda_handler"
2023-12-06 09:25:35 [INFO]: Building SAM application...
2023-12-06 09:25:35 [INFO]: Command: (not started) [/opt/homebrew/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctkxpZJyr/output --template /tmp/aws-toolkit-vscode/vsctkxpZJyr/app___vsctk___template.yaml --base-dir /Users/xxxxxx/Documents/Devs/xxx/infrastructure-lambda-functions/lambda/iam_key_rotation/src --manifest /tmp/aws-toolkit-vscode/vsctkxpZJyr/debug-requirements.txt]
2023-12-06 09:25:36 [INFO]: Building codeuri: /Users/xxxxxx/Documents/Devs/xxx/infrastructure-lambda-functions/lambda/iam_key_rotation/src runtime: python3.11 metadata: {} architecture: arm64 functions: src

2023-12-06 09:25:36 [INFO]:  Running PythonPipBuilder:ResolveDependencies

2023-12-06 09:25:38 [INFO]:  Running PythonPipBuilder:CopySource

2023-12-06 09:25:38 [INFO]: 
Build Succeeded

2023-12-06 09:25:38 [INFO]: 
Built Artifacts  : output
Built Template   : output/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke -t output/template.yaml
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided --template-file output/template.yaml

2023-12-06 09:25:38 [INFO]: Build complete.
2023-12-06 09:25:38 [INFO]: Starting SAM application locally
2023-12-06 09:25:38 [INFO]: AWS.running.command
Invoking index.lambda_handler (python3.11)
Local image is out of date and will be updated to the latest runtime. To skip this, pass in the parameter --skip-pull-image
Building image..........................
Using local image: public.ecr.aws/lambda/python:3.11-rapid-arm64.

Mounting /tmp/aws-toolkit-vscode/vsctkxpZJyr/output/src as /var/task:ro,delegated, inside runtime container
START RequestId: 85a98d08-07cf-4dc2-865b-b9089e08b44f Version: $LATEST
Prepending Lambda task root to path: /var/task
Starting debugger...
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Traceback (most recent call last):
  File "/tmp/lambci_debug_files/py_debug_wrapper.py", line 46, in <module>
    runpy.run_module('debugpy', run_name='__main__')
  File "<frozen runpy>", line 229, in run_module
  File "<frozen runpy>", line 88, in _run_code
  File "/var/task/debugpy/__main__.py", line 37, in <module>
    from debugpy.server import cli
  File "/var/task/debugpy/server/__init__.py", line 7, in <module>
    import debugpy._vendored.force_pydevd  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/debugpy/_vendored/force_pydevd.py", line 44, in <module>
    preimport('pydevd', [
  File "/var/task/debugpy/_vendored/__init__.py", line 126, in preimport
    import_module(name)
  File "/var/lang/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/debugpy/_vendored/pydevd/pydevd.py", line 45, in <module>
    from _pydevd_bundle import pydevd_utils
  File "/var/task/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_utils.py", line 9, in <module>
    import ctypes
  File "/var/lang/lib/python3.11/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: libffi.so.6: cannot open shared object file: No such file or directory
06 Dec 2023 09:25:44,888 [ERROR] (rapid) Init failed error=Runtime exited with error: exit status 1 InvokeID=
Prepending Lambda task root to path: /var/task
Starting debugger...
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Traceback (most recent call last):
  File "/tmp/lambci_debug_files/py_debug_wrapper.py", line 46, in <module>
    runpy.run_module('debugpy', run_name='__main__')
  File "<frozen runpy>", line 229, in run_module
  File "<frozen runpy>", line 88, in _run_code
  File "/var/task/debugpy/__main__.py", line 37, in <module>
    from debugpy.server import cli
  File "/var/task/debugpy/server/__init__.py", line 7, in <module>
    import debugpy._vendored.force_pydevd  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/debugpy/_vendored/force_pydevd.py", line 44, in <module>
    preimport('pydevd', [
  File "/var/task/debugpy/_vendored/__init__.py", line 126, in preimport
    import_module(name)
  File "/var/lang/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/debugpy/_vendored/pydevd/pydevd.py", line 45, in <module>
    from _pydevd_bundle import pydevd_utils
  File "/var/task/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_utils.py", line 9, in <module>
    import ctypes
  File "/var/lang/lib/python3.11/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: libffi.so.6: cannot open shared object file: No such file or directory
END RequestId: 987b837b-674f-4702-aa9e-cc2f7c716120
REPORT RequestId: 987b837b-674f-4702-aa9e-cc2f7c716120	Init Duration: 0.56 ms	Duration: 241.64 ms	Billed Duration: 242 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	


Command stopped: "sam local invoke"
2023-12-06 09:25:46 [INFO]: Attaching debugger to SAM application...

The debugger of course fails to attach to the app.

Expected result:

I should be able to run the lambda function with the debugger.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS Ventura 13.6.2
2. sam --version: SAM CLI, version 1.103.0
3. AWS region: eu-west-1

# Paste the output of `sam --info` here
{
  "version": "1.103.0",
  "system": {
    "python": "3.12.0",
    "os": "macOS-13.6.2-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.7",
    "aws_cdk": "Not available",
    "terraform": "1.6.5"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

I think Python is expecting libffi.so.6 but I can only see libffi.so.8

ImportError: libffi.so.6: cannot open shared object file: No such file or directory

[blieusong]

version 3.11.2023.12.06.11 of the Python Lambda docker images fixes the issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.