Test_DaprSecretStore_Manual - Container state is 'Terminated'

[lakshmimsft]

Steps to reproduce

Creating an issue to track occurence of this error, occured once so far.
#7147

Observed behavior

2024-02-07T08:18:01.0242439Z === �[31mFAIL�[0m: test/functional/daprrp/resources Test_DaprSecretStore_Manual/deploy_testdata/daprrp-resources-secretstore-manual.bicep (22.93s)
2024-02-07T08:18:01.0244019Z     rptest.go:272: running step 0 of 1: deploy testdata/daprrp-resources-secretstore-manual.bicep
2024-02-07T08:18:01.0245523Z     deployexecutor.go:80: deploying deploy testdata/daprrp-resources-secretstore-manual.bicep from file testdata/daprrp-resources-secretstore-manual.bicep
2024-02-07T08:18:01.0247278Z 2024/02/07 08:16:49 Start streaming Kubernetes logs - Pod dapr-backend-5857679997-q7bnm is in state: Running
2024-02-07T08:18:01.0248434Z 2024/02/07 08:16:50 Start streaming Kubernetes logs - Pod dapr-frontend-5cd57bf44f-b6nwg is in state: Running
2024-02-07T08:18:01.0249781Z 2024/02/07 08:16:50 Saved container logs to /home/runner/work/radius/radius/dist/container_logs/dapr-backend-5857679997-q7bnm.dapr-backend.log
2024-02-07T08:18:01.0251574Z 2024/02/07 08:16:50 Start streaming Kubernetes logs - Pod dapr-frontend-5cd57bf44f-b6nwg is in state: Failed
2024-02-07T08:18:01.0253529Z 2024/02/07 08:16:50 Saved container logs to /home/runner/work/radius/radius/dist/container_logs/dapr-backend-5857679997-q7bnm.daprd.log
2024-02-07T08:18:01.0255020Z 2024/02/07 08:16:50 Saved container logs to /home/runner/work/radius/radius/dist/container_logs/dapr-frontend-5cd57bf44f-b6nwg.daprd.log
2024-02-07T08:18:01.0256311Z 2024/02/07 08:16:50 Start streaming Kubernetes logs - Pod dapr-frontend-5cd57bf44f-b6nwg is in state: Failed
2024-02-07T08:18:01.0257686Z 2024/02/07 08:16:50 Saved container logs to /home/runner/work/radius/radius/dist/container_logs/dapr-frontend-5cd57bf44f-b6nwg.dapr-frontend.log
2024-02-07T08:18:01.0258987Z 2024/02/07 08:16:50 Start streaming Kubernetes logs - Pod dapr-frontend-5cd57bf44f-b6nwg is in state: Failed
2024-02-07T08:18:01.0260102Z 2024/02/07 08:16:51 Start streaming Kubernetes logs - Pod dapr-frontend-5cd57bf44f-b6nwg is in state: Failed
2024-02-07T08:18:01.0261218Z 2024/02/07 08:16:51 Start streaming Kubernetes logs - Pod dapr-backend-5857679997-q7bnm is in state: Failed
2024-02-07T08:18:01.0262334Z 2024/02/07 08:16:51 Start streaming Kubernetes logs - Pod dapr-backend-5857679997-q7bnm is in state: Failed
2024-02-07T08:18:01.0263599Z 2024/02/07 08:16:51 Start streaming Kubernetes logs - Pod dapr-backend-5857679997-q7bnm is in state: Failed
2024-02-07T08:18:01.0264739Z 2024/02/07 08:16:51 Start streaming Kubernetes logs - Pod dapr-backend-5857679997-q7bnm is in state: Failed
2024-02-07T08:18:01.0267110Z     cli.go:471: [heartbeat] command rad deploy /home/runner/work/radius/radius/test/functional/daprrp/resources/testdata/daprrp-resources-secretstore-manual.bicep --parameters magpieimage=ghcr.io/radius-project/dev/magpiego:pr-11a6478866 is still running after 10.000938062s
2024-02-07T08:18:01.0269250Z 2024/02/07 08:16:58 Start streaming Kubernetes logs - Pod gnrc-scs-ctnr-68998f7944-qchvl is in state: Running
2024-02-07T08:18:01.0270578Z 2024/02/07 08:16:58 Streaming Kubernetes logs to /home/runner/work/radius/radius/dist/container_logs/gnrc-scs-ctnr-68998f7944-qchvl.daprd.log
2024-02-07T08:18:01.0272173Z 2024/02/07 08:16:58 Streaming Kubernetes logs to /home/runner/work/radius/radius/dist/container_logs/gnrc-scs-ctnr-68998f7944-qchvl.gnrc-scs-ctnr.log
2024-02-07T08:18:01.0273728Z 2024/02/07 08:16:59 Saved container logs to /home/runner/work/radius/radius/dist/container_logs/gnrc-scs-ctnr-68998f7944-qchvl.daprd.log
2024-02-07T08:18:01.0274997Z 2024/02/07 08:17:00 Start streaming Kubernetes logs - Pod gnrc-scs-ctnr-68998f7944-qchvl is in state: Running
2024-02-07T08:18:01.0276145Z 2024/02/07 08:17:00 Start streaming Kubernetes logs - Pod gnrc-scs-ctnr-68998f7944-qchvl is in state: Running
2024-02-07T08:18:01.0277248Z 2024/02/07 08:17:01 Start streaming Kubernetes logs - Pod gnrc-scs-ctnr-68998f7944-qchvl is in state: Running
2024-02-07T08:18:01.0279610Z     cli.go:471: [heartbeat] command rad deploy /home/runner/work/radius/radius/test/functional/daprrp/resources/testdata/daprrp-resources-secretstore-manual.bicep --parameters magpieimage=ghcr.io/radius-project/dev/magpiego:pr-11a6478866 is still running after 20.001067123s
2024-02-07T08:18:01.0282304Z     cli.go:418: [rad] Building /home/runner/work/radius/radius/test/functional/daprrp/resources/testdata/daprrp-resources-secretstore-manual.bicep...
2024-02-07T08:18:01.0284711Z     cli.go:418: [rad] Deploying template '/home/runner/work/radius/radius/test/functional/daprrp/resources/testdata/daprrp-resources-secretstore-manual.bicep' into environment 'kind-radius' from workspace 'radiuse2e00-aks-admin'...
2024-02-07T08:18:01.0286247Z     cli.go:418: [rad] 
2024-02-07T08:18:01.0286778Z     cli.go:418: [rad] Deployment In Progress...
2024-02-07T08:18:01.0287229Z     cli.go:418: [rad] 
2024-02-07T08:18:01.0287583Z     cli.go:418: [rad] Error: {
2024-02-07T08:18:01.0288055Z     cli.go:418: [rad]   "code": "DeploymentFailed",
2024-02-07T08:18:01.0289124Z     cli.go:418: [rad]   "message": "At least one resource deployment operation failed. Please see the details for the specific operation that failed.",
2024-02-07T08:18:01.0291170Z     cli.go:418: [rad]   "target": "/planes/radius/local/resourceGroups/kind-radius/providers/Microsoft.Resources/deployments/rad-deploy-eecdcdab-fe32-4de1-a319-bfc8abc7db29",
2024-02-07T08:18:01.0292266Z     cli.go:418: [rad]   "details": [
2024-02-07T08:18:01.0292689Z     cli.go:418: [rad]     {
2024-02-07T08:18:01.0293210Z     cli.go:418: [rad]       "code": "ResourceDeploymentFailure",
2024-02-07T08:18:01.0293790Z     cli.go:418: [rad]       "message": "Failed",
2024-02-07T08:18:01.0294941Z     cli.go:418: [rad]       "target": "/planes/radius/local/resourceGroups/kind-radius/providers/Applications.Core/containers/gnrc-scs-ctnr",
2024-02-07T08:18:01.0295841Z     cli.go:418: [rad]       "details": [
2024-02-07T08:18:01.0296259Z     cli.go:418: [rad]         {
2024-02-07T08:18:01.0296715Z     cli.go:418: [rad]           "code": "Internal",
2024-02-07T08:18:01.0297548Z     cli.go:418: [rad]           "message": "Container state is 'Terminated' Reason: Error, Message: "
2024-02-07T08:18:01.0298188Z     cli.go:418: [rad]         }
2024-02-07T08:18:01.0298570Z     cli.go:418: [rad]       ]
...

logs_105226.zip
all_container_logs.zip

Dapr frontend/backend pods get into a failed state before deployment error "message": "Container state is 'Terminated' Reason: Error, Message: ".

Desired behavior

no error

Workaround

error not occurring on rerun

rad Version

v0.30

Operating system

Scheduled functional test

Additional context

No response

Would you like to support us?

• Yes, I would like to support you

AB#11151

[radius-triage-bot]

👋 @lakshmimsft Thanks for filing this bug report.

A project maintainer will review this report and get back to you soon. If you'd like immediate help troubleshooting, please visit our Discord server.

For more information on our triage process please visit our triage overview

[sylvainsf]

Closing non reproducible.

[kachawla]

Re-opening since this is still happening: #7265.

Looks like this error had happened last week as well, but just wasn't linked: #7220.

[radius-triage-bot]

👍 We've reviewed this issue and have agreed to add it to our backlog. Please subscribe to this issue for notifications, we'll provide updates when we pick it up.

We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

[kachawla]

Happened again for Test_DaprPubSubBroker_Manual: https://github.com/radius-project/radius/actions/runs/11241090980/job/31251912840

 cli.go:341: [rad]       "target": "/planes/radius/local/resourceGroups/kind-radius/providers/Applications.Core/containers/dpsb-manual-app-ctnr",
    cli.go:341: [rad]       "details": [
    cli.go:341: [rad]         {
    cli.go:341: [rad]           "code": "Internal",
    cli.go:341: [rad]           "message": "Container state is 'Terminated' Reason: Error, Message: "
    cli.go:341: [rad]         }
    cli.go:341: [rad]       ]
    cli.go:341: [rad]     },