Missing logs in Application Insights

[mlqn]

Description of the bug

Exceptions that occur in Program.cs, during the deployment of an application, are not logged in Application Insights, making it difficult for users to debug their app.

These logs can still be found in the ContainerLog table of the Kubernetes service, but our users don't have access to AKS.

Steps To Reproduce

1. Create a new application in Studio
2. Open the repository in Gitea
3. Add throw new System.Exception(); in App/Program.cs
4. Commit changes
5. Fetch changes in Studio
6. Create a new release
7. Deploy the new release. This release should fail to deploy and throw an exception

missing_ai.mov

8. Open logs in Application Insights and notice that the exception is missing

9. Open logs in AKS

[bjosttveit]

Does this issue belong in app-frontend-react? Maybe it should be moved to app-template-dotnet or app-lib-dotnet instead, depending on where the problem lies?

[mlqn]

@bjosttveit Nice catch, thanks! I moved it to app-template-dotnet as I mentioned the Program.cs file, but it might be an issue for app-lib-dotnet, I'm not sure. Feel free to move it to correct repo

[martinothamar]

I don't think it is possible to make logs be shipped through App Insights SDK (or OTel SDK for that matter) consistently in this way. The only place we know that logs will appear regardless of where or how early the code crashes is stdout/stderr from the container/process. In k8s these are stored on the host filesystem, and this is typically what log-scrapers (such as the one shipping to ContainerLog) uses to reliably capture all output. The AppInsights SDK (and OTel SDK) does flush telemetry on process exit, but only for clean exits.

Examples include

• OOM (Out Of Memory) crashes,
• SIGKILL from the host
• InvalidProgramException - weird case, but there are libraries out there doing runtime codegen which can be buggy

The case our users will most commonly run into I think are exceptions originating from configuration on startup. For example code that is run in RegisterCustomAppServices currently runs before AddAltinnAppServices which is currently what adds AppInsights/OTel.

Some options I can think of (that are not mutually exclusive):

• Register AppInsights/OTel SDK earlier in Program.cs - can be done for v9 I guess
• During release in Studio, have functionality for tailing logs so that users can see the early output and catch startup errors. Options:
  • Logs endpoint from k8s API
  • Query App Insights
• OOMs, SIGKILLs
  • k8s-level metrics
  • k8s events
  • k8s status/state fields in the deployment/pod resources

[mlqn]

Owners will soon have access to two new Log Analytics workspaces: application-{org}-{env}-law and operational-{org}-{env}-law, which will give them access to all logs. App logs will go to application-{org}-{env}-law, and infra/system logs will go to operational-{org}-{env}-law. So this issue might no longer be particularly relevant 🤔, except perhaps for the suggestion to move the initialization of AppInsights/OTel SDK earlier in Program.cs. 🤷‍♂️

[martinothamar]

Yeah I guess the original issue is reoslved. I think it would be beneficial to pull more app observability into the Studio UI over time, perhaps something for 2025 😄