LAMBDA_NET_SERIALIZER_DEBUG=true splits payloads over multiple lines

[martincostello]

Describe the bug

When LAMBDA_NET_SERIALIZER_DEBUG=true is set as an environment variable to try and debug JSON (de)serialization issues, if the JSON contains newlines it is split over multiple CloudWatch log messages, making it difficult to actually consume/understand the output.

Expected Behavior

The JSON payload is contained in a single CloudWatch log entry.

Current Behavior

The JSON payload is split across in multiple CloudWatch log entries.

Reproduction Steps

Enable LAMBDA_NET_SERIALIZER_DEBUG=true and have a Lambda function process a JSON payload that contains new lines, for example:

{
  "foo": "bar",
  "baz": "qux"
}

Possible Solution

For net6.0 and later, string.ReplaceLineEndings() could be used to transform the string before writing the payload to the console:

aws-lambda-dotnet/Libraries/src/Amazon.Lambda.Serialization.SystemTextJson/AbstractLambdaJsonSerializer.cs

Lines 60 to 61 in 28cb636

	var jsonDocument = debugReader.ReadToEnd();
	Console.WriteLine($"Lambda Serialize {response.GetType().FullName}: {jsonDocument}");

For example:

- var jsonDocument = debugReader.ReadToEnd();
+ var jsonDocument = debugReader.ReadToEnd().ReplaceLineEndings(string.Empty);

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

Amazon.Lambda.Serialization.SystemTextJson 2.4.1

Targeted .NET Platform

.NET 8

Operating System and version

Amazon Linux 2023

[ashishdhingra]

@martincostello Good morning. Please let me know your kind of deployment, is it container based? Kindly refer to #1265 (comment), it might be related.

Thanks,
Ashish

[martincostello]

No it is not a container, it is an Amazon Linux 2023 custom runtime, inside which there is a .NET 8 AoT application.

[ashishdhingra]

Reproducible with project created using Visual Studio AWS custom runtime template and sending JSON payload. Deployed with LAMBDA_NET_SERIALIZER_DEBUG set to true. It logs JSON payload on separate lines, including any error stack trace:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                                        message                                                                                                                                                         |
|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1711388585027 | INIT_START Runtime Version: provided:al2023.v14 Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:03a38c888e1c30b5347fd6c19fbb4e795676bb49160eee95751fb66e988ce013                                                                                                                                                |
| 1711388585527 | START RequestId: 76f41555-1b70-47c4-8417-c4f69d75bddd Version: $LATEST                                                                                                                                                                                                                                                 |
| 1711388585857 | 2024-03-25T17:43:05.657Z 76f41555-1b70-47c4-8417-c4f69d75bddd info Lambda Deserialize System.String: {                                                                                                                                                                                                                 |
| 1711388585857 | "foo": "bar",                                                                                                                                                                                                                                                                                                          |
| 1711388585857 | "baz": "qux"                                                                                                                                                                                                                                                                                                           |
| 1711388585857 | }                                                                                                                                                                                                                                                                                                                      |
...                                                                                                                                                                                                               |
| 1711388586400 | END RequestId: 76f41555-1b70-47c4-8417-c4f69d75bddd                                                                                                                                                                                                                                                                    |
| 1711388586400 | REPORT RequestId: 76f41555-1b70-47c4-8417-c4f69d75bddd Duration: 871.14 ms Billed Duration: 1368 ms Memory Size: 256 MB Max Memory Used: 69 MB Init Duration: 496.78 ms                                                                                                                                                |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CCing @normj for any inputs.

[normj]

This is a limitation of the provided.alX runtimes. Like containers the provided.alX custom runtimes don't have access to the telemetry API used by managed runtimes to send logs to Lambda as a discrete chunk and instead rely on scrapping stdout.

We hope to get .NET onboarded with Lambda's structured logging support soon. That would allow you to capture all of the log messages as a single line JSON and remove the issue of stdout line breaks causing issues.

[normj]

I'm going to close this issue because Lambda has no plans for adding the telemetry API for the provided.alX runtimes. If you want to track the structured logging support follow this PR.
#1588

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.