Doubled response when using Amazon.Lambda.AspNetCoreServer.Hosting with .Net 6 Minimal API with custom endpoints extension

[heyjoey]

Description

When following the description to create a Minimal API with Amazon.Lambda.AspNetCoreServer.Hosting and added FastEndpoints, the response Dto is sent twice.

Reproduction Steps

Program.cs attached, created using serverless.AspNetCoreWebAPI project template and add a nuget package called FastEndpoints. Build and Publish to Lambda in Visual Studios 2022. Post payload as:

curl --location --request POST 'https://xxxxx.execute-api.ap-northeast-1.amazonaws.com/Prod/api/author' \
--header 'Content-Type: application/json' \
--data-raw '{
    "firstName": "John",
    "lastName": "Doe",
    "age": 16
}'

Response returned:

{
    "fullName": "John Doe",
    "isOver18": false
}{
    "fullName": "John Doe",
    "isOver18": false
}

Logs

screenshot attached.

Environment

• Build Version:
  • Amazon.Lambda.AspNetCoreServer.Hosting 1.0.0
  • FastEndpoints 3.9.1
• OS Info: Local Dev: Windows 10 | Lambda runtime: dotnet6 on AmazonLinux 2
• Build Environment: Visual Studio Community 2022
• Targeted .NET Platform: net6.0 (6.0.200)

Resolution

• 👋 I can/would-like-to implement a fix for this problem myself

It seems that Amazon.Lambda.AspNetCoreServer did not set the HasStarted property of the IHttpResponseFeature which the HttpContext reads from.

---

This is a 🐛 bug-report
Program.cs.txt

[dj-nitehawk]

the double response writing behavior is caused by the fact that FastEndpoints relies on the HttpContext.Response.HasStarted property. which works as expected when kestrel server is used and does not work when Amazon.Lambda.AspNetCoreServer is used. here's a minimal repro:

using Amazon.Lambda.Core;
using Amazon.Lambda.Serialization.SystemTextJson;

[assembly: LambdaSerializer(typeof(DefaultLambdaJsonSerializer))]
var builder = WebApplication.CreateBuilder();
builder.Services.AddAWSLambdaHosting(LambdaEventSource.RestApi);

var app = builder.Build();

app.MapGet("/test", async (HttpContext ctx) =>
{
    ctx.Response.StatusCode = 200;
    await ctx.Response.StartAsync();

    if (!ctx.Response.HasStarted)
        throw new InvalidOperationException("response has started but HasStarted says it's not, in aws lambda!");

    Console.WriteLine("this will only be printed with kestrel. won't print with aws lambda!");
});

app.Run();

i believe this is where the underlying value for this property is set by kestrel. so aws server should also do the same to keep in sync with the kestrel behavior i think.

[ashishdhingra]

@heyjoey Is it fine to mark this issue as feature-request instead of a bug since 3rd party dependency relies on some feature that is set by Kestrel server and it is desired to be set by Lambda runtime? Please advise.

[normj]

Thanks for reporting the issue @heyjoey and @dj-nitehawk. I agree this is a bug with how we are handling the response HasStarted bug. Looks like you have done a work around for 3.10.0 but we should still get this fixed in our library.

[ashishdhingra]

@heyjoey Good morning. Based on the testing, the issue doesn't appear to be reproducible in latest .NET 8 runtime. Below is sample code and output.
Code:
.csproj

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <AWSProjectType>Lambda</AWSProjectType>
    <!-- This property makes the build directory similar to a publish directory and helps the AWS .NET Lambda Mock Test Tool find project dependencies. -->
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    <!-- Generate ready to run images during publishing to improvement cold starts. -->
    <PublishReadyToRun>true</PublishReadyToRun>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.AspNetCoreServer.Hosting" Version="1.7.1" />
    <PackageReference Include="FastEndpoints" Version="5.30.0" />
  </ItemGroup>
</Project>

Program.cs

global using FastEndpoints;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddControllers();

// Add AWS Lambda support. When application is run in Lambda Kestrel is swapped out as the web server with Amazon.Lambda.AspNetCoreServer. This
// package will act as the webserver translating request and responses between the Lambda event source and ASP.NET Core.
builder.Services.AddAWSLambdaHosting(LambdaEventSource.RestApi);
builder.Services.AddFastEndpoints();

var app = builder.Build();

app.UseHttpsRedirection();
app.UseAuthorization();
app.UseFastEndpoints();
app.MapControllers();

app.MapGet("/", () => "Welcome to running ASP.NET Core Minimal API on AWS Lambda");
app.MapGet("/test", () => new { testProp1 = "hellow", testProp2 = "world" });

app.Run();

public class MyRequest : Amazon.Lambda.APIGatewayEvents.APIGatewayHttpApiV2ProxyRequest
{
    public string FirstName { get; set; } = "John";
    public string LastName { get; set; } = "Doe";
    public int Age { get; set; } = 0;
}

public class MyResponse
{
    public string FullName { get; set; }
    public bool IsOver18 { get; set; }
}

public class MyEndpoint : Endpoint<MyRequest>
{
    public override void Configure()
    {
        Verbs(Http.POST);
        Routes("/api/author");
        AllowAnonymous();
    }

    public override async Task HandleAsync(MyRequest req, CancellationToken ct)
    {
        Logger.LogInformation($"Request Received: \n{req}");

        var response = new MyResponse()
        {
            FullName = req.FirstName + " " + req.LastName,
            IsOver18 = req.Age > 18
        };
        Logger.LogInformation(System.Text.Json.JsonSerializer.Serialize(response));

        // Added to see if AWS reports back event started
        HttpContext.Response.OnStarting(() =>
        {
            Logger.LogInformation($"on starting event fires in aws");
            return Task.CompletedTask;
        });

        await SendAsync(response);
        Logger.LogInformation($"request started: {HttpContext.Response.HasStarted}");
    }
}

Tested using PowerShell (on Windows)
Command

 Invoke-WebRequest -Method POST -Uri 'https://<<REDACTED>>.execute-api.us-east-2.amazonaws.com/Prod/api/author' -Headers @{ "Content-Type" = "application/json"} -Body '{ "firstName": "John", "lastName": "Doe", "age": 16 }'

Response

StatusCode        : 200
StatusDescription : OK
Content           : {"fullName":"John Doe","isOver18":false}
RawContent        : HTTP/1.1 200 OK
                    Connection: keep-alive
                    X-Amzn-Trace-Id: Root=1-670ff9e2-16e670c6395ae63134e30328;Parent=2dda96de91fabe6c;Sampled=0;Lineage=1:37bd4e89:0
                    x-amzn-RequestId: 52956bec-128a-43b8-89e9-3fe...
Forms             : {}
Headers           : {[Connection, keep-alive], [X-Amzn-Trace-Id,
                    Root=1-670ff9e2-16e670c6395ae63134e30328;Parent=2dda96de91fabe6c;Sampled=0;Lineage=1:37bd4e89:0], [x-amzn-RequestId,
                    52956bec-128a-43b8-89e9-3fe1f6d6a399], [x-amz-apigw-id, fwP7dFVKCYcEdSA=]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 40

Tested using curl (on Unix/Mac)
Command

curl --location --request POST 'https://<<REDACTED>>.execute-api.us-east-2.amazonaws.com/Prod/api/author' \
--header 'Content-Type: application/json' \
--data-raw '{
    "firstName": "John",
    "lastName": "Doe",
    "age": 16
}'

Response

{"fullName":"John Doe","isOver18":false}%

CloudWatch Logs

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                 message                                                                                  |
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1729100175630 | INIT_START Runtime Version: dotnet:8.v20 Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:29df78d1d6c137bfb61f30e56b3fa81810740ad42ddcce8a715767c72d7d4504         |
| 1729100176015 | info: FastEndpoints.StartupTimer[0]                                                                                                                                      |
| 1729100176015 | Registered 1 endpoints in 127 milliseconds.                                                                                                                              |
| 1729100176198 | START RequestId: 7eb577ae-73f3-4936-8d64-90d9821adb05 Version: $LATEST                                                                                                   |
| 1729100176547 | END RequestId: 7eb577ae-73f3-4936-8d64-90d9821adb05                                                                                                                      |
| 1729100258807 | REPORT RequestId: e30f8a1e-fb21-44f8-b680-8566f410b1a7 Duration: 148.09 ms Billed Duration: 149 ms Memory Size: 512 MB Max Memory Used: 95 MB                            |
| 1729100320909 | START RequestId: 740045e1-6249-42cd-a9d8-08f45d6e68e1 Version: $LATEST                                                                                                   |
| 1729100320914 | info: MyEndpoint[0]                                                                                                                                                      |
| 1729100320914 | Request Received:                                                                                                                                                        |
| 1729100320914 | MyRequest                                                                                                                                                                |
| 1729100320914 | info: MyEndpoint[0]                                                                                                                                                      |
| 1729100320914 | {"FullName":"John Doe","IsOver18":false}                                                                                                                                 |
| 1729100320914 | info: MyEndpoint[0]                                                                                                                                                      |
| 1729100320914 | request started: False                                                                                                                                                   |
| 1729100320914 | info: MyEndpoint[0]                                                                                                                                                      |
| 1729100320914 | on starting event fires in aws                                                                                                                                           |
| 1729100320926 | END RequestId: 740045e1-6249-42cd-a9d8-08f45d6e68e1                                                                                                                      |
| 1729100320926 | REPORT RequestId: 740045e1-6249-42cd-a9d8-08f45d6e68e1 Duration: 16.25 ms Billed Duration: 17 ms Memory Size: 512 MB Max Memory Used: 96 MB                              |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Please verify the same at your end. (I'm unsure on why curl displays % character at the end of response though).

Thanks,
Ashish

[heyjoey]

Thanks. Fix confirmed.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.