IOutputCache does not work with Lambda runtime

[tvytrykush]

Describe the bug

When using IOutputCache feature of asp.net core the cached value is not populated from the response body and is empty

Expected Behavior

Response's cache value is properly cached from response body alongside the key(that is cached properly)

Current Behavior

Because empty value is being cached with valid key - on subsequent requests using same path the response body is going to be empty.

Reproduction Steps

1. Create new project using "Lambda ASP.NET Core Web API" template
2. Get redis instance running
3. Add Output cache to service collection in Startup:

services
            .AddOutputCache(options =>
                options.AddBasePolicy(builder =>
                        builder.AddPolicy<OutputCacheRedisPolicy>().Cache(),
                    true))
            .AddStackExchangeRedisOutputCache(opts =>
            {
                opts.Configuration = "localhost:<redis-port>";
                opts.InstanceName = "lambda";
            });

4. Add app.UseOutputCache(); before app.UseEndpoints in Configure
5. Add IOutputCachePolicy implementation class into project:

public class OutputCacheRedisPolicy : IOutputCachePolicy
{
    private const string TagPrefix = "response-cache-";
    private static readonly TimeSpan CacheExpirationPeriod = TimeSpan.FromMinutes(15);

    ValueTask IOutputCachePolicy.CacheRequestAsync(OutputCacheContext context, CancellationToken cancellationToken)
    {
        var attemptOutputCaching = AttemptOutputCaching(context);
        
        context.EnableOutputCaching = true;
        context.AllowCacheLookup = attemptOutputCaching;
        context.AllowCacheStorage = attemptOutputCaching;
        context.AllowLocking = true;
        context.ResponseExpirationTimeSpan = CacheExpirationPeriod;
        context.CacheVaryByRules.QueryKeys = "*";

        context.Tags.Add(TagPrefix + context.HttpContext.Request.RouteValues["controller"]);
        
        return ValueTask.CompletedTask;
    }
    
    private static bool AttemptOutputCaching(OutputCacheContext context)
    {
        var request = context.HttpContext.Request;

        return HttpMethods.IsGet(request.Method) || HttpMethods.IsHead(request.Method);
    }
    
    public ValueTask ServeFromCacheAsync(OutputCacheContext context, CancellationToken cancellation) 
        => ValueTask.CompletedTask;

    public ValueTask ServeResponseAsync(OutputCacheContext context, CancellationToken cancellation)
    {
        var response = context.HttpContext.Response;

        // Verify existence of cookie headers
        if (!StringValues.IsNullOrEmpty(response.Headers.SetCookie))
        {
            context.AllowCacheStorage = false;
            return ValueTask.CompletedTask;
        }

        // Check response code
        if (response.StatusCode is StatusCodes.Status200OK or StatusCodes.Status301MovedPermanently)
        {
            return ValueTask.CompletedTask;
        }
        
        context.AllowCacheStorage = false;
        return ValueTask.CompletedTask;
    }
}

6. Run with the AWS Test Mock Tool (or any tool running within Lambda runtime)
7. Send GET request to /api/values -> value will be cached
8. Send GET request again to /api/values -> response body is empty -> value stored for cached key in redis is empty

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

Amazon.Lambda.AspNetCoreServer v9.0.0
Microsoft.AspNetCore.OutputCaching.StackExchangeRedis v8.0.0

Targeted .NET Platform

.NET 8

Operating System and version

MacOS Sonoma 14.2.1

[ashishdhingra]

@tvytrykush Good morning. I do not think you would be able to test the IOutputCache feature with the Lambda test tool. Lambda Test Tool is recommended for testing simple functional handlers and it would be better to use IIS/IIS Express/Kestrel to test REST endpoints. However, you could refer the guidance provided in #589 (comment) to test API methods locally.

Have you tried testing IOutputCache feature after deploying to actual AWS environment and see if it works (you would need to setup Redis cluster and setup configuration accordingly, including any IAM policies)?

CCing @normj for any additional inputs.

Thanks,
Ashish

[tvytrykush]

Hello @ashishdhingra, that's the point I wanted to emphasize here. Running same app via Kestrel works just fine and correct cache value is being stored and returned in subsequent requests. The problem only arises when running via LambdaEntryPoint and corresponding handler.

And regarding your second point - I've also tried running in live AWS Lambda, but the behavior is completely the same - invalid cache value being stored.

[ashishdhingra]

I've also tried running in live AWS Lambda, but the behavior is completely the same - invalid cache value being stored.

@tvytrykush Good morning. Could you please describe your live AWS Lambda setup, including ElastiCache Redis cluster? I'm assuming there might be some connectivity/permissions issue that you might want to troubleshoot.

Thanks,
Ashish

[tvytrykush]

Hi @ashishdhingra, I'm pretty sure the connectivity is fine, because lambda can connect to redis and it does write something to cache, but not what is expected. But if of any interest here's the config.
Also Redis has security group with inbound rule for Lambda's security group at TCP 6379, and Lambda has outbound rule to any IP:port.
But you should be able to reproduce it when running locally.

and SAM for Lambda:

# This AWS SAM template has been generated from your function's configuration. If
# your function has one or more triggers, note that the AWS resources associated
# with these triggers aren't fully specified in this template and include
# placeholder values. Open this template in AWS Application Composer or your
# favorite IDE and modify it to specify a serverless application with other AWS
# resources.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: An AWS Serverless Application Model template describing your function.
Resources:
  mylambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Description: ''
      MemorySize: 512
      Timeout: 100
      Architectures:
        - x86_64
      EphemeralStorage:
        Size: 512
      Environment:
        Variables:
          ASPNETCORE_ENVIRONMENT: Nonprod
      EventInvokeConfig:
        MaximumEventAgeInSeconds: 21600
        MaximumRetryAttempts: 2
      ImageUri: >-
        <edited>:amd64_latest
      PackageType: Image
      Policies:
        - Statement:
            - Sid: AWSLambdaVPCAccessExecutionPermissions
              Effect: Allow
              Action:
                - logs:CreateLogGroup
                - logs:CreateLogStream
                - logs:PutLogEvents
                - ec2:CreateNetworkInterface
                - ec2:DescribeNetworkInterfaces
                - ec2:DescribeSubnets
                - ec2:DeleteNetworkInterface
                - ec2:AssignPrivateIpAddresses
                - ec2:UnassignPrivateIpAddresses
              Resource: '*'
            - Sid: BasePermissions
              Effect: Allow
              Action:
                - secretsmanager:*
                - cloudformation:CreateChangeSet
                - cloudformation:DescribeChangeSet
                - cloudformation:DescribeStackResource
                - cloudformation:DescribeStacks
                - cloudformation:ExecuteChangeSet
                - docdb-elastic:GetCluster
                - docdb-elastic:ListClusters
                - ec2:DescribeSecurityGroups
                - ec2:DescribeSubnets
                - ec2:DescribeVpcs
                - kms:DescribeKey
                - kms:ListAliases
                - kms:ListKeys
                - lambda:ListFunctions
                - rds:DescribeDBClusters
                - rds:DescribeDBInstances
                - redshift:DescribeClusters
                - redshift-serverless:ListWorkgroups
                - redshift-serverless:GetNamespace
                - tag:GetResources
              Resource: '*'
            - Sid: LambdaPermissions
              Effect: Allow
              Action:
                - lambda:AddPermission
                - lambda:CreateFunction
                - lambda:GetFunction
                - lambda:InvokeFunction
                - lambda:UpdateFunctionConfiguration
              Resource: arn:aws:lambda:*:*:function:SecretsManager*
            - Sid: SARPermissions
              Effect: Allow
              Action:
                - serverlessrepo:CreateCloudFormationChangeSet
                - serverlessrepo:GetApplication
              Resource: arn:aws:serverlessrepo:*:*:applications/SecretsManager*
            - Sid: S3Permissions
              Effect: Allow
              Action:
                - s3:GetObject
              Resource: '*'
            - Effect: Allow
              Action:
                - logs:CreateLogGroup
                - logs:CreateLogStream
                - logs:PutLogEvents
              Resource: '*'
            - Sid: ElastiCacheManagementActions
              Effect: Allow
              Action:
                - elasticache:*
              Resource: '*'
            - Sid: CreateServiceLinkedRole
              Effect: Allow
              Action:
                - iam:CreateServiceLinkedRole
              Resource: >-
                arn:aws:iam::*:role/aws-service-role/elasticache.amazonaws.com/AWSServiceRoleForElastiCache
              Condition:
                StringLike:
                  iam:AWSServiceName: elasticache.amazonaws.com
            - Sid: CreateVPCEndpoints
              Effect: Allow
              Action:
                - ec2:CreateVpcEndpoint
              Resource: arn:aws:ec2:*:*:vpc-endpoint/*
              Condition:
                StringLike:
                  ec2:VpceServiceName: com.amazonaws.elasticache.serverless.*
            - Sid: AllowAccessToElastiCacheTaggedVpcEndpoints
              Effect: Allow
              Action:
                - ec2:CreateVpcEndpoint
              NotResource: arn:aws:ec2:*:*:vpc-endpoint/*
            - Sid: TagVPCEndpointsOnCreation
              Effect: Allow
              Action:
                - ec2:CreateTags
              Resource: arn:aws:ec2:*:*:vpc-endpoint/*
              Condition:
                StringEquals:
                  ec2:CreateAction: CreateVpcEndpoint
                  aws:RequestTag/AmazonElastiCacheManaged: 'true'
            - Sid: AllowAccessToEc2
              Effect: Allow
              Action:
                - ec2:DescribeVpcs
                - ec2:DescribeSubnets
                - ec2:DescribeSecurityGroups
              Resource: '*'
            - Sid: AllowAccessToKMS
              Effect: Allow
              Action:
                - kms:DescribeKey
                - kms:ListAliases
                - kms:ListKeys
              Resource: '*'
            - Sid: AllowAccessToCloudWatch
              Effect: Allow
              Action:
                - cloudwatch:GetMetricStatistics
                - cloudwatch:GetMetricData
              Resource: '*'
            - Sid: AllowAccessToAutoScaling
              Effect: Allow
              Action:
                - application-autoscaling:DescribeScalableTargets
                - application-autoscaling:DescribeScheduledActions
                - application-autoscaling:DescribeScalingPolicies
                - application-autoscaling:DescribeScalingActivities
              Resource: '*'
            - Sid: DescribeLogGroups
              Effect: Allow
              Action:
                - logs:DescribeLogGroups
              Resource: '*'
            - Sid: ListLogDeliveryStreams
              Effect: Allow
              Action:
                - firehose:ListDeliveryStreams
              Resource: '*'
            - Sid: DescribeS3Buckets
              Effect: Allow
              Action:
                - s3:ListAllMyBuckets
              Resource: '*'
            - Sid: AllowAccessToOutposts
              Effect: Allow
              Action:
                - outposts:ListOutposts
              Resource: '*'
            - Sid: AllowAccessToSNS
              Effect: Allow
              Action:
                - sns:ListTopics
              Resource: '*'
      ReservedConcurrentExecutions: 10
      SnapStart:
        ApplyOn: None
      VpcConfig:
        SecurityGroupIds:
          - sg-<edited>
        SubnetIds:
          - subnet-<edited>
          - subnet-<edited>
          - subnet-<edited>
        Ipv6AllowedForDualStack: false
      Events:
        Api1:
          Type: Api
          Properties:
            Path: /{proxy+}
            Method: ANY

[normj]

Since you called out you have a LambdaEntryPoint class I assume you were using the Startup initialization style. With my startup of

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    // This method gets called by the runtime. Use this method to add services to the container
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddControllers();

        services.AddOutputCache(options =>
                options.AddBasePolicy(builder =>
                        builder.AddPolicy<OutputCacheRedisPolicy>().Cache(),
                    true))
            .AddStackExchangeRedisOutputCache(opts =>
            {
                opts.Configuration = "localhost:6379";
                opts.InstanceName = "lambda";
            });
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseHttpsRedirection();

        app.UseOutputCache();

        app.UseRouting();

        app.UseAuthorization();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
            endpoints.MapGet("/", async context =>
            {
                await context.Response.WriteAsync($"{DateTime.Now}: Welcome to running ASP.NET Core on AWS Lambda");
            });
        });
    }
}

and using your OutputCacheRedisPolicy. I was able to see the the output cache being used via the Lambda test tool using the following event and running the Redis container locally.

{
  "body": null,
  "resource": "/{proxy+}",
  "path": "/",
  "httpMethod": "GET",
  "queryStringParameters": {
    "foo": "bar"
  },
  "pathParameters": {
    "proxy": ""
  },
  "stageVariables": {
    "baz": "qux"
  },
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "en-US,en;q=0.8",
    "Cache-Control": "max-age=0",
    "CloudFront-Forwarded-Proto": "https",
    "CloudFront-Is-Desktop-Viewer": "true",
    "CloudFront-Is-Mobile-Viewer": "false",
    "CloudFront-Is-SmartTV-Viewer": "false",
    "CloudFront-Is-Tablet-Viewer": "false",
    "CloudFront-Viewer-Country": "US",
    "Host": "1234567890.execute-api.{dns_suffix}",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Custom User Agent String",
    "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)",
    "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==",
    "X-Forwarded-For": "127.0.0.1, 127.0.0.2",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https"
  },
  "requestContext": {
    "accountId": "123456789012",
    "resourceId": "123456",
    "stage": "prod",
    "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
    "identity": {
      "cognitoIdentityPoolId": null,
      "accountId": null,
      "cognitoIdentityId": null,
      "caller": null,
      "apiKey": null,
      "sourceIp": "127.0.0.1",
      "cognitoAuthenticationType": null,
      "cognitoAuthenticationProvider": null,
      "userArn": null,
      "userAgent": "Custom User Agent String",
      "user": null
    },
    "resourcePath": "/{proxy+}",
    "httpMethod": "POST",
    "apiId": "1234567890"
  }
}

[tvytrykush]

@normj Actually I've just tried root path as you're using, but it's completely the same, you get valid response on the first request and invalid on all subsequent requests. It's same as using (for reference) the api/values path and my first response (in Lambda test tool) is:

{"statusCode":200,"headers":{},"multiValueHeaders":{"Content-Type":["application/json; charset=utf-8"],"Date":["Tue, 26 Mar 2024 16:48:07 GMT"]},"body":"[\"value1\",\"value2\"]","isBase64Encoded":false}

and when I execute again:

{"statusCode":200,"headers":{},"multiValueHeaders":{"Content-Length":["0"],"Content-Type":["application/json; charset=utf-8"],"Date":["Tue, 26 Mar 2024 16:48:07 GMT"],"Age":["39"]},"body":"","isBase64Encoded":false}

Values stored in cache:

• key: test-lambda__MSOCV_GET�HTTPS�LOCALHOST:5001/API/VALUES�Q�*=
• value: \x02\xe4\xb6\xfc\x9b\xc8\xb6\x93\xee\b\x00\xc8\x01\x02;\x01>application/json; charset=utf-8A\x01:Tue, 26 Mar 2024 16:48:07 GMT\x00

Don't you ming checking what's stored in the cache value?