ec2.Vpc.fromLookup: Unable to find VPC by matching tags (2.167.x)

[paulthiele1311]

Describe the bug

We noticed that with versions 2.167.x we are not able to do a lookup on the VPC based on tags.
What we can see when deploying with -vv, is that the lookup is done using random values that are not provided by ourselves.

[15:42:18] Listing VPCs in <ACCOUNT_ID>:eu-central-1
[15:42:19] Setting "vpc-provider:account=<ACCOUNT_ID>:filter.vpc-id=vpc-12345:region=eu-central-1:returnAsymmetricSubnets=true" context to {"$providerError":"Could not find any VPCs matching {"account":"<ACCOUNT_ID>","region":"eu-central-1","filter":{"vpc-id":"vpc-12345"},"returnAsymmetricSubnets":true,"lookupRoleArn":"arn:aws:iam::<ACCOUNT_ID>:role/cdk-hnb659fds-lookup-role-<ACCOUNT_ID>-eu-central-1"}","$dontSaveContext":true}

As you can see CDK is trying to find the VPC using the vpc-id "vpc-12345", which is clearly not from us.

The lookup we are performing is the following:

lookupVpc(props: ServiceStackProps) {
  return ec2.Vpc.fromLookup(this, 'VpcLookup', {
    tags: {
      vertical: props.vertical,
      environment: props.stage,
    },
  })
}

What we noticed is that when looking up the ALB, CDK produces this output:

[12:47:29] Assuming role 'arn:aws:iam::<ACCOUNT_ID>:role/cdk-hnb659fds-lookup-role-<ACCOUNT_ID>-eu-central-1'.
[12:47:30] [trace] SDK#elbv2()
[12:47:31] Setting "load-balancer:account=<ACCOUNT_ID>:loadBalancerTags.0.key=environment:loadBalancerTags.0.value=develop:loadBalancerTags.1.key=Name:loadBalancerTags.1.value=ALB order-mgmt develop:loadBalancerTags.2.key=scheme:loadBalancerTags.2.value=Internet-facing:loadBalancerTags.3.key=team:loadBalancerTags.3.value=ft4:loadBalancerTags.4.key=vertical:loadBalancerTags.4.value=order-mgmt:loadBalancerType=application:region=eu-central-1" context to {"loadBalancerArn":"<LOAD_BALANCER_ARN>","loadBalancerCanonicalHostedZoneId":"<ZONE_ID>","loadBalancerDnsName":"<DNS_NAME>","vpcId":"<CORRECT_VPC_ID>","securityGroupIds":[<CORRECT_SECURITY_GROUPS>],"ipAddressType":"dualstack"}

What's interesting here is that (fixed in 2.167.2*) the lookup for ALB works fine and also gets the correct VPC and Security Groups. Currently our workaround to get the VPC (and therefore the Security Groups) is to reference it from the ALB:

this.publicAlb = this.lookupAlb(props, 'AlbLookupPublicAlb', 'Internet-facing')
this.internalAlb = this.lookupAlb(props, 'AlbLookupInternalAlb', 'Internal')
this.vpc = this.publicAlb.vpc!! <-- VPC Lookup not working, so we need to do this.
this.ecsSecurityGroup = ec2.SecurityGroup.fromLookupByName(this, 'ecsSecurityGroupLookUp', 'ecsSecurityGroup', this.vpc)

The problem when not finding the VPC of course is, that we also can't find the Security Groups.

Even though in the end the ALB is found we see outputs like these:

[12:48:00] Setting "load-balancer:account=<ACCOUNT_ID>:loadBalancerTags.0.key=environment:loadBalancerTags.0.value=develop:loadBalancerTags.1.key=Name:loadBalancerTags.1.value=ALB order-mgmt develop:loadBalancerTags.2.key=scheme:loadBalancerTags.2.value=Internal:loadBalancerTags.3.key=team:loadBalancerTags.3.value=ft4:loadBalancerTags.4.key=vertical:loadBalancerTags.4.value=order-mgmt:loadBalancerType=application:region=eu-central-1" context to {"$providerError":"No load balancers found matching {"dummyValue":{"ipAddressType":"dualstack","loadBalancerArn":"arn:aws:elasticloadbalancing:us-west-2:123456789012:loadbalancer/application/my-load-balancer/50dc6c495c0c9188","loadBalancerCanonicalHostedZoneId":"Z3DZXE0EXAMPLE","loadBalancerDnsName":"my-load-balancer-1234567890.us-west-2.elb.amazonaws.com","securityGroupIds":["sg-1234"],"vpcId":"vpc-12345"},"account":"<ACCOUNT_ID>","region":"eu-central-1","loadBalancerTags":[{"key":"environment","value":"develop"},{"key":"Name","value":"ALB order-mgmt develop"},{"key":"scheme","value":"Internal"},{"key":"team","value":"ft4"},{"key":"vertical","value":"order-mgmt"}],"loadBalancerType":"application","lookupRoleArn":"arn:aws:iam::<ACCOUNT_ID>:role/cdk-hnb659fds-lookup-role-<ACCOUNT_ID>-eu-central-1"}","$dontSaveContext":true}

'security-group:account=<ACCOUNT_ID>:region=eu-central-1:securityGroupName=ecsSecurityGroup:vpcId=vpc-12345': {
'$providerError': 'No security groups found matching {"dummyValue":{"securityGroupId":"sg-12345678","allowAllOutbound":true},"account":"<ACCOUNT_ID>","region":"eu-central-1","securityGroupName":"ecsSecurityGroup","vpcId":"vpc-12345","lookupRoleArn":"arn:aws:iam::<ACCOUNT_ID>:role/cdk-hnb659fds-lookup-role-<ACCOUNT_ID>-eu-central-1"}',
'$dontSaveContext': true
},

* There was another issue with looking up Load Balancers, which is already fixed:
#32161

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

2.166.0

Expected Behavior

The expected behavior is that our lookup function which calls ec2.Vpc.fromLookup is finding the correct VPC using the correct passed tags.

Current Behavior

Currently the ec2.Vpc.fromLookup method is behaving not intended and is using random ("dummy") values and therefore can't find a VPC which blocks the deployment.

Reproduction Steps

1. Use cdk-cli 2.167.2
2. Create a VPC with different tags
3. Try to lookup the VPC using the following method:

lookupVpc(props: ServiceStackProps) {
  return ec2.Vpc.fromLookup(this, 'VpcLookup', {
    tags: {
      vertical: props.vertical,
      environment: props.stage,
    },
  })
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.167.2

Framework Version

No response

Node.js Version

20.18.0

OS

Ubuntu 22.04.5 LTS (on GitHub Runner)

Language

TypeScript

Language Version

No response

Other information

No response

[ashishdhingra]

@paulthiele1311 Good morning. Thanks for reporting the issue. I tested using CDK version 2.165.0 and current latest 2.168.0, using code below:

import * as cdk from 'aws-cdk-lib';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

export class CdktestStackNew extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const testVpcLookup =  ec2.Vpc.fromLookup(this, 'VpcLookup', {
      tags: {
        vertical: "testvertical",
        environment: "testenvironment",
      },
    })
    
    console.log(testVpcLookup.vpcId);
  }
}

Running cdk synth returns the following:

vpc-12345
vpc-04c4f1a40d0XXXXXX
Resources:
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Analytics: v2:deflate64:H4sIAAAAAAAA/zPSMzSz0DNQTCwv1k1OydbNyUzSqw4uSUzO1glKLc4vLUpO1XFOy4Oxa3Xy8lNS9bKK9csMLfSMDPRMFLOKMzN1i0rzSjJzU/WCIDQAP6I4o1YAAAA=
    Metadata:
      aws:cdk:path: CdktestStackNew/CDKMetadata/Default
Parameters:
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]

The reason it returns vpc-12345 first is because lookups are usually done via context. When we run cdk synth and looking up for VPC for the first time, the VPC is not yet available in context; in other words cdk.context.json is empty. Once the lookup is done, it returns the correct VPC Id and cdk.context.json is populated with context values:

{
  "vpc-provider:account=<<ACCOUNT_ID>>:filter.tag:environment=testenvironment:filter.tag:vertical=testvertical:region=us-east-2:returnAsymmetricSubnets=true": {
    "vpcId": "vpc-04c4f1a40d0XXXXXX",
    "vpcCidrBlock": "10.0.0.0/16",
    "ownerAccountId": "<<ACCOUNT_ID>>",
    "availabilityZones": [],
    "subnetGroups": [
      {
        "name": "Private",
        "type": "Private",
        "subnets": [
          {
            "subnetId": "subnet-0f2ac00efb1XXXXXX",
            "cidr": "10.0.96.0/19",
            "availabilityZone": "us-east-2a",
            "routeTableId": "rtb-047542de8dXXXXXX"
          },
          {
            "subnetId": "subnet-0973d1b6f4XXXXXX",
            "cidr": "10.0.128.0/19",
            "availabilityZone": "us-east-2b",
            "routeTableId": "rtb-0a6a99913bXXXXXX"
          },
          {
            "subnetId": "subnet-0831ed5867eXXXXXX",
            "cidr": "10.0.160.0/19",
            "availabilityZone": "us-east-2c",
            "routeTableId": "rtb-0a055f39223XXXXXX"
          }
        ]
      },
      {
        "name": "Public",
        "type": "Public",
        "subnets": [
          {
            "subnetId": "subnet-01d2dc3b68eXXXXXX",
            "cidr": "10.0.0.0/19",
            "availabilityZone": "us-east-2a",
            "routeTableId": "rtb-0bf9d36b94XXXXXX"
          },
          {
            "subnetId": "subnet-0255a18d22cXXXXXX",
            "cidr": "10.0.32.0/19",
            "availabilityZone": "us-east-2b",
            "routeTableId": "rtb-047d0e4f0eXXXXXX"
          },
          {
            "subnetId": "subnet-0a8f926ec8XXXXXX",
            "cidr": "10.0.64.0/19",
            "availabilityZone": "us-east-2c",
            "routeTableId": "rtb-003d2ad607XXXXXX"
          }
        ]
      }
    ]
  }
}

Refer to #14066 (comment) for possible workaround to pre-populate CDK context before executing lookup on main stack.

Thanks,
Ashish

[pahud]

Hi @paulthiele1311

export class DummyStack2 extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    function lookupVpc(scope: Construct) {
      return ec2.Vpc.fromLookup(scope, 'VpcLookup', {
        vpcId: 'vpc-1f5b7e78'
      })
    }

    const vpc = lookupVpc(this);
    console.log("VpcId=" + vpc.vpcId);
    new CfnOutput(this, 'VpcId', { value: vpc.vpcId });

  }
}

% rm -f cdk.context.json   
% npx cdk synth 2>&1 | grep "VpcId"
VpcId=vpc-12345
VpcId=vpc-1f5b7e78

% npx cdk synth  | grep -A2 Outputs
Outputs:
  VpcId:
    Value: vpc-1f5b7e78

% npx cdk --version                 
2.168.0 (build f4a02ab)

In CDK, some from* method would use context provider to invoke SDK calls to query resources if they are not cached in the context(cdk.context.json). In synth stage, CDK would first insert a placeholder(vpc-12345 in this case) for the first attempt and get it replaced with the real value when resolved.

[paulthiele1311]

Hi @ashishdhingra and @pahud, thank you for your answers.
But the problem is not only arising when synthesizing, but when the deployment is going on. Maybe I was not detailed enough, but the synth is not the problem, the problem is that it can't find and uses these values when deploying.

[ashishdhingra]

Hi @ashishdhingra and @pahud, thank you for your answers. But the problem is not only arising when synthesizing, but when the deployment is going on. Maybe I was not detailed enough, but the synth is not the problem, the problem is that it can't find and uses these values when deploying.

@paulthiele1311 Have you tried using workaround in #14066 (comment)? I'm unsure if we could change the behavior of ec2.Vpc.fromLookup() since it always loads from CDK context.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[munch9]

Although the exact error is different this feels related to #32228

While the error we see is proxy timeouts, it is also while trying to fetch VPC lookups for context and both occur on the same version which aligns to the sdkv3 upgrade.

[ashishdhingra]

The issue is different in that the context lookup works once populated initially.