Sam builds only one of several AWS::Serverless::Function with the same CodeUri #3348

komapa · 2021-10-08T20:53:04Z

Description:

We have a repository that has 8 lambda functions which are all managed from the same sam template.yaml file. We are currently migration them from go1.x to the provided.al2 runtime. The problem is that with the new runtime, the behavior of sam build has changed where the same CodeUri no longer works because now you are using the CodeUri as the root of the source code for lambda which was not the case with go1.x.

Simplified directory structure is this:

Makefile
go.mod
go.sum
services
└── queue
    ├── cmd
    │   ├── one
    │   │   └── main.go
    │   ├── two
    │   │   └── main.go
    │   ├── three
    │   │   └── main.go
    │   ├── four
    │   │   └── main.go
    ├── config.go
    ├── const.go
    ├── errors.go

Steps to reproduce:

With runtime go1.x, we had CodeUri: ./services/queue/cmd/one/, CodeUri: ./services/queue/cmd/two/, etc. That would correctly build the main.go file. Now, just changing Runtime: go1.x to Runtime: provided.al2, it fails because CustomMakeBuilder:CopySource only copies files/directories under CodeUri which in our case does not work at all.

Observed result:

So then, I was like, okay makes sense, I will just set the CodeUri for all functions to be the root of the project, and then that succeeds but only the FIRST function is actually built and that executable is copied to all functions because I think you are de-duping on CodeUri somewhere?

Expected result:

I then hacked around it and made CodeUri: ./services/queue/cmd/one/.aws-sam, CodeUri: ./services/queue/cmd/two/.aws-sam, etc. and made the .aws-sam directory a symlink to ../../../../, and then all functions were built correctly.

You would ask why .aws-sam because you have hardcoded two directories to be excluded from copying, that is .aws-sam and .git and if I do not call it .aws-sam, it will create an infinite recursion :) (btw, I will be creating another ticket to allow us to extend the directories which are excluded by CustomMakeBuilder:CopySource)

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Big Sir 11.6
sam --version: SAM CLI, version 1.33.0
AWS region: N/A

Add --debug flag to command you are running

The text was updated successfully, but these errors were encountered:

qingchm · 2021-10-14T17:28:30Z

Hi thanks for the report! If this is indeed reproducible and the behaviour is affecting you experience., we will evaluate your feature request with our team!

cap10morgan · 2021-10-15T18:02:12Z

I'm seeing this exact same issue with a custom runtime. If I run sam build RESOURCE_LOGICAL_ID for any of my functions, it does the right thing. However if I just run sam build it builds one of my functions and copies its artifacts to every function's directory in the .aws-sam/build dir. Fixing this would be very helpful!

I have separate build-RESOURCE_LOGICAL_ID make targets in my project root's Makefile and the CodeUri is set to . for all of my function resources. These make targets copy each function's built artifact to the $(ARTIFACTS_DIR) and like I said if I build any of them individually everything works great.

komapa · 2021-12-03T03:23:41Z

Any update folks?

marklauyq · 2023-12-13T09:25:33Z

Still no update on this? the go1.x runtime is about to be deprecated and I am trying to swap over but this issue is causing me a problem

marklauyq · 2023-12-19T09:51:23Z

I decided to separate the go build process from the sam build command by doing the following. During the build process, it will create a custom .build folder with the binaries in their own individual folder and an auto generated Makefile that will copy the binaries to the artifacts folder.

project/
|-- app
|-- cmd
|   |-- function1
|   |	|-- main.go
|   |-- function2
|   |   |-- main.go
|-- deployment
|   |-- lambda
|   |   |-- .build
|   |   |   |-- function1
|   |   |   |   |-- bootstrap 
|   |   |   |   |-- Makefile 
|   |   |   |-- function2
|   |   |   |   |-- bootstrap 
|   |   |   |   |-- Makefile 
|-- Makefile

The base Makefile

GO = CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go
deployment_folder = deployment/lambda/
GOLANG_VERSION = golang:1.19

build:
	$(GO) build -o $(deployment_folder).build/$(fn_name)/bootstrap $(main_go_dir)
	@echo "build-$(fn_name): \n\tcp bootstrap \$$(ARTIFACTS_DIR)" > $(deployment_folder).build/$(fn_name)/Makefile

build-all: 
	# clear out the build directory
	rm -rf $(deployment_folder)/.build

	@make build fn_name=func1 	main_go_dir='./controller/func1/main.go'
	@make build fn_name=func2 	main_go_dir='./controller/func2/main.go'

#Bonus#you can use the following for building locally without installing go
docker-build:
	docker run --rm -v "$(CURDIR)":/usr/src/myapp -w /usr/src/myapp -it $(GOLANG_VERSION) make build-all

the template file looks something like this

func1:
	Type: AWS::Serverless::Function
	    Properties:
	      Handler: bootstrap
	      CodeUri: .build/func1
	      Runtime: provided.al2
...
func2:
	Type: AWS::Serverless::Function
	    Properties:
	      Handler: bootstrap
	      CodeUri: .build/func2
	      Runtime: provided.al2

so all I will do is run the commands in the following order

make build-all
sam build
sam deploy

mndeveci · 2023-12-20T19:03:51Z

Apologies for missing this issue but it seems like this is a duplicate of #3894

You can build multiple Go lambda functions, using the provided runtimes today. Here is an example setup.

.
├── hello-world
│   ├── cmd
│   │   ├── one
│   │   │   └── main.go
│   │   ├── three
│   │   │   └── main.go
│   │   └── two
│   │       └── main.go
│   ├── go.mod
│   ├── go.sum
│   └── main.go
├── samconfig.toml
└── template.yaml

Here is my template.yaml contents;

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app

  Sample SAM Template for sam-app

Globals:
  Function:
    Timeout: 5
    MemorySize: 128

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function 
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: hello-world/cmd/one/
      Handler: bootstrap
      Runtime: provided.al2
  HelloWorldFunction2:
    Type: AWS::Serverless::Function 
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: hello-world/cmd/two/
      Handler: bootstrap
      Runtime: provided.al2
  HelloWorldFunction3:
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: hello-world/cmd/three/
      Handler: bootstrap
      Runtime: provided.al2

If I build and locally invoke my lambda functions I can get desired outputs;

❯ sam build && sam local invoke HelloWorldFunction3 && sam local invoke HelloWorldFunction2 && sam local invoke HelloWorldFunction
Starting Build use cache
Valid cache found, copying previously built resources for following functions (HelloWorldFunction)
Valid cache found, copying previously built resources for following functions (HelloWorldFunction3)
Valid cache found, copying previously built resources for following functions (HelloWorldFunction2)

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
Invoking bootstrap (provided.al2)
Local image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-provided.al2
Building image.........
Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.

Mounting /private/tmp/test-multiple-go/sam-app/.aws-sam/build/HelloWorldFunction3 as /var/task:ro,delegated, inside runtime container
START RequestId: 872c340b-1d8a-4984-b824-53ac0ae7f006 Version: $LATEST
END RequestId: a7046b75-7186-49e2-81e7-5d26e8e11aac
REPORT RequestId: a7046b75-7186-49e2-81e7-5d26e8e11aac	Init Duration: 0.07 ms	Duration: 116.23 ms	Billed Duration: 117 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
{"statusCode": 200, "headers": null, "multiValueHeaders": null, "body": "Hello, world, CDM 3!\n"}
Invoking bootstrap (provided.al2)
Local image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-provided.al2
Building image........
Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.

Mounting /private/tmp/test-multiple-go/sam-app/.aws-sam/build/HelloWorldFunction2 as /var/task:ro,delegated, inside runtime container
START RequestId: cecff7df-05e4-437b-a0a1-e8c78b7fe426 Version: $LATEST
END RequestId: 456d65b5-d1e9-423e-9d2e-345a9d6f574a
REPORT RequestId: 456d65b5-d1e9-423e-9d2e-345a9d6f574a	Init Duration: 0.09 ms	Duration: 109.33 ms	Billed Duration: 110 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
{"statusCode": 200, "headers": null, "multiValueHeaders": null, "body": "Hello, world, CMD 2!\n"}
Invoking bootstrap (provided.al2)
Local image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-provided.al2
Building image............
Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.

Mounting /private/tmp/test-multiple-go/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: 497070c6-fd54-4ed3-8298-abf2290b358e Version: $LATEST
END RequestId: 3a79cd14-42e4-4845-9268-ba4318de45ff
REPORT RequestId: 3a79cd14-42e4-4845-9268-ba4318de45ff	Init Duration: 0.09 ms	Duration: 139.59 ms	Billed Duration: 140 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
{"statusCode": 200, "headers": null, "multiValueHeaders": null, "body": "Hello, world from CDM 1!\n"}

Couple of things when coming from the go1.x runtime;

Since it will be using provided.al2 or provided.al2023 runtimes, build method should be defined inside the Metadata of the resource.
Handler should be named as bootstrap as it is required by provided runtimes
As mentioned in the initial comment of this issue, CodeUri of the each lambda function should point to the folder where their main.go file resides.
Cached builds had some issues which is patched in v1.101.0 version, please update to the latest SAM CLI version (see: Bug: sam build multiple lambdas only copies first handler (Golang) #6117)

marklauyq · 2023-12-26T04:22:30Z

hi @mndeveci
I was not able to replicate your success on my own project.
I was just testing your template with a single function project first to see if it worked and already ran into the same issue as the OP where the sam build mounts the codeURI folder. This is an issue because the go.mod file is not located in the CodeURI folder meaning the build process cannot pick it up.

...
Resources:
  LogStreamFunc:
    Type: AWS::Serverless::Function
    Properties:
      Handler: bootstrap
      CodeUri: cmd/log_stream
      Runtime: provided.al2
      Tracing: Active
      MemorySize: 128
      Timeout: 120
      Role: !GetAtt LambdaRole.Arn
    Metadata:
      BuildMethod: go1.x
...

The build command I am using is this. I had to use the build image because it was trying to use the provided.al2 image which does not have the go command.

sam build  -u --build-image public.ecr.aws/sam/build-go1.x

output

Starting Build inside a container                                                                                                                
Building codeuri: /Users/marklau/src/aws-log-stream/cmd/log_stream runtime: provided.al2 metadata: {'BuildMethod': 'go1.x'} architecture:  
x86_64 functions: LogStreamFunc                                                                                                                  

Fetching public.ecr.aws/sam/build-go1.x:latest Docker container image....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Mounting /Users/marklau/src/aws-log-stream/cmd/log_stream as /tmp/samcli/source:ro,delegated, inside runtime container                     

Build Failed
Workflow GoModulesBuilder does not support value "False" for building in source. Using default value "True".
 Running GoModulesBuilder:Build
Error: GoModulesBuilder:Build - Builder Failed: go: go.mod file not found in current directory or any parent directory; see 'go help modules'
make: *** [test] Error 1

~~If it is possible, can you try running your sam build with the --no-cached flag and post the result and maybe try loading some libraries through the go.mod.~~

--UPDATED---

I decided to try running without the --use-container by spinning up a new environment for it and it seems to work as you showed.

mndeveci · 2023-12-26T07:33:22Z

@marklauyq thanks for the feedback, can you create a new issue for the problem that you are facing? We can take a closer look at it later.

github-actions · 2023-12-26T07:33:38Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

koote · 2024-01-21T08:18:37Z

Apologies for missing this issue but it seems like this is a duplicate of #3894

You can build multiple Go lambda functions, using the provided runtimes today. Here is an example setup.

.
├── hello-world
│   ├── cmd
│   │   ├── one
│   │   │   └── main.go
│   │   ├── three
│   │   │   └── main.go
│   │   └── two
│   │       └── main.go
│   ├── go.mod
│   ├── go.sum
│   └── main.go
├── samconfig.toml
└── template.yaml

Here is my template.yaml contents;

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app

  Sample SAM Template for sam-app

Globals:
  Function:
    Timeout: 5
    MemorySize: 128

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function 
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: hello-world/cmd/one/
      Handler: bootstrap
      Runtime: provided.al2
  HelloWorldFunction2:
    Type: AWS::Serverless::Function 
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: hello-world/cmd/two/
      Handler: bootstrap
      Runtime: provided.al2
  HelloWorldFunction3:
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: hello-world/cmd/three/
      Handler: bootstrap
      Runtime: provided.al2

If I build and locally invoke my lambda functions I can get desired outputs;

❯ sam build && sam local invoke HelloWorldFunction3 && sam local invoke HelloWorldFunction2 && sam local invoke HelloWorldFunction
Starting Build use cache
Valid cache found, copying previously built resources for following functions (HelloWorldFunction)
Valid cache found, copying previously built resources for following functions (HelloWorldFunction3)
Valid cache found, copying previously built resources for following functions (HelloWorldFunction2)

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
Invoking bootstrap (provided.al2)
Local image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-provided.al2
Building image.........
Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.

Mounting /private/tmp/test-multiple-go/sam-app/.aws-sam/build/HelloWorldFunction3 as /var/task:ro,delegated, inside runtime container
START RequestId: 872c340b-1d8a-4984-b824-53ac0ae7f006 Version: $LATEST
END RequestId: a7046b75-7186-49e2-81e7-5d26e8e11aac
REPORT RequestId: a7046b75-7186-49e2-81e7-5d26e8e11aac	Init Duration: 0.07 ms	Duration: 116.23 ms	Billed Duration: 117 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
{"statusCode": 200, "headers": null, "multiValueHeaders": null, "body": "Hello, world, CDM 3!\n"}
Invoking bootstrap (provided.al2)
Local image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-provided.al2
Building image........
Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.

Mounting /private/tmp/test-multiple-go/sam-app/.aws-sam/build/HelloWorldFunction2 as /var/task:ro,delegated, inside runtime container
START RequestId: cecff7df-05e4-437b-a0a1-e8c78b7fe426 Version: $LATEST
END RequestId: 456d65b5-d1e9-423e-9d2e-345a9d6f574a
REPORT RequestId: 456d65b5-d1e9-423e-9d2e-345a9d6f574a	Init Duration: 0.09 ms	Duration: 109.33 ms	Billed Duration: 110 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
{"statusCode": 200, "headers": null, "multiValueHeaders": null, "body": "Hello, world, CMD 2!\n"}
Invoking bootstrap (provided.al2)
Local image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-provided.al2
Building image............
Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.

Mounting /private/tmp/test-multiple-go/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: 497070c6-fd54-4ed3-8298-abf2290b358e Version: $LATEST
END RequestId: 3a79cd14-42e4-4845-9268-ba4318de45ff
REPORT RequestId: 3a79cd14-42e4-4845-9268-ba4318de45ff	Init Duration: 0.09 ms	Duration: 139.59 ms	Billed Duration: 140 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
{"statusCode": 200, "headers": null, "multiValueHeaders": null, "body": "Hello, world from CDM 1!\n"}

Couple of things when coming from the go1.x runtime;

* Since it will be using `provided.al2` or `provided.al2023` runtimes, build method should be defined inside the Metadata of the resource.

* Handler should be named as `bootstrap` as it is required by provided runtimes

* As mentioned in the initial comment of this issue, `CodeUri` of the each lambda function should point to the folder where their `main.go` file resides.

* Cached builds had some issues which is patched in v1.101.0 version, please update to the latest SAM CLI version (see: [Bug: sam build multiple lambdas only copies first handler (Golang) #6117](https://github.com/aws/aws-sam-cli/issues/6117))

@mndeveci So actually the fix is introducing a new value go1.x to BuildMethod property, correct? With this new value, the source will not be copied to temp folder (behavior of CustomMakeBuilder) but instead, like running go build ./.. from root folder? Is it recommended to use BuildMethod: go1.x instead of BuildMethod: makefile when upgrading golang Lambda to provided.al2? Thanks.

mndeveci · 2024-01-23T02:08:19Z

So actually the fix is introducing a new value go1.x to BuildMethod property, correct? With this new value, the source will not be copied to temp folder (behavior of CustomMakeBuilder) but instead, like running go build ./.. from root folder? Is it recommended to use BuildMethod: go1.x instead of BuildMethod: makefile when upgrading golang Lambda to provided.al2? Thanks.

@koote I think it depends for your case. If BuildMethod: go1.x works for you, I would recommend it since it won't require Makefile.

koote · 2024-01-23T02:17:54Z

So actually the fix is introducing a new value go1.x to BuildMethod property, correct? With this new value, the source will not be copied to temp folder (behavior of CustomMakeBuilder) but instead, like running go build ./.. from root folder? Is it recommended to use BuildMethod: go1.x instead of BuildMethod: makefile when upgrading golang Lambda to provided.al2? Thanks.

@koote I think it depends for your case. If BuildMethod: go1.x works for you, I would recommend it since it won't require Makefile.

Thanks, when using BuildMethod: go1.x, how do I specify go build options like -tags lambda.norpc?

mndeveci · 2024-01-23T18:46:56Z

@koote default go1.x builder is not adding that flag. In this case you need to use Makefile builds. If building in temporary directory is hurting the build time, you can use --build-in-source flag to build in the same directory (which should improve the build times).

aidansteele · 2024-01-24T01:11:05Z

Another option is to use the GOFLAGS env var when running sam build, e.g. GOFLAGS=-tags lambda.norpc. This is how I shrink my Go Lambda packages without having to use Makefile builds in SAM.

koote · 2024-01-24T20:19:04Z

Another option is to use the GOFLAGS env var when running sam build, e.g. GOFLAGS=-tags lambda.norpc. This is how I shrink my Go Lambda packages without having to use Makefile builds in SAM.

Thanks @aidansteele, will setting the flag in GOFLAGS global environment variable affect other Go projects which are not Lambda functions?
BTW according to this, the lambda.norpc seems doesn't give any significant improvements on size of binary or cold start time of Lambda, is it expected? @mndeveci Thanks!

qingchm added the area/build sam build command label Oct 8, 2021

komapa mentioned this issue Oct 8, 2021

Allow customization of which files to exclude during CustomMakeBuilder:CopySource #3350

Closed

aws-sam-cli-bot added the maintainer/need-response label Oct 11, 2021

aws-sam-cli-bot removed the maintainer/need-response label Oct 14, 2021

aws-sam-cli-bot added the maintainer/need-followup label Oct 18, 2021

aws-sam-cli-bot added maintainer/need-followup and removed maintainer/need-followup labels Dec 3, 2021

mndeveci added the blocked/close-if-inactive Blocked for >14 days with no response, will be closed if still inactive after 7 days label Dec 20, 2023

mndeveci closed this as completed Dec 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sam builds only one of several AWS::Serverless::Function with the same CodeUri #3348

Sam builds only one of several AWS::Serverless::Function with the same CodeUri #3348

komapa commented Oct 8, 2021 •

edited

Loading

qingchm commented Oct 14, 2021

cap10morgan commented Oct 15, 2021 •

edited

Loading

komapa commented Dec 3, 2021

marklauyq commented Dec 13, 2023

marklauyq commented Dec 19, 2023

mndeveci commented Dec 20, 2023

marklauyq commented Dec 26, 2023 •

edited

Loading

mndeveci commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

koote commented Jan 21, 2024 •

edited

Loading

mndeveci commented Jan 23, 2024

koote commented Jan 23, 2024

mndeveci commented Jan 23, 2024

aidansteele commented Jan 24, 2024

koote commented Jan 24, 2024 •

edited

Loading

Sam builds only one of several AWS::Serverless::Function with the same CodeUri #3348

Sam builds only one of several AWS::Serverless::Function with the same CodeUri #3348

Comments

komapa commented Oct 8, 2021 • edited Loading

Description:

Steps to reproduce:

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

qingchm commented Oct 14, 2021

cap10morgan commented Oct 15, 2021 • edited Loading

komapa commented Dec 3, 2021

marklauyq commented Dec 13, 2023

marklauyq commented Dec 19, 2023

mndeveci commented Dec 20, 2023

marklauyq commented Dec 26, 2023 • edited Loading

mndeveci commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

koote commented Jan 21, 2024 • edited Loading

mndeveci commented Jan 23, 2024

koote commented Jan 23, 2024

mndeveci commented Jan 23, 2024

aidansteele commented Jan 24, 2024

koote commented Jan 24, 2024 • edited Loading

komapa commented Oct 8, 2021 •

edited

Loading

cap10morgan commented Oct 15, 2021 •

edited

Loading

marklauyq commented Dec 26, 2023 •

edited

Loading

koote commented Jan 21, 2024 •

edited

Loading

koote commented Jan 24, 2024 •

edited

Loading