From a57b238c55ae229ce39c5abf1398ec7720fb9359 Mon Sep 17 00:00:00 2001 From: Aaron Yee Date: Tue, 29 Oct 2024 15:15:17 -1000 Subject: [PATCH 1/3] added more context --- README.md | 170 +++++++++++++++++++++++++++++------------------------- 1 file changed, 91 insertions(+), 79 deletions(-) diff --git a/README.md b/README.md index ceaf8d3..5dc7b11 100644 --- a/README.md +++ b/README.md @@ -4,138 +4,150 @@ ![Portage CD Logo](./static/portage-cd-logo.svg) -Portage CD is a secure, continuous delivery pipeline designed to orchestrate the process of building and scanning an -application image for security vulnerabilities. -It solves the problem of having to configure a hardened-predefined security pipeline using traditional CI/CD. -Portage CD can be statically compiled as a binary and run on virtually any platform, CI/CD -environment, or locally. +## Basic Overview + +Portage CD is a secure, continuous delivery pipeline built on open source. Portage CD is designed to orchestrate the process of building and scanning an application image for security vulnerabilities. The unique aspect of Portage CD is that it is meant to be portable, meaning that a developer can run the entire pipeline locally, address any security vulnerabilities or code issues before pushing before pushing a branch to a CI/CD pipeline based in the cloud that is also running Portage CD. + +This project aims to simplify the CI/CD build process such that developers can focus on building their unit and end to end tests, and spend less time working on security pipeline tooling, and standing up continuous delivery (CD). + +Portage CD is designed to make CI/CD easier by providing a preconfigured security and container building pipeline that can be configured such that tools can be overridden where an enterprise SaaS tool may exist. The tool has also been designed so that i can be incorporated into existing CI pipelines running in any CI platform. Portage CD can be statically compiled as a binary and run on virtually any environment or CI/CD platform. + +At it's core, Portage CD is a golang based CLI tool that is configured to chain the security and container building open source tools below to make the CI/CD process easier. ## Getting Started +NOTE: *While Portage CD will easily work with any Windows or Linux environment, the following instructions are done on a MacOS PC environment. In the future we will update this Readme with instructions for other environments.* + Install Prerequisites: -- Container Engine -- Docker or Podman CLI -- Golang >= v1.22.0 +- Container Engine (e.g. [Docker](https://docs.docker.com/desktop/), Openshift, etc.) +- [Docker](https://docs.docker.com/desktop/) or Podman CLI +- Golang >= v1.22.0 ([brew install](https://formulae.brew.sh/formula/go)) - [Just](https://github.com/casey/just?tab=readme-ov-file#installation) (optional) +```bash +brew install go@1.22 just +``` + Prerequisite Tools (For running `portage` local) - Gitleaks -- Semgrep -- Syft -- Grype - ClamAV (only the `clamscan` and `freshclam` cli utilities are needed) - ORAS -## Compiling Portage CD +If you are running on a mac and have brew installed, you can install these dependencies using the following command. -Running the just recipe will put the compiled-binary into `./bin` +```bash +brew install gitleaks semgrep syft grype clamav oras +``` + +Once ClamAV is installed, you will need to configure it to download the virus definitions. This can be done by adding the following lines to your `/etc/freshclam.conf` file. + +``` +# Allow freshclam to run as root +DatabaseMirror database.clamav.net +Foreground yes +``` + +Freschclam will download and run the latest virus definitions every time you run it. WARNING: This will generate a lot of network traffic. Please be aware that if you pull too many times, you will be rate limited by the virus definition servers. + +Run the following command to download the virus definitions. ```bash -just build +freshclam ``` -OR compile manually +## Compiling Portage CD for Local Use + +Clone the repo and navigate to the directory in the shell. ```bash git clone cd mkdir bin -go build -o bin/portage ./cmd/portage ``` -Optionally, if you care to include metadata about the version of `portage` (displayed when you run `portage version`), use the following build arguments +Running the just recipe will put the compiled-binary into `./bin` -```shell +```bash +just build +``` + +OR you can compile Portage CD manually + +```bash go build -ldflags="-X 'main.cliVersion=$(git describe --tags)' -X 'main.gitCommit=$(git rev-parse HEAD)' -X 'main.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)' -X 'main.gitDescription=$(git log -1 --pretty=%B)'" -o ./bin ./cmd/portage ``` +## Running A Pipeline Locally + +First, map the portage binary to your $PATH via ~/.zshrc or copy the binary into your /usr/local/bin. -## Running A Pipeline +```bash +echo 'export PATH="$PATH:/path/to/portage/bin"' >> ~/.zshrc +source ~/.zshrc +``` +OR +```bash +sudo cp /path/to/portage/bin/portage /usr/local/bin/ +``` -You can run the executable directory +Then navigate to the source code directory that you wish to run the pipeline scanning tools on. To run the security pipeline directly, run the following command. ```bash portage run debug ``` -## Configuring a Pipeline +### Configuring a Pipeline Configuration Options: -- Configuration via CLI flags -- Environment Variables -- Config File in JSON -- Config File in YAML -- Config File in TOML +Note: `(none)` means unset, left blank -Configuration Order-of-Precedence: +| Configuration Option | Environment Variable | Default Value | Description | +|-----------------------------------|----------------------------------------|--------------------------------------|----------------------------------------------------------------------------------| +| imagescan.grypefilename | PORTAGE_IMAGE_SCAN_GRYPE_FILENAME | grype-vulnerability-report-full.json | The filename for the grype vulnerability report - must contain 'grype' | +| imagescan.syftfilename | PORTAGE_IMAGE_SCAN_SYFT_FILENAME | syft-sbom-report.json | The filename for the syft SBOM report - must contain 'syft' | -1. CLI Flag -2. Environment Variable -3. Config File Value -4. Default Value +To run a pipeline with all the defaults, run the following command. +```bash +portage run all --tag "ttl.sh/$(uuidgen | tr '[:upper:]' '[:lower:]'):1h" +``` -Note: `(none)` means unset, left blank -| Config Key | Environment Variable | Default Value | Description | -| --------------------------------- |------------------------------------------| ------------------------------------ |------------------------------------------------------------------------------------| -| codescan.enabled | PORTAGE_CODE_SCAN_ENABLED | 1 | Enable/Disable the code scan pipeline | -| codescan.gitleaksfilename | PORTAGE_CODE_SCAN_GITLEAKS_FILENAME | gitleaks-secrets-report.json | The filename for the gitleaks secret report - must contain 'gitleaks' | -| codescan.gitleakssrcdir | PORTAGE_CODE_SCAN_GITLEAKS_SRC_DIR | . | The target directory for the gitleaks scan | -| codescan.semgrepfilename | PORTAGE_CODE_SCAN_SEMGREP_FILENAME | semgrep-sast-report.json | The filename for the semgrep SAST report - must contain 'semgrep' | -| codescan.semgreprules | PORTAGE_CODE_SCAN_SEMGREP_RULES | p/default | Semgrep ruleset manual override | -| codescan.semgrepexperimental | PORTAGE_CODE_SCAN_SEMGREP_EXPERIMENTAL | false | Enable the use of the semgrep experimental CLI | -| deploy.enabled | PORTAGE_IMAGE_PUBLISH_ENABLED | 1 | Enable/Disable the deploy pipeline | -| deploy.gatecheckconfigfilename | PORTAGE_DEPLOY_GATECHECK_CONFIG_FILENAME | - | The filename for the gatecheck config | -| gatecheckbundlefilename | PORTAGE_GATECHECK_BUNDLE_FILENAME | artifacts/gatecheck-bundle.tar.gz | The filename for the gatecheck bundle, a validatable archive of security artifacts | -| imagebuild.args | PORTAGE_IMAGE_BUILD_ARGS | - | Comma seperated list of build time variables | -| imagebuild.builddir | PORTAGE_IMAGE_BUILD_DIR | . | The build directory to using during an image build | -| imagebuild.cachefrom | PORTAGE_IMAGE_BUILD_CACHE_FROM | - | External cache sources (e.g., "user/app:cache", "type=local,src=path/to/dir") | -| imagebuild.cacheto | PORTAGE_IMAGE_BUILD_CACHE_TO | - | Cache export destinations (e.g., "user/app:cache", "type=local,src=path/to/dir") | -| imagebuild.dockerfile | PORTAGE_IMAGE_BUILD_DOCKERFILE | Dockerfile | The Dockerfile/Containerfile to use during an image build | -| imagebuild.enabled | PORTAGE_IMAGE_BUILD_ENABLED | 1 | Enable/Disable the image build pipeline | -| imagebuild.platform | PORTAGE_IMAGE_BUILD_PLATFORM | - | The target platform for build | -| imagebuild.squashlayers | PORTAGE_IMAGE_BUILD_SQUASH_LAYERS | 0 | squash image layers - Only Supported with Podman CLI | -| imagebuild.target | PORTAGE_IMAGE_BUILD_TARGET | - | The target build stage to build (e.g., [linux/amd64]) | -| imagepublish.bundletag | PORTAGE_IMAGE_PUBLISH_BUNDLE_TAG | | The full image tag for the target gatecheck bundle image blob | -| imagepublish.enabled | PORTAGE_IMAGE_PUBLISH_ENABLED | 1 | Enable/Disable the image publish pipeline | -| imagescan.clamavfilename | PORTAGE_IMAGE_SCAN_CLAMAV_FILENAME | clamav-virus-report.txt | The filename for the clamscan virus report - must contain 'clamav' | -| imagescan.enabled | PORTAGE_IMAGE_SCAN_ENABLED | 1 | Enable/Disable the image scan pipeline | -| imagescan.grypeconfigfilename | PORTAGE_IMAGE_SCAN_GRYPE_CONFIG_FILENAME | - | The config filename for the grype vulnerability report | -| imagescan.grypefilename | PORTAGE_IMAGE_SCAN_GRYPE_FILENAME | grype-vulnerability-report-full.json | The filename for the grype vulnerability report - must contain 'grype' | -| imagescan.syftfilename | PORTAGE_IMAGE_SCAN_SYFT_FILENAME | syft-sbom-report.json | The filename for the syft SBOM report - must contain 'syft' | - - -## Running in Docker +For Example if you want to run portage scans only and not build the container nor deploy it, you can run the following command. -When running portage in a docker container there are some pipelines that need to run docker commands. -In order for the docker CLI in the portage to connect to the docker daemon running on the host machine, -you must either mount the `/var/run/docker.sock` in the `portage` container, or provide configuration for -accessing the docker daemon remotely with the `DOCKER_HOST` environment variable. +```bash +portage run all --tag "ttl.sh/$(uuidgen | tr '[:upper:]' '[:lower:]'):1h" --imagebuild.enabled 0 --deploy.enabled 0 +``` -If you don't have access to Artifactory to pull in the Omnibus base image, you can build the image manually which is -in `images/omnibus/Dockerfile`. -### Using `/var/run/docker.sock` +## Running a Pipeline in Docker -This approach assumes you have the docker daemon running on your host machine. +The latest Portage Docker container can be found here: -Example: +``` +TBD +``` +When running portage in a docker container there are some pipelines that need to run docker commands. +In order for the docker CLI in the portage to connect to the docker daemon running on the host machine, + + +### Run the portage container with the desired arguments +Using a .portage config files +```bash +portage run image-build ``` -docker run -it --rm \ - `# Mount your Dockerfile and supporting files in the working directory: /app` \ - -v "$(pwd):/app:ro" \ - `# Mount docker.sock for use by the docker CLI running inside the container` \ - -v "/var/run/docker.sock:/var/run/docker.sock" \ - `# Run the portage container with the desired arguments` \ - portage run image-build +//Example config file here + +Using flags + +```bash +docker run -it --rm -v "$(pwd):/app:rw" -v "/var/run/docker.sock:/var/run/docker.sock:rw" ghcr.io/easy-up/portage:v0.0.1-rc.19 run all --tag "ttl.sh/$(uuidgen | tr [:upper:] [:lower:]):1h" ``` ### Using a Remote Daemon - For more information see the [Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/#environment-variables) and [Docker Daemon](https://docs.docker.com/config/daemon/remote-access/) documentation pages. From 9991f844dea9615a4b973b8c5988b710c8de6eaf Mon Sep 17 00:00:00 2001 From: Aaron Yee Date: Wed, 20 Nov 2024 11:06:40 -1000 Subject: [PATCH 2/3] add example portage.yml content --- README.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5dc7b11..d5c9f55 100644 --- a/README.md +++ b/README.md @@ -139,7 +139,16 @@ Using a .portage config files ```bash portage run image-build ``` -//Example config file here + +```yaml +imageTag: user/image-name +imagePublish: + enabled: false +codeScan: + coverageFile: coverage/cobertura-coverage.xml +deploy: + gatecheckConfigFilename: .gatecheck.yml +``` Using flags From 2bc19bc5272d4ac97410814efeb79850dedb3684 Mon Sep 17 00:00:00 2001 From: Aaron Yee Date: Sun, 1 Dec 2024 00:19:43 -1000 Subject: [PATCH 3/3] provide portage.yml example and added logging on what configuration portage is using --- README.md | 73 +++++++++++++++++++------------------- cmd/portage/cli/v0/root.go | 18 +++++++++- docs/.portage.yml.example | 46 ++++++++++++++++++++++++ 3 files changed, 100 insertions(+), 37 deletions(-) create mode 100644 docs/.portage.yml.example diff --git a/README.md b/README.md index d5c9f55..cdaf63d 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,9 @@ Portage CD is a secure, continuous delivery pipeline built on open source. Portage CD is designed to orchestrate the process of building and scanning an application image for security vulnerabilities. The unique aspect of Portage CD is that it is meant to be portable, meaning that a developer can run the entire pipeline locally, address any security vulnerabilities or code issues before pushing before pushing a branch to a CI/CD pipeline based in the cloud that is also running Portage CD. -This project aims to simplify the CI/CD build process such that developers can focus on building their unit and end to end tests, and spend less time working on security pipeline tooling, and standing up continuous delivery (CD). +This project aims to simplify the CI/CD build process by providing an easy to use wrapper around security scanning tools and container building tools. The result is that the development team can build a simpler CI/CD pipeline that is easier to understand and maintain. -Portage CD is designed to make CI/CD easier by providing a preconfigured security and container building pipeline that can be configured such that tools can be overridden where an enterprise SaaS tool may exist. The tool has also been designed so that i can be incorporated into existing CI pipelines running in any CI platform. Portage CD can be statically compiled as a binary and run on virtually any environment or CI/CD platform. +Portage CD provides a preconfigured security and container build pipeline that can be configured such that tools can be overridden where an enterprise SaaS tool may exist. The tool has also been designed so that it can be incorporated into existing CI pipelines running in any CI platform (Gitlab CI, Github Actions, Jenkins, etc.). At it's core, Portage CD is a golang based CLI tool that is configured to chain the security and container building open source tools below to make the CI/CD process easier. @@ -92,63 +92,64 @@ OR sudo cp /path/to/portage/bin/portage /usr/local/bin/ ``` -Then navigate to the source code directory that you wish to run the pipeline scanning tools on. To run the security pipeline directly, run the following command. +Then navigate to the source code directory that you wish to run the pipeline scanning tools on. To run the security pipeline directly, make a copy of .portage.yml.example and name it .portage.yml. Edit the .portage.yml file to configure the pipeline to your needs,then run the following command. ```bash -portage run debug +portage run all ``` -### Configuring a Pipeline +### Running parts of a pipeline -Configuration Options: +If you want to only run the code scan portion of the pipeline, you can run the following command. -Note: `(none)` means unset, left blank - -| Configuration Option | Environment Variable | Default Value | Description | -|-----------------------------------|----------------------------------------|--------------------------------------|----------------------------------------------------------------------------------| -| imagescan.grypefilename | PORTAGE_IMAGE_SCAN_GRYPE_FILENAME | grype-vulnerability-report-full.json | The filename for the grype vulnerability report - must contain 'grype' | -| imagescan.syftfilename | PORTAGE_IMAGE_SCAN_SYFT_FILENAME | syft-sbom-report.json | The filename for the syft SBOM report - must contain 'syft' | +```bash +portage run code-scan +``` +For more information on running parts of the pipeline, you can run portage run and see the available commands. -To run a pipeline with all the defaults, run the following command. ```bash -portage run all --tag "ttl.sh/$(uuidgen | tr '[:upper:]' '[:lower:]'):1h" +portage run ``` +## Running Portage in Docker -For Example if you want to run portage scans only and not build the container nor deploy it, you can run the following command. +The latest Portage Docker container is available at: ```bash -portage run all --tag "ttl.sh/$(uuidgen | tr '[:upper:]' '[:lower:]'):1h" --imagebuild.enabled 0 --deploy.enabled 0 +ghcr.io/easy-up/portage:latest ``` +You can run Portage in Docker using either a configuration file or command-line flags: -## Running a Pipeline in Docker +#### Using a configuration file +1. Create a `.portage.yml` file in your project directory +2. Run Portage with: -The latest Portage Docker container can be found here: - -``` -TBD +```bash +docker run -it --rm \ + -v "$(pwd):/app:rw" \ + -v "/var/run/docker.sock:/var/run/docker.sock" \ + ghcr.io/easy-up/portage:latest \ + run all ``` -When running portage in a docker container there are some pipelines that need to run docker commands. -In order for the docker CLI in the portage to connect to the docker daemon running on the host machine, - +#### Using command-line flags +For one-off runs, you can specify options directly: -### Run the portage container with the desired arguments -Using a .portage config files ```bash -portage run image-build +docker run -it --rm \ + -v "$(pwd):/app:rw" \ + -v "/var/run/docker.sock:/var/run/docker.sock" \ + ghcr.io/easy-up/portage:latest \ + run all --tag "your-image-tag:latest" ``` -```yaml -imageTag: user/image-name -imagePublish: - enabled: false -codeScan: - coverageFile: coverage/cobertura-coverage.xml -deploy: - gatecheckConfigFilename: .gatecheck.yml -``` +Key points: +- Mount your project directory to `/app` to give Portage access to your code +- Mount the Docker socket if you need to build containers +- The container runs as read-write (`rw`) to allow for artifact generation + +Portage has multiple configuration options that can be set via environment variables or a .portage.yml file. For more information see the [docs](./docs/.portage.yml.example). Using flags diff --git a/cmd/portage/cli/v0/root.go b/cmd/portage/cli/v0/root.go index 1dd31e0..68d4fc2 100644 --- a/cmd/portage/cli/v0/root.go +++ b/cmd/portage/cli/v0/root.go @@ -1,6 +1,7 @@ package cli import ( + "fmt" "log/slog" "portage/pkg/pipelines" @@ -14,7 +15,22 @@ var ( ) func NewPortageCommand() *cobra.Command { - viper.SetConfigName("portage") + // Set up Viper configuration + viper.SetConfigName(".portage") // name of config file (without extension) + viper.SetConfigType("yaml") // REQUIRED if the config file does not have the extension in the name + viper.AddConfigPath(".") // look for config in the working directory + + // Try to read the config file + if err := viper.ReadInConfig(); err != nil { + if _, ok := err.(viper.ConfigFileNotFoundError); ok { + fmt.Printf("No config file found: %v\n", err) + } else { + fmt.Printf("Error reading config file: %v\n", err) + } + } else { + fmt.Printf("Using config file: %s\n", viper.ConfigFileUsed()) + } + pipelines.BindViper(viper.GetViper()) versionCmd := newBasicCommand("version", "print version information", runVersion) diff --git a/docs/.portage.yml.example b/docs/.portage.yml.example new file mode 100644 index 0000000..d791201 --- /dev/null +++ b/docs/.portage.yml.example @@ -0,0 +1,46 @@ +# Base Configuration +version: "1" +imageTag: "owner/my-app:latest" # The full image tag for the target container image (e.g. my-org/my-app:latest) +artifactDir: "artifacts" # Directory for generated artifacts (e.g. ./artifacts) +gatecheckBundleFilename: "gatecheck-bundle.tar.gz" # Filename for the gatecheck bundle (e.g. gatecheck-bundle.tar.gz) + +# Image Build Configuration +imageBuild: + enabled: true # Enable/Disable the image build pipeline (true/false) + buildDir: "." # Build directory for image (e.g. ./cmd/portage) + dockerfile: "Dockerfile" # Dockerfile to use (e.g. ./cmd/portage/Dockerfile) + platform: "" # Target platform (e.g. linux/amd64, linux/arm64) + target: "" # Target stage for multi-stage builds (e.g. build, test, publish) + cacheTo: "" # Cache export location (e.g. type=local,dest=path) + cacheFrom: "" # Cache import location (e.g. type=local,src=path) + squashLayers: false # Whether to squash layers (true/false) + args: {} # Build arguments (e.g. BUILD_ARGS=--build-arg=key=value) + +# Image Scan Configuration +imageScan: + enabled: true # Enable/Disable the image scan pipeline (true/false) + syftFilename: "syft-sbom-report.json" # Filename for the syft sbom report (e.g. syft-sbom-report.json) + grypeConfigFilename: "" # Filename for the grype config (e.g. grype-config.json) + grypeFilename: "grype-vulnerability-report-full.json" # Filename for the grype vulnerability report (e.g. grype-vulnerability-report-full.json) + clamavFilename: "clamav-virus-report.txt" # Filename for the clamav virus report (e.g. clamav-virus-report.txt) + +# Code Scan Configuration +codeScan: + enabled: true # Enable/Disable the code scan pipeline (true/false) + gitleaksFilename: "gitleaks-secrets-report.json" + gitleaksSrcDir: "." + semgrepFilename: "semgrep-sast-report.json" # Filename for the semgrep sast report (e.g. semgrep-sast-report.json) + semgrepRules: "p/default" # Semgrep rules to use (e.g. p/default) + semgrepExperimental: false # Whether to use experimental semgrep rules (true/false) + coverageFile: "" # Externally generated code coverage file + semgrepSrcDir: "." # Target directory for semgrep scan (e.g. ./cmd/portage) + +# Image Publish Configuration +imagePublish: + enabled: true # Enable/Disable the image publish pipeline (true/false) + bundleTag: "" # Image tag for gatecheck bundle image blob (e.g. my-org/my-app:latest) + +# Deploy Configuration +deploy: + enabled: true # Enable/Disable the deploy pipeline (true/false) + gatecheckConfigFilename: "" # Filename for gatecheck config (e.g. gatecheck-config.json) \ No newline at end of file