Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation Improvements #12

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Documentation Improvements #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a57b238
added more context
ayee808 Oct 30, 2024
9991f84
add example portage.yml content
ayee808 Nov 20, 2024
2bc19bc
provide portage.yml example and added logging on what configuration p…
ayee808 Dec 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 102 additions & 80 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,138 +4,160 @@

![Portage CD Logo](./static/portage-cd-logo.svg)

Portage CD is a secure, continuous delivery pipeline designed to orchestrate the process of building and scanning an
application image for security vulnerabilities.
It solves the problem of having to configure a hardened-predefined security pipeline using traditional CI/CD.
Portage CD can be statically compiled as a binary and run on virtually any platform, CI/CD
environment, or locally.
## Basic Overview

Portage CD is a secure, continuous delivery pipeline built on open source. Portage CD is designed to orchestrate the process of building and scanning an application image for security vulnerabilities. The unique aspect of Portage CD is that it is meant to be portable, meaning that a developer can run the entire pipeline locally, address any security vulnerabilities or code issues before pushing before pushing a branch to a CI/CD pipeline based in the cloud that is also running Portage CD.

This project aims to simplify the CI/CD build process by providing an easy to use wrapper around security scanning tools and container building tools. The result is that the development team can build a simpler CI/CD pipeline that is easier to understand and maintain.

Portage CD provides a preconfigured security and container build pipeline that can be configured such that tools can be overridden where an enterprise SaaS tool may exist. The tool has also been designed so that it can be incorporated into existing CI pipelines running in any CI platform (Gitlab CI, Github Actions, Jenkins, etc.).

At it's core, Portage CD is a golang based CLI tool that is configured to chain the security and container building open source tools below to make the CI/CD process easier.

## Getting Started

NOTE: *While Portage CD will easily work with any Windows or Linux environment, the following instructions are done on a MacOS PC environment. In the future we will update this Readme with instructions for other environments.*

Install Prerequisites:

- Container Engine
- Docker or Podman CLI
- Golang >= v1.22.0
- Container Engine (e.g. [Docker](https://docs.docker.com/desktop/), Openshift, etc.)
- [Docker](https://docs.docker.com/desktop/) or Podman CLI
- Golang >= v1.22.0 ([brew install](https://formulae.brew.sh/formula/go))
- [Just](https://github.com/casey/just?tab=readme-ov-file#installation) (optional)

```bash
brew install [email protected] just
```

Prerequisite Tools (For running `portage` local)

- Gitleaks
- Semgrep
- Syft
- Grype
- ClamAV (only the `clamscan` and `freshclam` cli utilities are needed)
- ORAS

## Compiling Portage CD
If you are running on a mac and have brew installed, you can install these dependencies using the following command.

Running the just recipe will put the compiled-binary into `./bin`
```bash
brew install gitleaks semgrep syft grype clamav oras
```

Once ClamAV is installed, you will need to configure it to download the virus definitions. This can be done by adding the following lines to your `/etc/freshclam.conf` file.

```
# Allow freshclam to run as root
DatabaseMirror database.clamav.net
Foreground yes
```

Freschclam will download and run the latest virus definitions every time you run it. WARNING: This will generate a lot of network traffic. Please be aware that if you pull too many times, you will be rate limited by the virus definition servers.

Run the following command to download the virus definitions.

```bash
just build
freshclam
```

OR compile manually
## Compiling Portage CD for Local Use

Clone the repo and navigate to the directory in the shell.

```bash
git clone <this-repo> <target-dir>
cd <target-dir>
mkdir bin
go build -o bin/portage ./cmd/portage
```

Optionally, if you care to include metadata about the version of `portage` (displayed when you run `portage version`), use the following build arguments
Running the just recipe will put the compiled-binary into `./bin`

```shell
go build -ldflags="-X 'main.cliVersion=$(git describe --tags)' -X 'main.gitCommit=$(git rev-parse HEAD)' -X 'main.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)' -X 'main.gitDescription=$(git log -1 --pretty=%B)'" -o ./bin ./cmd/portage
```bash
just build
```


## Running A Pipeline

You can run the executable directory
OR you can compile Portage CD manually

```bash
portage run debug
go build -ldflags="-X 'main.cliVersion=$(git describe --tags)' -X 'main.gitCommit=$(git rev-parse HEAD)' -X 'main.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)' -X 'main.gitDescription=$(git log -1 --pretty=%B)'" -o ./bin ./cmd/portage
```

## Configuring a Pipeline
## Running A Pipeline Locally

Configuration Options:
First, map the portage binary to your $PATH via ~/.zshrc or copy the binary into your /usr/local/bin.

- Configuration via CLI flags
- Environment Variables
- Config File in JSON
- Config File in YAML
- Config File in TOML
```bash
echo 'export PATH="$PATH:/path/to/portage/bin"' >> ~/.zshrc
source ~/.zshrc
```
OR
```bash
sudo cp /path/to/portage/bin/portage /usr/local/bin/
```

Configuration Order-of-Precedence:
Then navigate to the source code directory that you wish to run the pipeline scanning tools on. To run the security pipeline directly, make a copy of .portage.yml.example and name it .portage.yml. Edit the .portage.yml file to configure the pipeline to your needs,then run the following command.

1. CLI Flag
2. Environment Variable
3. Config File Value
4. Default Value
```bash
portage run all
```

Note: `(none)` means unset, left blank
### Running parts of a pipeline

| Config Key | Environment Variable | Default Value | Description |
| --------------------------------- |------------------------------------------| ------------------------------------ |------------------------------------------------------------------------------------|
| codescan.enabled | PORTAGE_CODE_SCAN_ENABLED | 1 | Enable/Disable the code scan pipeline |
| codescan.gitleaksfilename | PORTAGE_CODE_SCAN_GITLEAKS_FILENAME | gitleaks-secrets-report.json | The filename for the gitleaks secret report - must contain 'gitleaks' |
| codescan.gitleakssrcdir | PORTAGE_CODE_SCAN_GITLEAKS_SRC_DIR | . | The target directory for the gitleaks scan |
| codescan.semgrepfilename | PORTAGE_CODE_SCAN_SEMGREP_FILENAME | semgrep-sast-report.json | The filename for the semgrep SAST report - must contain 'semgrep' |
| codescan.semgreprules | PORTAGE_CODE_SCAN_SEMGREP_RULES | p/default | Semgrep ruleset manual override |
| codescan.semgrepexperimental | PORTAGE_CODE_SCAN_SEMGREP_EXPERIMENTAL | false | Enable the use of the semgrep experimental CLI |
| deploy.enabled | PORTAGE_IMAGE_PUBLISH_ENABLED | 1 | Enable/Disable the deploy pipeline |
| deploy.gatecheckconfigfilename | PORTAGE_DEPLOY_GATECHECK_CONFIG_FILENAME | - | The filename for the gatecheck config |
| gatecheckbundlefilename | PORTAGE_GATECHECK_BUNDLE_FILENAME | artifacts/gatecheck-bundle.tar.gz | The filename for the gatecheck bundle, a validatable archive of security artifacts |
| imagebuild.args | PORTAGE_IMAGE_BUILD_ARGS | - | Comma seperated list of build time variables |
| imagebuild.builddir | PORTAGE_IMAGE_BUILD_DIR | . | The build directory to using during an image build |
| imagebuild.cachefrom | PORTAGE_IMAGE_BUILD_CACHE_FROM | - | External cache sources (e.g., "user/app:cache", "type=local,src=path/to/dir") |
| imagebuild.cacheto | PORTAGE_IMAGE_BUILD_CACHE_TO | - | Cache export destinations (e.g., "user/app:cache", "type=local,src=path/to/dir") |
| imagebuild.dockerfile | PORTAGE_IMAGE_BUILD_DOCKERFILE | Dockerfile | The Dockerfile/Containerfile to use during an image build |
| imagebuild.enabled | PORTAGE_IMAGE_BUILD_ENABLED | 1 | Enable/Disable the image build pipeline |
| imagebuild.platform | PORTAGE_IMAGE_BUILD_PLATFORM | - | The target platform for build |
| imagebuild.squashlayers | PORTAGE_IMAGE_BUILD_SQUASH_LAYERS | 0 | squash image layers - Only Supported with Podman CLI |
| imagebuild.target | PORTAGE_IMAGE_BUILD_TARGET | - | The target build stage to build (e.g., [linux/amd64]) |
| imagepublish.bundletag | PORTAGE_IMAGE_PUBLISH_BUNDLE_TAG | | The full image tag for the target gatecheck bundle image blob |
| imagepublish.enabled | PORTAGE_IMAGE_PUBLISH_ENABLED | 1 | Enable/Disable the image publish pipeline |
| imagescan.clamavfilename | PORTAGE_IMAGE_SCAN_CLAMAV_FILENAME | clamav-virus-report.txt | The filename for the clamscan virus report - must contain 'clamav' |
| imagescan.enabled | PORTAGE_IMAGE_SCAN_ENABLED | 1 | Enable/Disable the image scan pipeline |
| imagescan.grypeconfigfilename | PORTAGE_IMAGE_SCAN_GRYPE_CONFIG_FILENAME | - | The config filename for the grype vulnerability report |
| imagescan.grypefilename | PORTAGE_IMAGE_SCAN_GRYPE_FILENAME | grype-vulnerability-report-full.json | The filename for the grype vulnerability report - must contain 'grype' |
| imagescan.syftfilename | PORTAGE_IMAGE_SCAN_SYFT_FILENAME | syft-sbom-report.json | The filename for the syft SBOM report - must contain 'syft' |
If you want to only run the code scan portion of the pipeline, you can run the following command.

```bash
portage run code-scan
```
For more information on running parts of the pipeline, you can run portage run and see the available commands.

## Running in Docker
```bash
portage run
```

When running portage in a docker container there are some pipelines that need to run docker commands.
In order for the docker CLI in the portage to connect to the docker daemon running on the host machine,
you must either mount the `/var/run/docker.sock` in the `portage` container, or provide configuration for
accessing the docker daemon remotely with the `DOCKER_HOST` environment variable.
## Running Portage in Docker

If you don't have access to Artifactory to pull in the Omnibus base image, you can build the image manually which is
in `images/omnibus/Dockerfile`.
The latest Portage Docker container is available at:

### Using `/var/run/docker.sock`
```bash
ghcr.io/easy-up/portage:latest
```

This approach assumes you have the docker daemon running on your host machine.
You can run Portage in Docker using either a configuration file or command-line flags:

Example:
#### Using a configuration file
1. Create a `.portage.yml` file in your project directory
2. Run Portage with:

```bash
docker run -it --rm \
-v "$(pwd):/app:rw" \
-v "/var/run/docker.sock:/var/run/docker.sock" \
ghcr.io/easy-up/portage:latest \
run all
```

#### Using command-line flags
For one-off runs, you can specify options directly:

```bash
docker run -it --rm \
`# Mount your Dockerfile and supporting files in the working directory: /app` \
-v "$(pwd):/app:ro" \
`# Mount docker.sock for use by the docker CLI running inside the container` \
-v "$(pwd):/app:rw" \
-v "/var/run/docker.sock:/var/run/docker.sock" \
`# Run the portage container with the desired arguments` \
portage run image-build
ghcr.io/easy-up/portage:latest \
run all --tag "your-image-tag:latest"
```

### Using a Remote Daemon
Key points:
- Mount your project directory to `/app` to give Portage access to your code
- Mount the Docker socket if you need to build containers
- The container runs as read-write (`rw`) to allow for artifact generation

Portage has multiple configuration options that can be set via environment variables or a .portage.yml file. For more information see the [docs](./docs/.portage.yml.example).

Using flags

```bash
docker run -it --rm -v "$(pwd):/app:rw" -v "/var/run/docker.sock:/var/run/docker.sock:rw" ghcr.io/easy-up/portage:v0.0.1-rc.19 run all --tag "ttl.sh/$(uuidgen | tr [:upper:] [:lower:]):1h"
```

### Using a Remote Daemon
For more information see the
[Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/#environment-variables) and
[Docker Daemon](https://docs.docker.com/config/daemon/remote-access/) documentation pages.
Expand Down
18 changes: 17 additions & 1 deletion cmd/portage/cli/v0/root.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package cli

import (
"fmt"
"log/slog"
"portage/pkg/pipelines"

Expand All @@ -14,7 +15,22 @@ var (
)

func NewPortageCommand() *cobra.Command {
viper.SetConfigName("portage")
// Set up Viper configuration
viper.SetConfigName(".portage") // name of config file (without extension)
viper.SetConfigType("yaml") // REQUIRED if the config file does not have the extension in the name
viper.AddConfigPath(".") // look for config in the working directory

// Try to read the config file
if err := viper.ReadInConfig(); err != nil {
if _, ok := err.(viper.ConfigFileNotFoundError); ok {
fmt.Printf("No config file found: %v\n", err)
} else {
fmt.Printf("Error reading config file: %v\n", err)
}
} else {
fmt.Printf("Using config file: %s\n", viper.ConfigFileUsed())
}

pipelines.BindViper(viper.GetViper())

versionCmd := newBasicCommand("version", "print version information", runVersion)
Expand Down
46 changes: 46 additions & 0 deletions docs/.portage.yml.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Base Configuration
version: "1"
imageTag: "owner/my-app:latest" # The full image tag for the target container image (e.g. my-org/my-app:latest)
artifactDir: "artifacts" # Directory for generated artifacts (e.g. ./artifacts)
gatecheckBundleFilename: "gatecheck-bundle.tar.gz" # Filename for the gatecheck bundle (e.g. gatecheck-bundle.tar.gz)

# Image Build Configuration
imageBuild:
enabled: true # Enable/Disable the image build pipeline (true/false)
buildDir: "." # Build directory for image (e.g. ./cmd/portage)
dockerfile: "Dockerfile" # Dockerfile to use (e.g. ./cmd/portage/Dockerfile)
platform: "" # Target platform (e.g. linux/amd64, linux/arm64)
target: "" # Target stage for multi-stage builds (e.g. build, test, publish)
cacheTo: "" # Cache export location (e.g. type=local,dest=path)
cacheFrom: "" # Cache import location (e.g. type=local,src=path)
squashLayers: false # Whether to squash layers (true/false)
args: {} # Build arguments (e.g. BUILD_ARGS=--build-arg=key=value)

# Image Scan Configuration
imageScan:
enabled: true # Enable/Disable the image scan pipeline (true/false)
syftFilename: "syft-sbom-report.json" # Filename for the syft sbom report (e.g. syft-sbom-report.json)
grypeConfigFilename: "" # Filename for the grype config (e.g. grype-config.json)
grypeFilename: "grype-vulnerability-report-full.json" # Filename for the grype vulnerability report (e.g. grype-vulnerability-report-full.json)
clamavFilename: "clamav-virus-report.txt" # Filename for the clamav virus report (e.g. clamav-virus-report.txt)

# Code Scan Configuration
codeScan:
enabled: true # Enable/Disable the code scan pipeline (true/false)
gitleaksFilename: "gitleaks-secrets-report.json"
gitleaksSrcDir: "."
semgrepFilename: "semgrep-sast-report.json" # Filename for the semgrep sast report (e.g. semgrep-sast-report.json)
semgrepRules: "p/default" # Semgrep rules to use (e.g. p/default)
semgrepExperimental: false # Whether to use experimental semgrep rules (true/false)
coverageFile: "" # Externally generated code coverage file
semgrepSrcDir: "." # Target directory for semgrep scan (e.g. ./cmd/portage)

# Image Publish Configuration
imagePublish:
enabled: true # Enable/Disable the image publish pipeline (true/false)
bundleTag: "" # Image tag for gatecheck bundle image blob (e.g. my-org/my-app:latest)

# Deploy Configuration
deploy:
enabled: true # Enable/Disable the deploy pipeline (true/false)
gatecheckConfigFilename: "" # Filename for gatecheck config (e.g. gatecheck-config.json)
Loading