Test unification

Signed-off-by: Eduardo Apolinario <[email protected]>

CHANGELOG/CHANGELOG-v1.11.0.md

@@ -0,0 +1,32 @@
+# Flyte v1.11.0 Release Notes
+
+We're excited to announce the release of Flyte v1.11.0! This version brings a host of improvements, bug fixes, and new features designed to enhance your experience with Flyte. From operational enhancements to documentation updates, this release aims to make Flyte more robust, user-friendly, and feature-rich.
+
+## Highlights
+
+- **Agents hit General Availability (GA):** Agents, now in General Availability, are long-running, stateless services that facilitate asynchronous job launches on platforms like Databricks or Snowflake and enable external service calls. They are versatile, supporting implementations in any language through a protobuf interface, enhancing Flyte's flexibility and operational efficiency.
+- **Improved Caching:** Support for loading cached sublists with multiple data types has been introduced, eliminating issues related to cache retrieval across varied data formats.
+- **Tracing and Observability:** The introduction of opentelemetry BlobstoreClientTracer in flyteadmin enhances observability, allowing for better monitoring and troubleshooting.
+- **Security Enhancements:** Added securityContext configuration to Flyte-core charts, strengthening the security posture of Flyte deployments.
+- **Documentation Overhaul:** Continuous improvements and updates have been made to the documentation, fixing broken links and updating content for better clarity and usability.
+- **Operational Improvements:** This release introduces enhancements such as adding a service account for V1 Ray Jobs, caching console assets in a single binary, and conditional mounting of secrets to improve the operational efficiency of Flyte. Additionally, we are removing `kustomize` from our deployment process to simplify the configuration and management of Flyte instances, making it easier for users to maintain and streamline their deployment workflows.
+
+
+## Bug Fixes
+
+- **Fixed Literal in Launchplan:** Added fixed_literal to the launchplan template, addressing issues with hardcoded values in workflows.
+- **Corrected Metadata and Resources:** Fixes have been applied to correct IsParent metadata in ArrayNode eventing and to address invalid "resources" scope issues in deployment configurations.
+- **Enhanced Stability and Performance:** Numerous bug fixes have been implemented to address stability and performance issues, including fixes for data catalog errors, yaml comment errors in pod template examples, and more.
+
+## Documentation and Guides
+
+- **Comprehensive Guides:** New guides and documentation updates have been added, including a ChatGPT Agent Setup guide and an Airflow migration guide. Improvements in documentation for developing agents have been integrated into the broader enhancements for this release.
+- **Updated Troubleshooting and Configuration Docs:** New troubleshooting guides for spark task execution and updates to deployment configuration documents enhance the knowledge base for Flyte users.
+
+## Contributors
+
+We extend our deepest gratitude to all the contributors who made this release possible. Special shoutouts to @neilisaur, @lowc1012, @MortalHappiness, @novahow, and @pryce-turner for making their first contributions!
+
+**For a full list of changes, enhancements, and bug fixes, visit our [changelog](https://github.com/flyteorg/flyte/compare/v1.10.7...v1.11.0).**
+
+Thank you for your continued support of Flyte. We look forward to hearing your feedback on this release!

CHANGELOG/CHANGELOG-v1.11.1-b0.md

@@ -0,0 +1,2 @@
+# Flyte v1.11.1-b0
+

Dockerfile

@@ -2,7 +2,7 @@ ARG FLYTECONSOLE_VERSION=latest
 FROM ghcr.io/flyteorg/flyteconsole:${FLYTECONSOLE_VERSION} AS flyteconsole
 
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-bookworm AS flytebuilder
+FROM --platform=${BUILDPLATFORM} golang:1.21-bookworm AS flytebuilder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"

Dockerfile.datacatalog

@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'FLYTEORG/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/flyteorg/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"
@@ -23,6 +23,7 @@ RUN GRPC_HEALTH_PROBE_VERSION=v0.4.11 && \
 
 WORKDIR /go/src/github.com/flyteorg/datacatalog
 
+COPY boilerplate ../boilerplate
 COPY datacatalog .
 COPY flyteadmin ../flyteadmin
 COPY flytecopilot ../flytecopilot

Dockerfile.flyteadmin

@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'LYFT/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/lyft/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"
@@ -23,6 +23,7 @@ RUN GRPC_HEALTH_PROBE_VERSION=v0.4.11 && \
 
 WORKDIR /go/src/github.com/flyteorg/flyteadmin
 
+COPY boilerplate ../boilerplate
 COPY datacatalog ../datacatalog
 COPY flyteadmin .
 COPY flytecopilot ../flytecopilot

Dockerfile.flytecopilot

@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'LYFT/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/lyft/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"
@@ -13,6 +13,7 @@ RUN apk add git openssh-client make curl
 
 WORKDIR /go/src/github.com/flyteorg/flytecopilot
 
+COPY boilerplate ../boilerplate
 COPY datacatalog ../datacatalog
 COPY flyteadmin ../flyteadmin
 COPY flytecopilot .

Dockerfile.flytepropeller

@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'LYFT/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/lyft/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"
@@ -13,6 +13,7 @@ RUN apk add git openssh-client make curl
 
 WORKDIR /go/src/github.com/flyteorg/flytepropeller
 
+COPY boilerplate ../boilerplate
 COPY datacatalog ../datacatalog
 COPY flyteadmin ../flyteadmin
 COPY flytecopilot ../flytecopilot

Dockerfile.flytescheduler

@@ -1,9 +1,9 @@
 # WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
 # ONLY EDIT THIS FILE FROM WITHIN THE 'LYFT/BOILERPLATE' REPOSITORY:
-# 
+#
 # TO OPT OUT OF UPDATES, SEE https://github.com/lyft/boilerplate/blob/master/Readme.rst
 
-FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder
+FROM --platform=${BUILDPLATFORM} golang:1.21-alpine3.18 as builder
 
 ARG TARGETARCH
 ENV GOARCH "${TARGETARCH}"

Makefile

@@ -112,3 +112,12 @@ go-tidy:
 	make -C flyteplugins go-tidy
 	make -C flytestdlib go-tidy
 	make -C flytecopilot go-tidy
+
+.PHONY: lint-helm-charts
+lint-helm-charts:
+	# This pressuposes that you have act installed
+	act pull_request -W .github/workflows/validate-helm-charts.yaml --container-architecture linux/amd64 -e charts/event.json
+
+.PHONY: clean
+clean: ## Remove the HTML files related to the Flyteconsole.
+	rm -rf cmd/single/dist

boilerplate/flyte/docker_build/Makefile

@@ -0,0 +1,12 @@
+# WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
+# ONLY EDIT THIS FILE FROM WITHIN THE 'FLYTEORG/BOILERPLATE' REPOSITORY:
+# 
+# TO OPT OUT OF UPDATES, SEE https://github.com/flyteorg/boilerplate/blob/master/Readme.rst
+
+.PHONY: docker_build
+docker_build:
+	IMAGE_NAME=$$REPOSITORY ./boilerplate/flyte/docker_build/docker_build.sh
+
+.PHONY: dockerhub_push
+dockerhub_push:
+	IMAGE_NAME=flyteorg/$$REPOSITORY REGISTRY=docker.io ./boilerplate/flyte/docker_build/docker_build.sh

boilerplate/flyte/docker_build/Readme.rst

@@ -0,0 +1,23 @@
+Docker Build and Push
+~~~~~~~~~~~~~~~~~~~~~
+
+Provides a ``make docker_build`` target that builds your image locally.
+
+Provides a ``make dockerhub_push`` target that pushes your final image to Dockerhub.
+
+The Dockerhub image will tagged ``<REPOSITORY>:<GIT COMMIT SHA>``
+
+If git head has a git tag, the Dockerhub image will also be tagged ``<IMAGE>:<GIT_TAG>``.
+
+**To Enable:**
+
+Add ``flyteorg/docker_build`` to your ``boilerplate/update.cfg`` file.
+
+Add ``include boilerplate/flyte/docker_build/Makefile`` in your main ``Makefile`` _after_ your REPOSITORY environment variable
+
+::
+
+    REPOSITORY=<myreponame>
+    include boilerplate/flyte/docker_build/Makefile
+
+(this ensures the extra Make targets get included in your main Makefile)

boilerplate/flyte/docker_build/docker_build.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+
+# WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
+# ONLY EDIT THIS FILE FROM WITHIN THE 'FLYTEORG/BOILERPLATE' REPOSITORY:
+# 
+# TO OPT OUT OF UPDATES, SEE https://github.com/flyteorg/boilerplate/blob/master/Readme.rst
+
+set -e
+
+echo ""
+echo "------------------------------------"
+echo "           DOCKER BUILD"
+echo "------------------------------------"
+echo ""
+
+if [ -n "$REGISTRY" ]; then
+  # Do not push if there are unstaged git changes
+  CHANGED=$(git status --porcelain)
+  if [ -n "$CHANGED" ]; then
+    echo "Please commit git changes before pushing to a registry"
+    exit 1
+  fi
+fi
+
+
+GIT_SHA=$(git rev-parse HEAD)
+
+IMAGE_TAG_SUFFIX=""
+# for intermediate build phases, append -$BUILD_PHASE to all image tags
+if [ -n "$BUILD_PHASE" ]; then
+  IMAGE_TAG_SUFFIX="-${BUILD_PHASE}"
+fi
+
+IMAGE_TAG_WITH_SHA="${IMAGE_NAME}:${GIT_SHA}${IMAGE_TAG_SUFFIX}"
+
+RELEASE_SEMVER=$(git describe --tags --exact-match "$GIT_SHA" 2>/dev/null) || true
+if [ -n "$RELEASE_SEMVER" ]; then
+  IMAGE_TAG_WITH_SEMVER="${IMAGE_NAME}:${RELEASE_SEMVER}${IMAGE_TAG_SUFFIX}"
+fi
+
+# build the image
+# passing no build phase will build the final image
+docker build -t "$IMAGE_TAG_WITH_SHA" --target=${BUILD_PHASE} .
+echo "${IMAGE_TAG_WITH_SHA} built locally."
+
+# if REGISTRY specified, push the images to the remote registry
+if [ -n "$REGISTRY" ]; then
+
+  if [ -n "${DOCKER_REGISTRY_PASSWORD}" ]; then
+    docker login --username="$DOCKER_REGISTRY_USERNAME" --password="$DOCKER_REGISTRY_PASSWORD"
+  fi
+
+  docker tag "$IMAGE_TAG_WITH_SHA" "${REGISTRY}/${IMAGE_TAG_WITH_SHA}"
+
+  docker push "${REGISTRY}/${IMAGE_TAG_WITH_SHA}"
+  echo "${REGISTRY}/${IMAGE_TAG_WITH_SHA} pushed to remote."
+
+  # If the current commit has a semver tag, also push the images with the semver tag
+  if [ -n "$RELEASE_SEMVER" ]; then
+
+    docker tag "$IMAGE_TAG_WITH_SHA" "${REGISTRY}/${IMAGE_TAG_WITH_SEMVER}"
+
+    docker push "${REGISTRY}/${IMAGE_TAG_WITH_SEMVER}"
+    echo "${REGISTRY}/${IMAGE_TAG_WITH_SEMVER} pushed to remote."
+
+  fi
+fi

boilerplate/flyte/flyte_golang_compile/Readme.rst

@@ -0,0 +1,16 @@
+Flyte Golang Compile
+~~~~~~~~~~~~~~~~~~~~
+
+Common compile script for Flyte golang services.
+
+**To Enable:**
+
+Add ``flyteorg/flyte_golang_compile`` to your ``boilerplate/update.cfg`` file.
+
+Add the following to your Makefile
+
+::
+
+  .PHONY: compile_linux
+  compile_linux:
+    PACKAGES={{ *your packages }} OUTPUT={{ /path/to/output }} ./boilerplate/flyte/flyte_golang_compile.sh

boilerplate/flyte/flyte_golang_compile/flyte_golang_compile.Template

@@ -0,0 +1,26 @@
+# WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
+# ONLY EDIT THIS FILE FROM WITHIN THE 'FLYTEORG/BOILERPLATE' REPOSITORY:
+# 
+# TO OPT OUT OF UPDATES, SEE https://github.com/flyteorg/boilerplate/blob/master/Readme.rst
+
+if [ -z "$PACKAGES" ]; then
+  echo "PACKAGES environment VAR not set"
+  exit 1
+fi
+
+if [ -z "$OUTPUT" ]; then
+  echo "OUTPUT environment VAR not set"
+  exit 1
+fi
+
+# get the GIT_SHA and RELEASE_SEMVER
+
+GIT_SHA=$(git rev-parse HEAD)
+RELEASE_SEMVER=$(git describe --tags --exact-match $GIT_SHA 2>/dev/null)
+
+CURRENT_PKG=github.com/flyteorg/{{ REPOSITORY }}
+VERSION_PKG="${CURRENT_PKG}/vendor/github.com/flyteorg/flytestdlib"
+
+LDFLAGS="-X ${VERSION_PKG}/version.Build=${GIT_SHA} -X ${VERSION_PKG}/version.Version=${RELEASE_SEMVER}"
+
+GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "$LDFLAGS" -o "$OUTPUT" "$PACKAGES"

boilerplate/flyte/flyte_golang_compile/update.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+# WARNING: THIS FILE IS MANAGED IN THE 'BOILERPLATE' REPO AND COPIED TO OTHER REPOSITORIES.
+# ONLY EDIT THIS FILE FROM WITHIN THE 'FLYTEORG/BOILERPLATE' REPOSITORY:
+# 
+# TO OPT OUT OF UPDATES, SEE https://github.com/flyteorg/boilerplate/blob/master/Readme.rst
+
+set -e
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+
+echo "     - generating ${DIR}/flyte_golang_compile.sh"
+sed -e "s/{{REPOSITORY}}/${REPOSITORY}/g" ${DIR}/flyte_golang_compile.Template > ${DIR}/flyte_golang_compile.sh

boilerplate/flyte/github_workflows/Readme.rst

@@ -0,0 +1,22 @@
+Golang Github Actions
+~~~~~~~~~~~~~~~~~
+
+Provides a two github actions workflows.
+
+**To Enable:**
+
+Add ``flyteorg/github_workflows`` to your ``boilerplate/update.cfg`` file.
+
+Add a github secret ``package_name`` with the name to use for publishing (e.g. ``flytepropeller``). Typically, this will be the same name as the repository.
+
+*Note*: If you are working on a fork, include that prefix in your package name (``myfork/flytepropeller``).
+
+The actions will push to 2 repos:
+
+	1. ``docker.pkg.github.com/flyteorg/<repo>/<package_name>``
+	2. ``docker.pkg.github.com/flyteorg/<repo>/<package_name>-stages`` : this repo is used to cache build stages to speed up iterative builds after.
+
+There are two workflows that get deployed:
+
+	1. A workflow that runs on Pull Requests to build and push images to github registry tagged with the commit sha.
+	2. A workflow that runs on master merges that bump the patch version of release tag, builds and pushes images to github registry tagged with the version, commit sha as well as "latest"

boilerplate/flyte/github_workflows/boilerplate_automation.yml

@@ -0,0 +1,36 @@
+name: Update Boilerplate Automation
+on:
+  workflow_dispatch:
+jobs:
+  update-boilerplate:
+    name: Update Boilerplate
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: "0"
+      - name: Update Boilerplate
+        run: |
+          make update_boilerplate
+      - name: Create Pull Request
+        id: cpr
+        uses: peter-evans/create-pull-request@v3
+        with:
+          token: ${{ secrets.FLYTE_BOT_PAT }}
+          commit-message: Update Boilerplate
+          committer: Flyte-Bot <[email protected]>
+          author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
+          signoff: true
+          branch: flyte-bot-update-boilerplate
+          delete-branch: true
+          title: 'Update Boilerplate'
+          body: |
+            Update Boilerplate
+            - Auto-generated by [flyte-bot]
+          labels: |
+            boilerplate
+          team-reviewers: |
+            owners
+            maintainers
+          draft: false 
+