Flyte v1.12.0-b0 milestone release

Flyte v1.12.0-b0

What's Changed

• [Docs] Remove kustomize link in secrets.md doc by @lowc1012 in #5043
• CI workflow for helm charts and manifests by @lowc1012 in #5027
• Update spark-on-k8s-operator address in helm charts by @eapolinario in #5057
• Fix wrong syntax for path filtering in validate-helm-charts.yaml by @lowc1012 in #5056
• Fix: flyte-secret-auth secret not mounted properly in flyte-core by @lowc1012 in #5054
• Match flytekit versions used to register and run functional tests by @eapolinario in #5059
• integration test config by @troychiu in #5058
• Add org as an optional request param to dataproxy CreateUploadLocation by @katrogan in #5060
• Add k8s env from by @neilisaur in #4969
• Implement GetProject endpoint in FlyteAdmin by @RRap0so in #4825
• Prepopulate ArrayNode output literals with TaskNode interface output variables by @hamersaw in #5080
• [House Keeping] deprecate MaxDatasetSizeBytes propeller config in favor of GetLimitMegabytes storage config by @pvditt in #4852
• Fix lint errors caught by chart-testing by @eapolinario in #5072
• Update Flyte components by @flyte-bot in #5093
• sagemaker agent backend setup documentation by @samhita-alla in #5064
• add first version of gke-starter values file by @cjidboon94 in #5026
• Fix open ai secret name by @eapolinario in #5098
• Allow setting a ExecutionClusterLabel when triggering a Launchplan/Workflow/Task by @RRap0so in #4998
• Improve audience mismatch debugging by @ddl-rliu in #5078
• docs(sandbox): Add guide for running newest flyteconsole in flyte sandbox by @MortalHappiness in #5100
• Remove unnecessary step and fix numbering in code examples by @eapolinario in #5104
• fix rendering of flyte-core and flyteagent charts by @pbrogan12 in #5048
• Bump golang.org/x/net from 0.3.1-0.20221206200815-1e63c2f08a10 to 0.7.0 in /docker/sandbox-bundled/bootstrap by @dependabot in #3390
• Update repeated value filters with ValueNotIn support by @troychiu in #5110
• Update container builds from go 1.21.5 to 1.21.latest by @ddl-ebrown in #5097
• Add optional org param to ProjectGetRequest by @katrogan in #5118
• Stop building read-the-docs for flyteidl by @eapolinario in #5120
• Bump google.golang.org/grpc and otelgrpc by @eapolinario in #5121
• Fix broken link in "Mapping Python to Flyte types" table by @neverett in #5122
• Bump version of otel and grpc in flytestdlib by @eapolinario in #5123
• [flyteadmin] Show diff structure when re-registration two different task with same ids by @austin362667 in #4924
• Set flyteadmin grpc port correctly in config / Flyte-core flyteadmin / datacatalog expose ports by @ddl-ebrown in #5013
• Add flyte-core missing priorityClassName to webhook values by @ddl-ebrown in #4987
• Update environment_setup.md by @ssen85 in #4963
• [Docs] Open external links in new tab by @MortalHappiness in #4966
• Fix separateGrpcIngress flag not working in flyte-binary helm chart by @lowc1012 in #4946
• docs(contribute): Change go mod tidy to make go-tidy by @MortalHappiness in #5131
• Fix execution phase by @troychiu in #5127
• Add trailing slash to compile make target by @eapolinario in #4648
• Change retry error from RuntimeError to FlyteRecoverableException by @dansola in #5128
• Adapt ray flyteplugin to Kuberay 1.1.0 by @ByronHsu in #5067
• Boilerplate simplification by @eapolinario in #5134
• Upgrade cloudevents to v2.15.2 by @ddl-ebrown in #5142
• update sagemaker agent setup doc as secrets aren't required anymore by @samhita-alla in #5138
• Upgrade logrus to v1.9.3 everywhere by @ddl-ebrown in #5139
• Upgrade go-restful to v3.12.0 by @ddl-ebrown in #5140
• Regenerate ray pflags by @eapolinario in #5149
• Split access token into half and store to avoid "securecookie: the value is too long" error by @yubofredwang in #4863
• Update Flyte components by @flyte-bot in #5150
• Bump golang.org/x/crypto from 0.11.0 to 0.17.0 in /boilerplate/flyte/golang_support_tools by @dependabot in #5148
• use javascript to open new tab for external links by @cosmicBboy in #5159
• Update K8s plugin config docs by @davidmirror-ops in #5070
• Update Propeller architecture documentation by @davidmirror-ops in #5117
• update protobuf v1.32.0 -> v1.33.0 by @ddl-ebrown in #5156
• Update boilerplate version by @flyte-bot in #5143
• Upgrade grpc health probe 0.4.11 -> 0.4.25 by @ddl-ebrown in #5153
• Upgrade go-jose v3.0.0 -> v3.0.3 by @ddl-ebrown in #5154
• docs: update agent development documentation by @pingsutw in #5130
• Add variables to ease separate bucket config by @davidmirror-ops in #5015
• Bump golang.org/x/net from 0.12.0 to 0.17.0 in /boilerplate/flyte/golang_support_tools by @dependabot in #5146
• Bump google.golang.org/protobuf from 1.30.0 to 1.33.0 in /boilerplate/flyte/golang_support_tools by @dependabot in #5147
• Upgrade lestrrat-go/jwx to v1.2.29 by @ddl-ebrown in #5141
• Revert "Pin flyteconsole version in release process (#5037)" by @eapolinario in #5176
• Add identity to task execution metadata by @noahjax in #5105
• Bump k8s.io/client-go from 0.0.0-20210217172142-7279fc64d847 to 0.17.16 in /boilerplate/flyte/golang_support_tools by @dependabot in #5145
• Upgrade jackc/pgconn v1.14.1 -> v1.14.3 / pgx/v5 v5.4.3 -> v5.5.5 / pgproto3 v2.3.2 -> v2.3.3 by @ddl-ebrown in #5155
• fix make link error by @novahow in #5175
• Stop admin launcher copying shard key from parent workflow by @Tom-Newton in #5174
• Fix Id bigint conversation for not yet created table by @ongkong in #5157
• Add tracking for active node and task execution counts in propeller by @sshardool in #4986
• [House keeping] include container statuses for all container exit errors by @pvditt in #5161
• docs: add missing key in auth guide by @Jeinhaus in #5169
• Shallow copying EnvironmentVariables map before injecting ArrayNode env vars by @hamersaw in #5182
• Feature/array node workflow parallelism by @pvditt in #5062
• Fix streak length metric reporting by @Tom-Newton in #5172
• Fix path to AuthMetadataService in flyte-binary chart by @eapolinario in #5185
• Change phase to queue on job submit for webapi plugins by @pingsutw in #5188
• [Docs] Testing agents in the development environment by @Future-Outlier in #5106
• Use ratelimiter config in webapi plugins by @kumare3 in #5190
• docs(ray): Update kuberay documentation by @MortalHappiness in #5179
• Change phase to WaitingForResources when quota exceeded by @pingsutw in #5195
• Fix: Update spark operator helm repository by @fg91 in #5198
• docs(troubleshoot): Add docker error troubleshooting guide by @MortalHappiness in #4972
• add cache client read and write otel tracing by @pvditt in #5184
• Fix FlyteIDL docs link by @neverett in #5199
• [easy] [flyteagent] Add ExecuteTaskSync function timeout setting by @Future-Outlier in https://github.com/flyte...

Flyte v1.11.1-b1 milestone release

Flyte v1.11.1-b1

What's Changed

• sagemaker agent backend setup documentation by @samhita-alla in #5064
• add first version of gke-starter values file by @cjidboon94 in #5026
• Fix open ai secret name by @eapolinario in #5098
• Allow setting a ExecutionClusterLabel when triggering a Launchplan/Workflow/Task by @RRap0so in #4998
• Improve audience mismatch debugging by @ddl-rliu in #5078
• docs(sandbox): Add guide for running newest flyteconsole in flyte sandbox by @MortalHappiness in #5100
• Remove unnecessary step and fix numbering in code examples by @eapolinario in #5104
• fix rendering of flyte-core and flyteagent charts by @pbrogan12 in #5048
• Bump golang.org/x/net from 0.3.1-0.20221206200815-1e63c2f08a10 to 0.7.0 in /docker/sandbox-bundled/bootstrap by @dependabot in #3390
• Update repeated value filters with ValueNotIn support by @troychiu in #5110
• Update container builds from go 1.21.5 to 1.21.latest by @ddl-ebrown in #5097
• Add optional org param to ProjectGetRequest by @katrogan in #5118
• Stop building read-the-docs for flyteidl by @eapolinario in #5120
• Bump google.golang.org/grpc and otelgrpc by @eapolinario in #5121
• Fix broken link in "Mapping Python to Flyte types" table by @neverett in #5122
• Bump version of otel and grpc in flytestdlib by @eapolinario in #5123
• [flyteadmin] Show diff structure when re-registration two different task with same ids by @austin362667 in #4924
• Set flyteadmin grpc port correctly in config / Flyte-core flyteadmin / datacatalog expose ports by @ddl-ebrown in #5013
• Add flyte-core missing priorityClassName to webhook values by @ddl-ebrown in #4987
• Update environment_setup.md by @ssen85 in #4963
• [Docs] Open external links in new tab by @MortalHappiness in #4966
• Fix separateGrpcIngress flag not working in flyte-binary helm chart by @lowc1012 in #4946
• docs(contribute): Change go mod tidy to make go-tidy by @MortalHappiness in #5131
• Fix execution phase by @troychiu in #5127
• Add trailing slash to compile make target by @eapolinario in #4648
• Change retry error from RuntimeError to FlyteRecoverableException by @dansola in #5128
• Adapt ray flyteplugin to Kuberay 1.1.0 by @ByronHsu in #5067
• Boilerplate simplification by @eapolinario in #5134
• Upgrade cloudevents to v2.15.2 by @ddl-ebrown in #5142
• update sagemaker agent setup doc as secrets aren't required anymore by @samhita-alla in #5138
• Upgrade logrus to v1.9.3 everywhere by @ddl-ebrown in #5139
• Upgrade go-restful to v3.12.0 by @ddl-ebrown in #5140
• Regenerate ray pflags by @eapolinario in #5149
• Split access token into half and store to avoid "securecookie: the value is too long" error by @yubofredwang in #4863

New Contributors

• @cjidboon94 made their first contribution in #5026
• @ddl-rliu made their first contribution in #5078
• @pbrogan12 made their first contribution in #5048
• @austin362667 made their first contribution in #4924
• @ssen85 made their first contribution in #4963
• @dansola made their first contribution in #5128

Flyte v1.11.1-b0 milestone release

Flyte v1.11.1-b0

Flyte v1.11.0 milestone release

Flyte v1.11.0 Release Notes

We're excited to announce the release of Flyte v1.11.0! This version brings a host of improvements, bug fixes, and new features designed to enhance your experience with Flyte. From operational enhancements to documentation updates, this release aims to make Flyte more robust, user-friendly, and feature-rich.

Breaking change

As of this version, overriding container image via task node overrides requires flytekit 1.11.0.

Highlights

• Agents hit General Availability (GA): Agents, now in General Availability, are long-running, stateless services that facilitate asynchronous job launches on platforms like Databricks or Snowflake and enable external service calls. They are versatile, supporting implementations in any language through a protobuf interface, enhancing Flyte's flexibility and operational efficiency.
• Improved Caching: Support for loading cached sublists with multiple data types has been introduced, eliminating issues related to cache retrieval across varied data formats.
• Tracing and Observability: The introduction of opentelemetry BlobstoreClientTracer in flyteadmin enhances observability, allowing for better monitoring and troubleshooting.
• Security Enhancements: Added securityContext configuration to Flyte-core charts, strengthening the security posture of Flyte deployments.
• Documentation Overhaul: Continuous improvements and updates have been made to the documentation, fixing broken links and updating content for better clarity and usability.
• Operational Improvements: This release introduces enhancements such as adding a service account for V1 Ray Jobs, caching console assets in a single binary, and conditional mounting of secrets to improve the operational efficiency of Flyte. Additionally, we are removing kustomize from our deployment process to simplify the configuration and management of Flyte instances, making it easier for users to maintain and streamline their deployment workflows.

Bug Fixes

• Fixed Literal in Launchplan: Added fixed_literal to the launchplan template, addressing issues with hardcoded values in workflows.
• Corrected Metadata and Resources: Fixes have been applied to correct IsParent metadata in ArrayNode eventing and to address invalid "resources" scope issues in deployment configurations.
• Enhanced Stability and Performance: Numerous bug fixes have been implemented to address stability and performance issues, including fixes for data catalog errors, yaml comment errors in pod template examples, and more.

Documentation and Guides

• Comprehensive Guides: New guides and documentation updates have been added, including a ChatGPT Agent Setup guide and an Airflow migration guide. Improvements in documentation for developing agents have been integrated into the broader enhancements for this release.
• Updated Troubleshooting and Configuration Docs: New troubleshooting guides for spark task execution and updates to deployment configuration documents enhance the knowledge base for Flyte users.

Contributors

We extend our deepest gratitude to all the contributors who made this release possible. Special shoutouts to @neilisaur, @lowc1012, @MortalHappiness, @novahow, and @pryce-turner for making their first contributions!

For a full list of changes, enhancements, and bug fixes, visit our changelog.

Thank you for your continued support of Flyte. We look forward to hearing your feedback on this release!

Flyte v1.11.0-b1 milestone release

Flyte v1.11.0-b1

Second beta release for 1.11.0.

This includes a refresh of flyteconsole.

Flyte v1.11.0-b0 milestone release

Flyte v1.11.0-b0

Beta release to test new idl

Flyte v1.10.7 milestone release

Flyte 1.10.7 Release Notes

We're excited to share the release of Flyte 1.10.7, featuring a broad spectrum of updates, improvements, and bug fixes across the Flyte ecosystem. This release marks a pivotal shift in our development approach, notably with our adoption of buf for protobuf stub generation. This move optimizes our development workflow and discontinues the automatic creation of Java and C++ stubs, making it easier to adapt the generated code for other languages as needed. Additionally, we've upgraded to gRPC-gateway v2, aligning with the latest advancements and recommendations found in the v2 migration guide.

Our sincere gratitude goes to all contributors for their invaluable efforts towards this release.

Core Improvements and Bug Fixes

• Improved error handling for transient secret sync issues, enhancing the robustness of secret management. [PR #4310]
• Introduced Sphinx build for Monodocs, improving documentation generation and integration. [PR #4347]
• Enhanced the Spark plugin by fixing the environment variable ValueFrom for pod templates, allowing for more dynamic configurations. [PR #4532]
• Optimized fastcache behavior to not cache lookups on node skip, reducing unnecessary cache hits. [PR #4524]
• Removed composition errors from branch nodes, streamlining execution paths. [PR #4528]
• Added support for ignoring warnings related to AWS SageMaker imports, improving integration compatibility. [PR #4540]
• Fixed a bug related to setting the service account from PodTemplate, ensuring correct service account usage. [PR #4536]
• Addressed flaky tests in test_monitor, enhancing test reliability. [PR #4537]
• Updated the boilerplate version and contribution guide, facilitating better community contributions. [PR #4541], [PR #4501]
• Improved documentation build processes by manually creating version files and introducing a conda-lock file for consistent environment setup. [PR #4556], [PR #4553]
• Enhanced array node evaluation frequency optimization by detecting subNode phase updates. [PR #4535]
• Introduced support for failure nodes, allowing workflows to handle failures more gracefully. [PR #4308]
• Made various updates to Go versions, plugin integrations, and GitHub workflows to enhance performance and developer experience. [PR #4534], [PR #4582], [PR #4589]
• Addressed several bugs and made improvements in caching, metadata handling, and task execution, further stabilizing the Flyte platform. [PR #4594], [PR #4590], [PR #4607]
• Streamlined development workflow with the transition to buf for generating protobuf stubs, ceasing the automatic generation of Java and C++ stubs.
• Upgraded to grpc-gateway v2, optimizing API performance and compatibility.

Plugin and Integration Enhancements

• Added new features and fixed bugs in the Spark plugin, Ray Autoscaler integration, and other areas, expanding Flyte's capabilities and integration ecosystem. [PR #4363]
• Updated various dependencies and configurations, ensuring compatibility and security. [PR #4571], [PR #4643]
• Improved the handling and documentation of plugin secrets management, making it easier for users to manage sensitive information. [PR #4732]

Documentation and Community

• Updated community meeting cadence and contribution guidelines, fostering a more engaged and welcoming community. [PR #4699]
• Enhanced documentation through various updates, including the introduction of a new architecture image for FlytePlugins and clarification of propeller scaling. [PR #4661], [PR #4741]

Full Changelog

• Fix transient secret sync error handling by @Tom-Newton in #4310
• Monodocs sphinx build by @cosmicBboy in #4347
• [Spark plugin] Fix environment variable ValueFrom for pod templates by @Tom-Newton in #4532
• fastcache should not cache lookup on node skip by @hamersaw in #4524
• Removed composition error from branch node by @hamersaw in #4528
• ignore warnings related to awssagemaker import by @cosmicBboy in #4540
• [BUG] Fix setting of service_account from PodTemplate by @pvditt in #4536
• Fix flaky test_monitor by @pingsutw in #4537
• Update boilerplate version by @flyte-bot in #4541
• remove hardcoded list of tests by @samhita-alla in #4521
• manually create flytekit/_version.py file in docs build by @cosmicBboy in #4556
• introduce conda-lock file for docs by @cosmicBboy in #4553
• Detect subNode phase updates to reduce evaluation frequency of ArrayNode by @hamersaw in #4535
• Add support failure node by @pingsutw in #4308
• Return InvalidArgument for workflow compilation failures in CreateWorkflow by @katrogan in #4566
• Update to go 1.21 by @eapolinario in #4534
• Update contribution guide by @pingsutw in #4501
• Add flyin plugin to monodocs integrations page by @neverett in #4582
• Use updated cronSchedule in CreateLaunchPlanModel by @pmahindrakar-oss in #4564
• Writing zero length inputs by @hamersaw in #4594
• Feature/add pod pending timeout config by @pvditt in #4590
• Run single-binary gh workflows on all PRs by @eapolinario in #4589
• auto-generate toctree from flytesnacks index.md docs by @cosmicBboy in #4587
• add repo tag and commit associated with the build by @cosmicBboy in #4571
• monodocs - gracefully handle case when external repo doesn't contain tags: use current commit by @cosmicBboy in #4598
• convert commit to string by @cosmicBboy in #4599
• Bug/abort map task subtasks by @pvditt in #4506
• Supporting parallelized workers in ArrayNode subNodes by @hamersaw in #4567
• Don't use experimental readthedocs build.commands config by @cosmicBboy in #4606
• Ignore cache variables by @hamersaw in #4618
• Feature/add cleanup non recoverable pod statuses by @pvditt in #4607
• Agent Metadata Servicer by @Future-Outlier in #4511
• Add Flyin propeller config by @eapolinario in #4610
• Correctly computing ArrayNode maximum attempts and system failures by @hamersaw in #4627
• Agent Sync Plugin by @Future-Outlier in #4107
• Add github token in buf gh action by @eapolinario in #4626
• Update flyte-binary values by @davidmirror-ops in #4604
• Fixing cache overwrite metadata update by @hamersaw in #4617
• Fixing 100 kilobyte max error message size by @hamersaw in #4631
• Add Ray Autoscaler to the Flyte-Ray plugin by @Yicheng-Lu-llll in #4363
• Artifact protos and related changes by @wild-endeavor in #4474
• Remove protoc-gen-validate by @eapolinario in #4643
• Readme update 2023 by @davidmirror-ops in #4549
• Fixing ArrayNode integration with backoff controller by @hamersaw in #4640
• Avoid to use the http.DefaultClient by @andresgomezfrr in #4667
• Update dns policy for sandbox buildkit instance to ClusterFirstWithHo… by @jeevb in #4678
• Updating ArrayNode ExternalResourceInfo ID by @hamersaw in #4677
• Feat: Inject user identity as pod label in K8s plugin by @fg91 in https://githu...

Flyte v1.10.7-b4 milestone release

Flyte v1.10.7-b4 Release

Pre-release testing.

Flyte v1.10.7-b3 milestone release

Flyte v1.10.7-b3 Release

Pre-release testing.

Flyte v1.10.7-b2 milestone release

Flyte v1.10.7-b2 Release

Pre-release testing.