diff --git a/.github/ISSUE_TEMPLATE/new_version.md b/.github/ISSUE_TEMPLATE/new_version.md index 1dfcec6e..8d44e464 100644 --- a/.github/ISSUE_TEMPLATE/new_version.md +++ b/.github/ISSUE_TEMPLATE/new_version.md @@ -7,25 +7,25 @@ assignees: '' --- -**Which new version of Apache Spark-on-Kubernetes should we support?** +## Which new version of Apache Spark-on-Kubernetes should we support? Please specify the version, version range or version numbers to support, please also add these to the issue title -**Additional information** +## Additional information If possible, provide a link to release notes/changelog -**Changes required** +## Changes required Are there any upstream changes that we need to support? e.g. new features, changed features, deprecated features etc. +## Implementation checklist - -**Implementation checklist** - -Please don't change anything in this list. -Not all of these steps are necessary for all versions. + - [ ] Update the Docker image - [ ] Update documentation to include supported version(s) diff --git a/.github/PULL_REQUEST_TEMPLATE/pre-release-getting-started-script.md b/.github/PULL_REQUEST_TEMPLATE/pre-release-getting-started-script.md new file mode 100644 index 00000000..0081b520 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/pre-release-getting-started-script.md @@ -0,0 +1,34 @@ +# Check and Update Getting Started Script + + + + + +Part of + +> [!NOTE] +> During a Stackable release we need to check (and optionally update) the +> getting-started scripts to ensure they still work after product and operator +> updates. + +```shell +# Some of the scripts are in a code/ subdirectory +# pushd docs/modules/superset/examples/getting_started +# pushd docs/modules/superset/examples/getting_started/code +pushd $(fd -td getting_started | grep examples); cd code 2>/dev/null || true + +# Make a fresh cluster (~12 seconds) +kind delete cluster && kind create cluster +./getting_started.sh stackablectl + +# Make a fresh cluster (~12 seconds) +kind delete cluster && kind create cluster +./getting_started.sh helm + +popd +``` diff --git a/.github/PULL_REQUEST_TEMPLATE/pre-release-rust-deps.md b/.github/PULL_REQUEST_TEMPLATE/pre-release-rust-deps.md new file mode 100644 index 00000000..86537df2 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/pre-release-rust-deps.md @@ -0,0 +1,38 @@ +# Bump Rust Dependencies for Stackable Release XX.(X)X + + + + + +Part of + +> [!NOTE] +> During a Stackable release we need to update various Rust dependencies before +> entering the final release period to ensure we run the latest versions of +> crates. These bumps also include previously updated and released crates from +> the `operator-rs` repository. + +```[tasklist] +### Tasks +- [ ] Bump Rust Dependencies, see below for more details. +- [ ] Add changelog entry stating which important crates were bumped (including the version). +``` + +> [!NOTE] +> The bumping / updating of Rust dependencies is done in multiple steps: +> +> 1. Update the minimum Version in the root `Cargo.toml` manifest. +> 2. Run the `cargo update` command, which also updates the `Cargo.lock` file. +> 3. Lastly, run `make regenerate-nix` to update the `Cargo.nix` file. + +```[tasklist] +### Bump Rust Dependencies +- [ ] Bump `stackable-operator` and friends. +- [ ] Bump `product-version`. +- [ ] Bump all other dependencies. +``` diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index f52aa8ea..bf0c4686 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -57,7 +57,7 @@ jobs: key: udeps cache-all-crates: "true" - uses: stackabletech/cargo-install-action@cargo-udeps - - run: cargo udeps --workspace + - run: cargo udeps --workspace --all-targets # This job evaluates the github environment to determine why this action is running and selects the appropriate # target repository for published Helm charts based on this. diff --git a/.markdownlint.yaml b/.markdownlint.yaml index dbfa4558..5af20365 100644 --- a/.markdownlint.yaml +++ b/.markdownlint.yaml @@ -14,6 +14,10 @@ MD013: # Number of characters for code blocks code_block_line_length: 9999 +# MD033/no-inline-html +MD033: + allowed_elements: [h1, img, p] + # MD024/no-duplicate-heading/no-duplicate-header - Multiple headings with the same content MD024: # Only check sibling headings diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 059d91ff..fc52721b 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,4 +1,6 @@ --- +exclude: ^(Cargo\.nix|crate-hashes\.json|nix/.*)$ + # See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: @@ -28,11 +30,19 @@ repos: rev: v0.40.0 hooks: - id: markdownlint + types: [text] + files: \.md(\.j2)*$ - - repo: https://github.com/PyCQA/flake8 - rev: 7.0.0 + # WARNING (@NickLarsenNZ): Nix users need to install ruff first. + # If you do not, you will need to delete the cached ruff binary shown in the + # error message + - repo: https://github.com/astral-sh/ruff-pre-commit + rev: v0.5.1 hooks: - - id: flake8 + # Run the linter. + - id: ruff + # Run the formatter. + - id: ruff-format - repo: local hooks: diff --git a/.readme/partials/borrowed/documentation.md.j2 b/.readme/partials/borrowed/documentation.md.j2 index f6d98756..88727cd5 100644 --- a/.readme/partials/borrowed/documentation.md.j2 +++ b/.readme/partials/borrowed/documentation.md.j2 @@ -1,3 +1,4 @@ + ## Documentation The stable documentation for this operator can be found [here](https://docs.stackable.tech/home/stable/{{operator_docs_slug}}). diff --git a/.readme/partials/borrowed/footer.md.j2 b/.readme/partials/borrowed/footer.md.j2 index 5276fa2a..2249051f 100644 --- a/.readme/partials/borrowed/footer.md.j2 +++ b/.readme/partials/borrowed/footer.md.j2 @@ -1,3 +1,4 @@ + ## About The Stackable Data Platform This operator is written and maintained by [Stackable](https://stackable.tech) and it is part of a larger data platform. @@ -24,7 +25,6 @@ We develop and test our operators on the following cloud platforms: * Kubernetes (for an up to date list of supported versions please check the release notes in our [docs](https://docs.stackable.tech)) * Red Hat OpenShift - ## Other Operators These are the operators that are currently part of the Stackable Data Platform: diff --git a/.readme/partials/borrowed/header.md.j2 b/.readme/partials/borrowed/header.md.j2 index 65b32584..81bd88c6 100644 --- a/.readme/partials/borrowed/header.md.j2 +++ b/.readme/partials/borrowed/header.md.j2 @@ -1,3 +1,4 @@ +

Stackable Logo

diff --git a/.readme/partials/borrowed/links.md.j2 b/.readme/partials/borrowed/links.md.j2 index 42f050ff..39e4e614 100644 --- a/.readme/partials/borrowed/links.md.j2 +++ b/.readme/partials/borrowed/links.md.j2 @@ -1,4 +1,4 @@ - + {% if no_jenkins_job_badge %}{% else %}![Build Actions Status](https://ci.stackable.tech/buildStatus/icon?job={{operator_name}}%2doperator%2dit%2dnightly&subject=Integration%20Tests){% endif %} [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://GitHub.com/stackabletech/{{operator_name}}-operator/graphs/commit-activity) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-green.svg)](https://docs.stackable.tech/home/stable/contributor/index.html) diff --git a/.readme/partials/borrowed/overview_blurb.md.j2 b/.readme/partials/borrowed/overview_blurb.md.j2 index bff68cb9..b4638b37 100644 --- a/.readme/partials/borrowed/overview_blurb.md.j2 +++ b/.readme/partials/borrowed/overview_blurb.md.j2 @@ -1 +1,2 @@ + It is part of the Stackable Data Platform, a curated selection of the best open source data apps like Apache Kafka, Apache Druid, Trino or Apache Spark, [all](#other-operators) working together seamlessly. Based on Kubernetes, it runs everywhere – [on prem or in the cloud](#supported-platforms). diff --git a/.readme/partials/borrowed/related_reading.md.j2 b/.readme/partials/borrowed/related_reading.md.j2 index 18102f89..b91339f7 100644 --- a/.readme/partials/borrowed/related_reading.md.j2 +++ b/.readme/partials/borrowed/related_reading.md.j2 @@ -1,3 +1,4 @@ + {%- if related_reading_links -%} ## Related Reading {% for (text, link) in related_reading_links %} diff --git a/README.md b/README.md index 6e3d8cc7..6474a561 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,11 @@ +

Stackable Logo

Stackable Operator for Apache Spark

+ ![Build Actions Status](https://ci.stackable.tech/buildStatus/icon?job=spark-k8s%2doperator%2dit%2dnightly&subject=Integration%20Tests) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://GitHub.com/stackabletech/spark-k8s-operator/graphs/commit-activity) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-green.svg)](https://docs.stackable.tech/home/stable/contributor/index.html) @@ -13,6 +15,7 @@ This is a Kubernetes operator to manage [Apache Spark](https://spark.apache.org/) jobs. + It is part of the Stackable Data Platform, a curated selection of the best open source data apps like Apache Kafka, Apache Druid, Trino or Apache Spark, [all](#other-operators) working together seamlessly. Based on Kubernetes, it runs everywhere – [on prem or in the cloud](#supported-platforms). ## Installation @@ -25,6 +28,7 @@ Read on to get started with it, or see it in action in one of our [demos](https: You can follow this [tutorial](https://docs.stackable.tech/home/stable/spark-k8s/getting_started/first_steps) . + ## Documentation The stable documentation for this operator can be found [here](https://docs.stackable.tech/home/stable/spark-k8s). @@ -34,6 +38,7 @@ The documentation for all Stackable products can be found at [docs.stackable.tec If you have a question about the Stackable Data Platform contact us via our [homepage](https://stackable.tech/) or ask a public questions in our [Discussions forum](https://github.com/orgs/stackabletech/discussions). + ## About The Stackable Data Platform This operator is written and maintained by [Stackable](https://stackable.tech) and it is part of a larger data platform. @@ -60,7 +65,6 @@ We develop and test our operators on the following cloud platforms: * Kubernetes (for an up to date list of supported versions please check the release notes in our [docs](https://docs.stackable.tech)) * Red Hat OpenShift - ## Other Operators These are the operators that are currently part of the Stackable Data Platform: @@ -98,3 +102,5 @@ This is enforced automatically when you submit a Pull Request where a bot will g ## Support Get started with the community edition! If you want professional support, [we offer subscription plans and custom licensing](https://stackable.tech/en/plans/). + + diff --git a/deploy/DO_NOT_EDIT.md b/deploy/DO_NOT_EDIT.md index 7faf564d..f7d5ff21 100644 --- a/deploy/DO_NOT_EDIT.md +++ b/deploy/DO_NOT_EDIT.md @@ -1,3 +1,5 @@ +# DO NOT EDIT + These Helm charts and manifests are automatically generated. Please do not edit anything except for files explicitly mentioned below in this directory manually. diff --git a/deploy/helm/spark-k8s-operator/README.md b/deploy/helm/spark-k8s-operator/README.md index 7af0a72c..f4819327 100644 --- a/deploy/helm/spark-k8s-operator/README.md +++ b/deploy/helm/spark-k8s-operator/README.md @@ -1,3 +1,4 @@ + # Helm Chart for Stackable Operator for Apache Spark-on-Kubernetes This Helm Chart can be used to install Custom Resource Definitions and the Operator for Apache Spark-on-Kubernetes provided by Stackable. @@ -24,4 +25,4 @@ The operator has example requests included in the [`/examples`](https://github.c ## Links -https://github.com/stackabletech/spark-k8s-operator + diff --git a/scripts/ensure_one_trailing_newline.py b/scripts/ensure_one_trailing_newline.py index 4e58cc10..a6f73c06 100644 --- a/scripts/ensure_one_trailing_newline.py +++ b/scripts/ensure_one_trailing_newline.py @@ -2,6 +2,7 @@ Given the location of a file, trims all trailing blank lines and places a single one. Used as post-processing step for README rendering. """ + import re import unittest @@ -34,7 +35,6 @@ def process_lines(lines): class TestCoreMethods(unittest.TestCase): - def test_trailing_new_line(self): self.assertTrue(has_trailing_newline("something\n")) self.assertTrue(has_trailing_newline("\n")) diff --git a/shell.nix b/shell.nix index 218363ee..7ef77f28 100644 --- a/shell.nix +++ b/shell.nix @@ -11,6 +11,7 @@ in pkgs.mkShell rec { packages = with pkgs; [ ## cargo et-al rustup # this breaks pkg-config if it is in the nativeBuildInputs + cargo-udeps ## Extra dependencies for use in a pure env (nix-shell --pure) ## These are mosuly useful for maintainers of this shell.nix