Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop Flink 1.15 support #32836

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@

## Deprecations

* Removed support for Flink 1.15
* X behavior is deprecated and will be removed in X versions ([#X](https://github.com/apache/beam/issues/X)).

## Bugfixes
Expand Down
2 changes: 1 addition & 1 deletion gradle.properties
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,6 @@ docker_image_default_repo_root=apache
docker_image_default_repo_prefix=beam_

# supported flink versions
flink_versions=1.15,1.16,1.17,1.18,1.19
flink_versions=1.16,1.17,1.18,1.19
# supported python versions
python_versions=3.8,3.9,3.10,3.11,3.12
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,8 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
{{if (eq .Sdk "java")}}

##### Portable
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.15`, `Flink 1.16`, `Flink 1.17`, `Flink 1.18`.
2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.16`, `Flink 1.17`, `Flink 1.18`, `Flink 1.19`.
2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.19_job_server:latest`
3. Submit the pipeline to the above endpoint by using the PortableRunner, job_endpoint set to localhost:8099 (this is the default address of the JobService). Optionally set environment_type set to LOOPBACK. For example:

```
Expand Down Expand Up @@ -233,8 +233,8 @@ mvn exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
{{end}}

{{if (eq .Sdk "python")}}
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.10`, `Flink 1.11`, `Flink 1.12`, `Flink 1.13`, `Flink 1.14`.
2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.16`, `Flink 1.17`, `Flink 1.18`, `Flink 1.19`.
2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.19_job_server:latest`
3. Submit the pipeline to the above endpoint by using the PortableRunner, job_endpoint set to localhost:8099 (this is the default address of the JobService). Optionally set environment_type set to LOOPBACK. For example:

```
Expand Down
25 changes: 0 additions & 25 deletions runners/flink/1.15/build.gradle

This file was deleted.

26 changes: 0 additions & 26 deletions runners/flink/1.15/job-server-container/build.gradle

This file was deleted.

31 changes: 0 additions & 31 deletions runners/flink/1.15/job-server/build.gradle

This file was deleted.

43 changes: 13 additions & 30 deletions runners/flink/flink_runner.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -173,36 +173,19 @@ dependencies {
implementation library.java.joda_time
implementation library.java.args4j

// Flink 1.15 shades all remaining scala dependencies and therefor does not depend on a specific version of Scala anymore
if (flink_version.compareTo("1.15") >= 0) {
implementation "org.apache.flink:flink-clients:$flink_version"
// Runtime dependencies are not included in Beam's generated pom.xml, so we must declare flink-clients in implementation
// configuration (https://issues.apache.org/jira/browse/BEAM-11732).
permitUnusedDeclared "org.apache.flink:flink-clients:$flink_version"

implementation "org.apache.flink:flink-streaming-java:$flink_version"
// RocksDB state backend (included in the Flink distribution)
provided "org.apache.flink:flink-statebackend-rocksdb:$flink_version"
testImplementation "org.apache.flink:flink-statebackend-rocksdb:$flink_version"
testImplementation "org.apache.flink:flink-streaming-java:$flink_version:tests"
testImplementation "org.apache.flink:flink-test-utils:$flink_version"

miniCluster "org.apache.flink:flink-runtime-web:$flink_version"
} else {
implementation "org.apache.flink:flink-clients_2.12:$flink_version"
// Runtime dependencies are not included in Beam's generated pom.xml, so we must declare flink-clients in implementation
// configuration (https://issues.apache.org/jira/browse/BEAM-11732).
permitUnusedDeclared "org.apache.flink:flink-clients_2.12:$flink_version"

implementation "org.apache.flink:flink-streaming-java_2.12:$flink_version"
// RocksDB state backend (included in the Flink distribution)
provided "org.apache.flink:flink-statebackend-rocksdb_2.12:$flink_version"
testImplementation "org.apache.flink:flink-statebackend-rocksdb_2.12:$flink_version"
testImplementation "org.apache.flink:flink-streaming-java_2.12:$flink_version:tests"
testImplementation "org.apache.flink:flink-test-utils_2.12:$flink_version"

miniCluster "org.apache.flink:flink-runtime-web_2.12:$flink_version"
}
implementation "org.apache.flink:flink-clients:$flink_version"
// Runtime dependencies are not included in Beam's generated pom.xml, so we must declare flink-clients in implementation
// configuration (https://issues.apache.org/jira/browse/BEAM-11732).
permitUnusedDeclared "org.apache.flink:flink-clients:$flink_version"

implementation "org.apache.flink:flink-streaming-java:$flink_version"
// RocksDB state backend (included in the Flink distribution)
provided "org.apache.flink:flink-statebackend-rocksdb:$flink_version"
testImplementation "org.apache.flink:flink-statebackend-rocksdb:$flink_version"
testImplementation "org.apache.flink:flink-streaming-java:$flink_version:tests"
testImplementation "org.apache.flink:flink-test-utils:$flink_version"

miniCluster "org.apache.flink:flink-runtime-web:$flink_version"

implementation "org.apache.flink:flink-core:$flink_version"
implementation "org.apache.flink:flink-metrics-core:$flink_version"
Expand Down
2 changes: 1 addition & 1 deletion sdks/typescript/src/apache_beam/runners/flink.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import { JavaJarService } from "../utils/service";
const MAGIC_HOST_NAMES = ["[local]", "[auto]"];

// These should stay in sync with gradle.properties.
const PUBLISHED_FLINK_VERSIONS = ["1.15", "1.16", "1.17", "1.18", "1.19"];
const PUBLISHED_FLINK_VERSIONS = ["1.16", "1.17", "1.18", "1.19"];

const defaultOptions = {
flinkMaster: "[local]",
Expand Down
4 changes: 0 additions & 4 deletions settings.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,6 @@ include(":runners:extensions-java:metrics")
* verify versions in website/www/site/content/en/documentation/runners/flink.md
* verify version in sdks/python/apache_beam/runners/interactive/interactive_beam.py
*/
// Flink 1.15
include(":runners:flink:1.15")
include(":runners:flink:1.15:job-server")
include(":runners:flink:1.15:job-server-container")
// Flink 1.16
include(":runners:flink:1.16")
include(":runners:flink:1.16:job-server")
Expand Down
3 changes: 1 addition & 2 deletions website/www/site/content/en/documentation/runners/flink.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,6 @@ The optional `flink_version` option may be required as well for older versions o

{{< paragraph class="language-portable" >}}
Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub:
[Flink 1.15](https://hub.docker.com/r/apache/beam_flink1.15_job_server).
[Flink 1.16](https://hub.docker.com/r/apache/beam_flink1.16_job_server).
[Flink 1.17](https://hub.docker.com/r/apache/beam_flink1.17_job_server).
[Flink 1.18](https://hub.docker.com/r/apache/beam_flink1.18_job_server).
Expand Down Expand Up @@ -350,7 +349,7 @@ To find out which version of Flink is compatible with Beam please see the table
<tr>
<td>1.15.x</td>
<td>beam-runners-flink-1.15</td>
<td>&ge; 2.40.0</td>
<td>2.40.0 - 2.60.0</td>
</tr>
<tr>
<td>1.14.x</td>
Expand Down
Loading