Skip to content

Commit

Permalink
Drop Flink 1.15 support
Browse files Browse the repository at this point in the history
  • Loading branch information
kennknowles committed Oct 17, 2024
1 parent b6e0a44 commit 7310329
Show file tree
Hide file tree
Showing 13 changed files with 21 additions and 124 deletions.
1 change: 1 addition & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@

## Deprecations

* Removed support for Flink 1.15
* X behavior is deprecated and will be removed in X versions ([#X](https://github.com/apache/beam/issues/X)).

## Bugfixes
Expand Down
2 changes: 1 addition & 1 deletion gradle.properties
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,6 @@ docker_image_default_repo_root=apache
docker_image_default_repo_prefix=beam_

# supported flink versions
flink_versions=1.15,1.16,1.17,1.18,1.19
flink_versions=1.16,1.17,1.18,1.19
# supported python versions
python_versions=3.8,3.9,3.10,3.11,3.12
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,8 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
{{if (eq .Sdk "java")}}

##### Portable
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.15`, `Flink 1.16`, `Flink 1.17`, `Flink 1.18`.
2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.16`, `Flink 1.17`, `Flink 1.18`, `Flink 1.19`.
2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.19_job_server:latest`
3. Submit the pipeline to the above endpoint by using the PortableRunner, job_endpoint set to localhost:8099 (this is the default address of the JobService). Optionally set environment_type set to LOOPBACK. For example:

```
Expand Down Expand Up @@ -233,8 +233,8 @@ mvn exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
{{end}}

{{if (eq .Sdk "python")}}
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.10`, `Flink 1.11`, `Flink 1.12`, `Flink 1.13`, `Flink 1.14`.
2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.16`, `Flink 1.17`, `Flink 1.18`, `Flink 1.19`.
2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.19_job_server:latest`
3. Submit the pipeline to the above endpoint by using the PortableRunner, job_endpoint set to localhost:8099 (this is the default address of the JobService). Optionally set environment_type set to LOOPBACK. For example:

```
Expand Down
25 changes: 0 additions & 25 deletions runners/flink/1.15/build.gradle

This file was deleted.

26 changes: 0 additions & 26 deletions runners/flink/1.15/job-server-container/build.gradle

This file was deleted.

31 changes: 0 additions & 31 deletions runners/flink/1.15/job-server/build.gradle

This file was deleted.

43 changes: 13 additions & 30 deletions runners/flink/flink_runner.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -173,36 +173,19 @@ dependencies {
implementation library.java.joda_time
implementation library.java.args4j

// Flink 1.15 shades all remaining scala dependencies and therefor does not depend on a specific version of Scala anymore
if (flink_version.compareTo("1.15") >= 0) {
implementation "org.apache.flink:flink-clients:$flink_version"
// Runtime dependencies are not included in Beam's generated pom.xml, so we must declare flink-clients in implementation
// configuration (https://issues.apache.org/jira/browse/BEAM-11732).
permitUnusedDeclared "org.apache.flink:flink-clients:$flink_version"

implementation "org.apache.flink:flink-streaming-java:$flink_version"
// RocksDB state backend (included in the Flink distribution)
provided "org.apache.flink:flink-statebackend-rocksdb:$flink_version"
testImplementation "org.apache.flink:flink-statebackend-rocksdb:$flink_version"
testImplementation "org.apache.flink:flink-streaming-java:$flink_version:tests"
testImplementation "org.apache.flink:flink-test-utils:$flink_version"

miniCluster "org.apache.flink:flink-runtime-web:$flink_version"
} else {
implementation "org.apache.flink:flink-clients_2.12:$flink_version"
// Runtime dependencies are not included in Beam's generated pom.xml, so we must declare flink-clients in implementation
// configuration (https://issues.apache.org/jira/browse/BEAM-11732).
permitUnusedDeclared "org.apache.flink:flink-clients_2.12:$flink_version"

implementation "org.apache.flink:flink-streaming-java_2.12:$flink_version"
// RocksDB state backend (included in the Flink distribution)
provided "org.apache.flink:flink-statebackend-rocksdb_2.12:$flink_version"
testImplementation "org.apache.flink:flink-statebackend-rocksdb_2.12:$flink_version"
testImplementation "org.apache.flink:flink-streaming-java_2.12:$flink_version:tests"
testImplementation "org.apache.flink:flink-test-utils_2.12:$flink_version"

miniCluster "org.apache.flink:flink-runtime-web_2.12:$flink_version"
}
implementation "org.apache.flink:flink-clients:$flink_version"
// Runtime dependencies are not included in Beam's generated pom.xml, so we must declare flink-clients in implementation
// configuration (https://issues.apache.org/jira/browse/BEAM-11732).
permitUnusedDeclared "org.apache.flink:flink-clients:$flink_version"

implementation "org.apache.flink:flink-streaming-java:$flink_version"
// RocksDB state backend (included in the Flink distribution)
provided "org.apache.flink:flink-statebackend-rocksdb:$flink_version"
testImplementation "org.apache.flink:flink-statebackend-rocksdb:$flink_version"
testImplementation "org.apache.flink:flink-streaming-java:$flink_version:tests"
testImplementation "org.apache.flink:flink-test-utils:$flink_version"

miniCluster "org.apache.flink:flink-runtime-web:$flink_version"

implementation "org.apache.flink:flink-core:$flink_version"
implementation "org.apache.flink:flink-metrics-core:$flink_version"
Expand Down
2 changes: 1 addition & 1 deletion sdks/typescript/src/apache_beam/runners/flink.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import { JavaJarService } from "../utils/service";
const MAGIC_HOST_NAMES = ["[local]", "[auto]"];

// These should stay in sync with gradle.properties.
const PUBLISHED_FLINK_VERSIONS = ["1.15", "1.16", "1.17", "1.18", "1.19"];
const PUBLISHED_FLINK_VERSIONS = ["1.16", "1.17", "1.18", "1.19"];

const defaultOptions = {
flinkMaster: "[local]",
Expand Down
4 changes: 0 additions & 4 deletions settings.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,6 @@ include(":runners:extensions-java:metrics")
* verify versions in website/www/site/content/en/documentation/runners/flink.md
* verify version in sdks/python/apache_beam/runners/interactive/interactive_beam.py
*/
// Flink 1.15
include(":runners:flink:1.15")
include(":runners:flink:1.15:job-server")
include(":runners:flink:1.15:job-server-container")
// Flink 1.16
include(":runners:flink:1.16")
include(":runners:flink:1.16:job-server")
Expand Down
3 changes: 1 addition & 2 deletions website/www/site/content/en/documentation/runners/flink.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,6 @@ The optional `flink_version` option may be required as well for older versions o

{{< paragraph class="language-portable" >}}
Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub:
[Flink 1.15](https://hub.docker.com/r/apache/beam_flink1.15_job_server).
[Flink 1.16](https://hub.docker.com/r/apache/beam_flink1.16_job_server).
[Flink 1.17](https://hub.docker.com/r/apache/beam_flink1.17_job_server).
[Flink 1.18](https://hub.docker.com/r/apache/beam_flink1.18_job_server).
Expand Down Expand Up @@ -350,7 +349,7 @@ To find out which version of Flink is compatible with Beam please see the table
<tr>
<td>1.15.x</td>
<td>beam-runners-flink-1.15</td>
<td>&ge; 2.40.0</td>
<td>2.40.0 - 2.60.0</td>
</tr>
<tr>
<td>1.14.x</td>
Expand Down

0 comments on commit 7310329

Please sign in to comment.