Setup Java 21 container #28833

Abacn · 2023-10-04T21:21:03Z

After this PR, beam will start publishing java21 dev sdk container. A wordcount compiled with Java21 locally and run on Dataflow v2 with this container has passed.

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

codecov · 2023-10-04T21:31:27Z

Codecov Report

Merging #28833 (95a06c6) into master (104c10b) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #28833      +/-   ##
==========================================
+ Coverage   72.16%   72.19%   +0.02%     
==========================================
  Files         686      686              
  Lines      101560   101590      +30     
==========================================
+ Hits        73287    73338      +51     
+ Misses      26695    26671      -24     
- Partials     1578     1581       +3

Flag	Coverage Δ
go	`53.49% <ø> (+0.07%)`	⬆️
python	`82.61% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 19 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Abacn · 2023-10-06T02:03:16Z

Example wordcount job (maven project) on Dataflow: https://console.cloud.google.com/dataflow/jobs/us-central1/2023-10-05_17_20_31-16427189156223040319?project=google.com%3Aclouddfe

However not able to setup Java21 test within beam repo yet as gradle not yet support Java 21. Currently Java invocation fails with

Unrecognized option: --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

but if not providing these args, it fails with

warning: The Checker Framework is tested with JDK 8, 11, 17, and 18. You are using version 21.
error: warnings found and -Werror specified
/Users/yathu/beam/sdks/java/testing/jpms-tests/src/main/java/org/apache/beam/sdk/jpmstests/WordCount.java:38: error: [type.checking.not.run] NullnessChecker did not run because of a previous error issued by javac
public class WordCount {

Abacn · 2023-10-06T02:17:26Z

.github/workflows/beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java11.yml

@@ -95,7 +95,7 @@ jobs:
            -PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
            -Prunner=:runners:google-cloud-dataflow-java \
            -Prunner.version=V2 \
-            -PcompileAndRunTestsWithJava11 \
+            -PtestJavaVersion=11 \


this flag change is to address #28726 (comment) (later planned as part of Java21 support: #28120 (comment))

Abacn · 2023-10-06T02:21:21Z

sdks/java/container/java21/option-jamm.json

@@ -0,0 +1,10 @@
+{


although same options are added as java17, it seems jamm still not working on Java21:

JVM prevents jamm from accessing subgraph - cache sizes may be underestimated

support left as next task

github-actions · 2023-10-06T05:34:54Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Abacn · 2023-10-06T14:11:53Z

sdks/java/container/Dockerfile

 ARG java_version
-FROM eclipse-temurin:${java_version}
+FROM ${base_image}:${java_version}


we switched from openjdk to temurin in #25402 due to that openjdk image for java8 and 11 was deprecated. For now we still use temurin for 8,11,17; openjdk for 21 temporarily until upstream https://adoptium.net/blog/2023/09/temurin21-delay/ resolved

github-actions · 2023-10-06T16:36:44Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.
R: @damccorm for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Abacn · 2023-10-09T20:33:24Z

Because this PR changed test parameters (Jenkins, GHA), should be merged after 2.51.0 release finalized

Abacn · 2023-10-10T18:27:02Z

Note that temurin java21 has released in Oct 10th: https://adoptium.net/temurin/releases/

Abacn · 2023-10-11T14:21:03Z

Run Java PreCommit

Abacn · 2023-10-11T14:21:19Z

Run PythonDocker PreCommit

Abacn · 2023-10-11T14:21:26Z

Run Python_PVR_Flink PreCommit

Abacn · 2023-10-11T14:21:33Z

Run Portable_Python PreCommit

Abacn · 2023-10-11T14:22:37Z

Run Java_PVR_Flink_Docker PreCommit

Abacn · 2023-10-12T01:15:59Z

tests either passed on GHA or Jenkins, merging

This reverts commit e8e3814.

* Add Java21 container * fix spotless * Update website * Fix jamm * align configs in recent change

This reverts commit e8e3814.

* Add Java21 container * fix spotless * Update website * Fix jamm * align configs in recent change

* Setup Java 21 container (#28833) * Add Java21 container * fix spotless * Update website * Fix jamm * align configs in recent change * incremental fixes * add more comments for when jpms test enabled

* Setup Java 21 container (apache#28833) * Add Java21 container * fix spotless * Update website * Fix jamm * align configs in recent change * incremental fixes * add more comments for when jpms test enabled

Abacn marked this pull request as draft October 4, 2023 21:21

github-actions bot added java build docker runners core labels Oct 4, 2023

Abacn force-pushed the java21container branch from 9fff59e to 2f1038f Compare October 4, 2023 21:22

Abacn force-pushed the java21container branch 4 times, most recently from 2552df3 to ca9ad81 Compare October 5, 2023 22:07

github-actions bot added the dataflow label Oct 5, 2023

Abacn force-pushed the java21container branch from ca9ad81 to af5e130 Compare October 6, 2023 01:55

github-actions bot added infra spark labels Oct 6, 2023

Abacn force-pushed the java21container branch from af5e130 to 4e84205 Compare October 6, 2023 02:09

Add Java21 container

06de3c9

Abacn force-pushed the java21container branch from 4e84205 to 06de3c9 Compare October 6, 2023 02:16

Abacn commented Oct 6, 2023

View reviewed changes

Abacn changed the title ~~[Draft] Setup Java 21 container~~ Setup Java 21 container Oct 6, 2023

Abacn commented Oct 6, 2023

View reviewed changes

Abacn marked this pull request as ready for review October 6, 2023 02:25

Abacn commented Oct 6, 2023

View reviewed changes

fix spotless

fe8543f

Abacn force-pushed the java21container branch from 310088e to fe8543f Compare October 6, 2023 15:00

Update website

95b0d3b

github-actions bot added the website label Oct 6, 2023

github-actions bot added the Next Action: Reviewers label Oct 6, 2023

kennknowles approved these changes Oct 9, 2023

View reviewed changes

Abacn changed the title ~~Setup Java 21 container~~ [Do not merge until 2.51 released] Setup Java 21 container Oct 9, 2023

Abacn mentioned this pull request Oct 10, 2023

Setup Java 21 container #28918

Merged

3 tasks

Fix jamm

f9732e8

Abacn added 2 commits October 11, 2023 16:46

Merge branch 'master' into java21container

6817ce0

align configs in recent change

95a06c6

Abacn changed the title ~~[Do not merge until 2.51 released] Setup Java 21 container~~ Setup Java 21 container Oct 12, 2023

Abacn merged commit e8e3814 into apache:master Oct 12, 2023
40 of 43 checks passed

Abacn deleted the java21container branch October 12, 2023 01:16

damccorm added a commit that referenced this pull request Oct 12, 2023

Revert "Setup Java 21 container (#28833)"

d71a949

This reverts commit e8e3814.

This was referenced Oct 12, 2023

Run examples on Java21 #28969

Merged

Revert "Setup Java 21 container (#28833)" #28968

Merged

Abacn pushed a commit that referenced this pull request Oct 13, 2023

Revert "Setup Java 21 container (#28833)" (#28968)

c0af78a

This reverts commit e8e3814.

damondouglas pushed a commit to damondouglas/beam that referenced this pull request Oct 16, 2023

Setup Java 21 container (apache#28833)

eedd1a8

* Add Java21 container * fix spotless * Update website * Fix jamm * align configs in recent change

damondouglas pushed a commit to damondouglas/beam that referenced this pull request Oct 16, 2023

Revert "Setup Java 21 container (apache#28833)" (apache#28968)

992627b

This reverts commit e8e3814.

Abacn added a commit to Abacn/beam that referenced this pull request Oct 17, 2023

Setup Java 21 container (apache#28833)

8b2a7de

* Add Java21 container * fix spotless * Update website * Fix jamm * align configs in recent change

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup Java 21 container #28833

Setup Java 21 container #28833

Abacn commented Oct 4, 2023 •

edited

Loading

codecov bot commented Oct 4, 2023 •

edited

Loading

Abacn commented Oct 6, 2023

Abacn Oct 6, 2023

Abacn Oct 6, 2023

github-actions bot commented Oct 6, 2023

Abacn Oct 6, 2023

github-actions bot commented Oct 6, 2023

Abacn commented Oct 9, 2023

Abacn commented Oct 10, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 12, 2023

Setup Java 21 container #28833

Setup Java 21 container #28833

Conversation

Abacn commented Oct 4, 2023 • edited Loading

GitHub Actions Tests Status (on master branch)

codecov bot commented Oct 4, 2023 • edited Loading

Codecov Report

Abacn commented Oct 6, 2023

Abacn Oct 6, 2023

Choose a reason for hiding this comment

Abacn Oct 6, 2023

Choose a reason for hiding this comment

github-actions bot commented Oct 6, 2023

Abacn Oct 6, 2023

Choose a reason for hiding this comment

github-actions bot commented Oct 6, 2023

Abacn commented Oct 9, 2023

Abacn commented Oct 10, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 11, 2023

Abacn commented Oct 12, 2023

Abacn commented Oct 4, 2023 •

edited

Loading

codecov bot commented Oct 4, 2023 •

edited

Loading