Update Google Cloud Java Libraries BOM from 26.22.0 to 26.23.0 #28656

kennknowles · 2023-09-25T21:24:40Z

Quite a lot of manual digging through transitive deps of the libraries listed at https://github.com/googleapis/java-cloud-bom/tree/v26.23.0 to figure out some of these :-(

Then @bvolpato tells me I should just run this :-)

cd /tmp; wget https://repo1.maven.org/maven2/com/google/cloud/libraries-bom/26.23.0/libraries-bom-26.23.0.pom -O base.pom && mvn help:effective-pom -f base.pom -Doutput=effective.pom && cat effective.pom | grep -v 'dependencyManagement' > cleanup.pom && mvn dependency:tree -f cleanup.pom

(still manual scraping to match up versions)

Fixes #28379

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

codecov · 2023-09-25T22:05:30Z

Codecov Report

Merging #28656 (7ac5c17) into master (b9398e0) will decrease coverage by 33.82%.
Report is 106 commits behind head on master.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #28656       +/-   ##
===========================================
- Coverage   72.21%   38.39%   -33.82%     
===========================================
  Files         684      686        +2     
  Lines      101235   101608      +373     
===========================================
- Hits        73104    39010    -34094     
- Misses      26556    61023    +34467     
  Partials     1575     1575

Flag	Coverage Δ
python	`30.01% <ø> (-52.74%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 314 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-09-25T22:34:45Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

kennknowles · 2023-09-26T00:32:15Z

run java precommit

kennknowles · 2023-09-26T13:38:14Z

Run Java_hadoop_IO_Direct PreCommit

kennknowles · 2023-09-26T13:38:40Z

On the Hadoop tests, gradle cache flake:

08:35:50 > Failed to load cache entry 10c9f03b28665c35a0f17f4ed29f54e2 for task ':sdks:java:io:hadoop-format:compileTestJava': Could not load from local cache: Timeout waiting to lock Build cache (/home/jenkins/.gradle/caches/build-cache-1). It is currently in use by another Gradle instance.

kennknowles · 2023-09-26T13:41:30Z

GCP test looks worrying, for now at least: https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Commit/3770/testReport/junit/org.apache.beam.sdk.io.gcp.spanner.changestreams/SpannerChangeStreamErrorTest/testAbortedExceptionNotRetriedithDefaultsForStreamSqlRetrySettings/

SpannerChangeStreamErrorTest.testAbortedExceptionNotRetriedithDefaultsForStreamSqlRetrySettings

org.junit.runners.model.TestTimedOutException: test timed out after 600 seconds
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at com.google.cloud.spanner.AbstractResultSet$GrpcStreamIterator.computeNext(AbstractResultSet.java:1010)
	at com.google.cloud.spanner.AbstractResultSet$GrpcStreamIterator.computeNext(AbstractResultSet.java:940)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
	at com.google.cloud.spanner.AbstractResultSet$ResumableStreamIterator.computeNext(AbstractResultSet.java:1249)
	at com.google.cloud.spanner.AbstractResultSet$ResumableStreamIterator.computeNext(AbstractResultSet.java:1085)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
	at com.google.cloud.spanner.AbstractResultSet$GrpcValueIterator.ensureReady(AbstractResultSet.java:289)
	at com.google.cloud.spanner.AbstractResultSet$GrpcValueIterator.getMetadata(AbstractResultSet.java:265)
	at com.google.cloud.spanner.AbstractResultSet$GrpcResultSet.next(AbstractResultSet.java:133)
	at com.google.cloud.spanner.SessionPool$PooledSession.determineDialect(SessionPool.java:1602)
	at com.google.cloud.spanner.SessionPool$PooledSession.access$5200(SessionPool.java:1370)
	at com.google.cloud.spanner.SessionPool.getDialect(SessionPool.java:2167)
	at com.google.cloud.spanner.DatabaseClientImpl.getDialect(DatabaseClientImpl.java:59)
	at org.apache.beam.sdk.io.gcp.spanner.SpannerIO.getDialect(SpannerIO.java:1710)
	at org.apache.beam.sdk.io.gcp.spanner.SpannerIO.access$400(SpannerIO.java:375)
	at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$ReadChangeStream.expand(SpannerIO.java:1623)
	at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$ReadChangeStream.expand(SpannerIO.java:1416)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:545)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:479)
	at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
	at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:175)
	at org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest.testAbortedExceptionNotRetriedithDefaultsForStreamSqlRetrySettings(SpannerChangeStreamErrorTest.java:234)

kennknowles · 2023-09-26T13:42:03Z

Run Java_GCP_IO_Direct PreCommit

bvolpato · 2023-09-26T13:46:57Z

json_org                                    : "org.json:json:20220320", // Keep in sync with everit-json-schema / google_cloud_platform_libraries_bom transitive deps.

everit-json-schema and google_cloud_platform_libraries_bom disagree here.

The current version 20220320 is in range for CVE-2022-45688, but likely something that we can handle outside this bump's scope.

libraries-bom transitively depends on 20230618

bvolpato

Cross-checked the diffs, LGTM, thanks Kenn!

kennknowles · 2023-09-26T16:00:12Z

Same error. Suspect it is real :-(

kennknowles · 2023-09-26T16:00:21Z

Run Java_GCP_IO_Direct PreCommit

bvolpato · 2023-09-26T16:16:19Z

09:59:25 org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest > testAbortedExceptionNotRetriedithDefaultsForStreamSqlRetrySettings FAILED
09:59:25     java.lang.AssertionError at MatcherAssert.java:18

Judging by the test, not hard to imagine that we'd have failures if they changed the defaults.

bvolpato · 2023-09-26T16:17:23Z

cc @manitgupta and @nancyxu123

bvolpato · 2023-09-27T13:59:51Z

I was pointed to @darshan-sj -- Darshan, can you please take a look at this?

kennknowles · 2023-09-27T14:10:52Z

If it is just the test, then I'm OK with a cherrypick. Otherwise I think we have to skip this BOM version.

kennknowles · 2023-09-29T14:48:34Z

I have logged in to Jenkins and ensured that this build will not be garbage collected: https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Phrase/238/

The failure is SpannerChangeStreamErrorTest.testAbortedExceptionNotRetriedithDefaultsForStreamSqlRetrySettings

It times out after 10 minutes. The log in its entirety appears to be:

Sep 26, 2023 4:17:33 PM org.apache.beam.sdk.io.gcp.spanner.SpannerAccessor getOrCreate
INFO: Connecting to SpannerConfig{projectId=my-project, instanceId=my-instance, databaseId=my-database, host=https://batch-spanner.googleapis.com/, emulatorHost=my-host, isLocalChannelProvider=true, commitDeadline=PT15S, maxCumulativeBackoff=PT900S, executeStreamingSqlRetrySettings=RetrySettings{totalTimeout=PT5M, initialRetryDelay=PT0S, retryDelayMultiplier=1.0, maxRetryDelay=PT0S, maxAttempts=0, jittered=true, initialRpcTimeout=PT1M, rpcTimeoutMultiplier=1.0, maxRpcTimeout=PT1M}, commitRetrySettings=null, retryableCodes=[UNAVAILABLE, ABORTED], rpcPriority=MEDIUM, databaseRole=null, partitionQueryTimeout=null, partitionReadTimeout=null, serviceFactory=null, dataBoostEnabled=null}

kennknowles · 2023-09-29T17:04:33Z

Now that I have had a minute to look at it rather than just copy/pasting the error, summarizing the stack trace and what it means: it seems to be hanging in getDialect which happens client-side as part of building the graph, not during pipeline execution.

kennknowles · 2023-09-29T17:06:05Z

I think the problem is with the Beam test. It is testing bug-for-bug compatibility of the getDialect call and perhaps the bug was, in fact, fixed.

kennknowles · 2023-09-29T17:07:03Z

No that is not the case. The test is expecting the request to time out but instead the test times out while the request is still active (possibly during a retry).

kennknowles · 2023-10-06T18:57:16Z

Run Java_IOs_Direct PreCommit

kennknowles · 2023-10-06T20:24:32Z

Run Java_Examples_Dataflow PreCommit

github-actions · 2023-10-14T12:13:46Z

Reminder, please take a look at this pr: @damccorm

bvolpato · 2023-10-15T22:34:54Z

Run Java PreCommit

bvolpato · 2023-10-15T22:35:31Z

Run Python_Coverage PreCommit

kennknowles · 2023-10-16T16:46:06Z

run java precommit

bvolpato

Seems like this is good to go, thanks Kenn

kennknowles · 2023-10-16T17:40:12Z

Ah, noting the "4 pending checks" that are actually green on Jenkins:

https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java17_Commit/8916/
https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Commit/3891/
https://ci-beam.apache.org/job/beam_PreCommit_Java_PVR_Flink_Batch_Commit/3448/
https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Commit/17501/

And the Python coverage is not relevant (but should probably not fail on this, so that's a bug but it is too long and flaky to kick a bunch of times here)

github-actions bot added the build label Sep 25, 2023

kennknowles force-pushed the libraries_bom branch 2 times, most recently from 4a4839d to 8ba2e3e Compare September 25, 2023 21:38

github-actions bot added the Next Action: Reviewers label Sep 25, 2023

kennknowles force-pushed the libraries_bom branch from 8ba2e3e to b396b29 Compare September 26, 2023 12:12

github-actions bot added java docker and removed java docker labels Sep 26, 2023

kennknowles force-pushed the libraries_bom branch from b396b29 to 3b44fcd Compare September 26, 2023 12:23

github-actions bot added java docker and removed java docker labels Sep 26, 2023

bvolpato approved these changes Sep 26, 2023

View reviewed changes

Update Google Cloud Java Libraries BOM

7ac5c17

kennknowles force-pushed the libraries_bom branch from 3b44fcd to 7ac5c17 Compare October 6, 2023 18:34

github-actions bot added java docker and removed java docker labels Oct 6, 2023

github-actions bot added the slow-review label Oct 14, 2023

github-actions bot removed the slow-review label Oct 15, 2023

bvolpato approved these changes Oct 16, 2023

View reviewed changes

kennknowles merged commit fddab44 into apache:master Oct 16, 2023
29 of 30 checks passed

kennknowles deleted the libraries_bom branch October 17, 2023 18:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Google Cloud Java Libraries BOM from 26.22.0 to 26.23.0 #28656

Update Google Cloud Java Libraries BOM from 26.22.0 to 26.23.0 #28656

kennknowles commented Sep 25, 2023 •

edited

Loading

codecov bot commented Sep 25, 2023 •

edited

Loading

github-actions bot commented Sep 25, 2023

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

bvolpato commented Sep 26, 2023 •

edited

Loading

bvolpato left a comment

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

bvolpato commented Sep 26, 2023

bvolpato commented Sep 26, 2023

bvolpato commented Sep 27, 2023

kennknowles commented Sep 27, 2023

kennknowles commented Sep 29, 2023

kennknowles commented Sep 29, 2023

kennknowles commented Sep 29, 2023

kennknowles commented Sep 29, 2023

kennknowles commented Oct 6, 2023

kennknowles commented Oct 6, 2023

github-actions bot commented Oct 14, 2023

bvolpato commented Oct 15, 2023

bvolpato commented Oct 15, 2023

kennknowles commented Oct 16, 2023

bvolpato left a comment

kennknowles commented Oct 16, 2023

Update Google Cloud Java Libraries BOM from 26.22.0 to 26.23.0 #28656

Update Google Cloud Java Libraries BOM from 26.22.0 to 26.23.0 #28656

Conversation

kennknowles commented Sep 25, 2023 • edited Loading

GitHub Actions Tests Status (on master branch)

codecov bot commented Sep 25, 2023 • edited Loading

Codecov Report

github-actions bot commented Sep 25, 2023

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

bvolpato commented Sep 26, 2023 • edited Loading

bvolpato left a comment

Choose a reason for hiding this comment

kennknowles commented Sep 26, 2023

kennknowles commented Sep 26, 2023

bvolpato commented Sep 26, 2023

bvolpato commented Sep 26, 2023

bvolpato commented Sep 27, 2023

kennknowles commented Sep 27, 2023

kennknowles commented Sep 29, 2023

kennknowles commented Sep 29, 2023

kennknowles commented Sep 29, 2023

kennknowles commented Sep 29, 2023

kennknowles commented Oct 6, 2023

kennknowles commented Oct 6, 2023

github-actions bot commented Oct 14, 2023

bvolpato commented Oct 15, 2023

bvolpato commented Oct 15, 2023

kennknowles commented Oct 16, 2023

bvolpato left a comment

Choose a reason for hiding this comment

kennknowles commented Oct 16, 2023

kennknowles commented Sep 25, 2023 •

edited

Loading

codecov bot commented Sep 25, 2023 •

edited

Loading

bvolpato commented Sep 26, 2023 •

edited

Loading