Skip to content

Commit

Permalink
Add GitHub Workflow Replacements for Jenkins job_PostCommit_Java_Nexm…
Browse files Browse the repository at this point in the history
…ark (#28468)
  • Loading branch information
aleksandr-dudko authored Sep 19, 2023
1 parent 3915c85 commit e6ecfac
Show file tree
Hide file tree
Showing 7 changed files with 671 additions and 0 deletions.
6 changes: 6 additions & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,12 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
| [ PostCommit Java Jpms Direct Java17 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Direct_Java17.yml) | N/A |`Run Jpms Direct Java 17 PostCommit`| [![.github/workflows/beam_PostCommit_Java_Jpms_Direct_Java17](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Direct_Java17.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Direct_Java17.yml) |
| [ PostCommit Java Jpms Flink Java11 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Flink_Java11.yml) | N/A |`Run Jpms Flink Java 11 PostCommit`| [![.github/workflows/beam_PostCommit_Java_Jpms_Flink_Java11](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Flink_Java11.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Flink_Java11.yml) |
| [ PostCommit Java Jpms Spark Java11 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Spark_Java11.yml) | N/A |`Run Jpms Spark Java 11 PostCommit`| [![.github/workflows/beam_PostCommit_Java_Jpms_Spark_Java11](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Spark_Java11.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Spark_Java11.yml) |
| [ PostCommit Java Nexmark Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Dataflow.yml) | N/A |`Run Dataflow Runner Nexmark Tests`| [![.github/workflows/beam_PostCommit_Java_Nexmark_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Dataflow.yml) |
| [ PostCommit Java Nexmark Dataflow V2 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2.yml) | N/A |`Run Dataflow Runner V2 Nexmark Tests`| [![.github/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2.yml) |
| [ PostCommit Java Nexmark Dataflow V2 Java ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml) | ['11','17'] |`Run Dataflow Runner V2 Java (matrix) Nexmark Tests`| [![.github/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml) |
| [ PostCommit Java Nexmark Direct ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Direct.yml) | N/A |`Run Direct Runner Nexmark Tests`| [![.github/workflows/beam_PostCommit_Java_Nexmark_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Direct.yml) |
| [ PostCommit Java Nexmark Flink ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Flink.yml) | N/A |`Run Flink Runner Nexmark Tests`| [![.github/workflows/beam_PostCommit_Java_Nexmark_Flink.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Flink.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Flink.yml) |
| [ PostCommit Java Nexmark Spark ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Spark.yml) | N/A |`Run Spark Runner Nexmark Tests`| [![.github/workflows/beam_PostCommit_Java_Nexmark_Spark.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Spark.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Spark.yml) |
| [ PostCommit Java PVR Flink Streaming ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Flink_Streaming.yml) | N/A |`Run Java Flink PortableValidatesRunner Streaming`| [![PostCommit Java PVR Flink Streaming](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Flink_Streaming.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Flink_Streaming.yml) |
| [ PostCommit Java PVR Samza ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Samza.yml) | N/A |`Run Java Samza PortableValidatesRunner`| [![PostCommit Java PVR Samza](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Samza.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Samza.yml) |
| [ PostCommit Java PVR Spark3 Streaming ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Spark3_Streaming.yml) | N/A |`Run Java Spark v3 PortableValidatesRunner Streaming`| [![PostCommit Java PVR Spark3 Streaming](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Spark3_Streaming.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Spark3_Streaming.yml) |
Expand Down
115 changes: 115 additions & 0 deletions .github/workflows/beam_PostCommit_Java_Nexmark_Dataflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: PostCommit Java Nexmark Dataflow

on:
issue_comment:
types: [created]
schedule:
- cron: '0 */6 * * *'
workflow_dispatch:

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.comment.body || github.event.sender.login}}'
cancel-in-progress: true

#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
permissions:
actions: write
pull-requests: write
checks: write
contents: read
deployments: read
id-token: none
issues: write
discussions: read
packages: read
pages: read
repository-projects: read
security-events: read
statuses: read

env:
GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
GRADLE_COMMAND_ARGUMENTS: |
-Pnexmark.args=--manageResources=false
--monitorJobs=true
--bigQueryTable=nexmark
--bigQueryDataset=nexmark
--project=apache-beam-testing
--resourceNameMode=QUERY_RUNNER_AND_MODE
--exportSummaryToBigQuery=true
--tempLocation=gs://temp-storage-for-perf-tests/nexmark
--influxDatabase=beam_test_metrics
--influxHost=http://10.128.0.96:8086
--baseInfluxMeasurement=nexmark
--exportSummaryToInfluxDB=true
--influxRetentionPolicy=forever
--region=us-central1
--suite=STRESS
--numWorkers=4
--maxNumWorkers=4
--autoscalingAlgorithm=NONE
--nexmarkParallel=16
--enforceEncodability=true
--enforceImmutability=true
--runner=DataflowRunner
jobs:
beam_PostCommit_Java_Nexmark_Dataflow:
name: ${{matrix.job_name}} (${{matrix.job_phrase}})
runs-on: [self-hosted, ubuntu-20.04, main]
timeout-minutes: 240
strategy:
fail-fast: false
matrix:
job_name: [beam_PostCommit_Java_Nexmark_Dataflow]
job_phrase: [Run Dataflow Runner Nexmark Tests]
streaming: [false, true]
queryLanguage: [sql, zetasql, none]
if: |
github.event_name == 'workflow_dispatch' ||
github.event_name == 'schedule' ||
github.event.comment.body == 'Run Dataflow Runner Nexmark Tests'
steps:
- uses: actions/checkout@v3
- name: Setup repository
uses: ./.github/actions/setup-action
with:
comment_phrase: ${{ matrix.job_phrase }}
github_token: ${{ secrets.GITHUB_TOKEN }}
github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
- name: run PostCommit Java Nexmark Dataflow (${{ matrix.streaming }} ${{ matrix.queryLanguage }}) script
if: matrix.queryLanguage != 'none'
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:java:testing:nexmark:run
arguments: |
-Pnexmark.runner=:runners:google-cloud-dataflow-java \
"${{ env.GRADLE_COMMAND_ARGUMENTS }}--streaming=${{ matrix.streaming }} --queryLanguage=${{ matrix.queryLanguage }}" \
- name: run PostCommit Java Nexmark Dataflow (${{ matrix.streaming }}) script
if: matrix.queryLanguage == 'none'
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:java:testing:nexmark:run
arguments: |
-Pnexmark.runner=:runners:google-cloud-dataflow-java \
"${{ env.GRADLE_COMMAND_ARGUMENTS }}--streaming=${{ matrix.streaming }}"
106 changes: 106 additions & 0 deletions .github/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: PostCommit Java Nexmark Dataflow V2

on:
issue_comment:
types: [created]
schedule:
- cron: '0 */6 * * *'
workflow_dispatch:

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.comment.body || github.event.sender.login}}'
cancel-in-progress: true

#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
permissions:
actions: write
pull-requests: write
checks: write
contents: read
deployments: read
id-token: none
issues: write
discussions: read
packages: read
pages: read
repository-projects: read
security-events: read
statuses: read

env:
GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
GRADLE_COMMAND_ARGUMENTS: |
-Pnexmark.args=--manageResources=false
--monitorJobs=true
--bigQueryTable=nexmark
--bigQueryDataset=nexmark
--project=apache-beam-testing
--resourceNameMode=QUERY_RUNNER_AND_MODE
--exportSummaryToBigQuery=false
--tempLocation=gs://temp-storage-for-perf-tests/nexmark
--influxDatabase=beam_test_metrics
--influxHost=http://10.128.0.96:8086
--baseInfluxMeasurement=nexmark
--exportSummaryToInfluxDB=true
--influxRetentionPolicy=forever
--influxTags={"runnerVersion":"V2","javaVersion":"8"}
--region=us-central1
--suite=STRESS
--numWorkers=4
--maxNumWorkers=4
--autoscalingAlgorithm=NONE
--nexmarkParallel=16
--enforceEncodability=true
--enforceImmutability=true
--runner=DataflowRunner
jobs:
beam_PostCommit_Java_Nexmark_Dataflow_V2:
name: ${{matrix.job_name}} (${{matrix.job_phrase}})
runs-on: [self-hosted, ubuntu-20.04, main]
timeout-minutes: 240
strategy:
fail-fast: false
matrix:
job_name: [beam_PostCommit_Java_Nexmark_Dataflow_V2]
job_phrase: [Run Dataflow Runner V2 Nexmark Tests]
streaming: [false, true]
if: |
github.event_name == 'workflow_dispatch' ||
github.event_name == 'schedule' ||
github.event.comment.body == 'Run Dataflow Runner V2 Nexmark Tests'
steps:
- uses: actions/checkout@v3
- name: Setup repository
uses: ./.github/actions/setup-action
with:
comment_phrase: ${{ matrix.job_phrase }}
github_token: ${{ secrets.GITHUB_TOKEN }}
github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
- name: run PostCommit Java Nexmark Dataflow V2 (streaming = ${{ matrix.streaming }}) script
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:java:testing:nexmark:run
arguments: |
-Pnexmark.runner=:runners:google-cloud-dataflow-java \
'${{ env.GRADLE_COMMAND_ARGUMENTS }}--streaming=${{ matrix.streaming }}'
116 changes: 116 additions & 0 deletions .github/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: PostCommit Java Nexmark Dataflow V2 Java

on:
issue_comment:
types: [created]
schedule:
- cron: '0 */6 * * *'
workflow_dispatch:

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.comment.body || github.event.sender.login}}'
cancel-in-progress: true

#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
permissions:
actions: write
pull-requests: write
checks: write
contents: read
deployments: read
id-token: none
issues: write
discussions: read
packages: read
pages: read
repository-projects: read
security-events: read
statuses: read

env:
GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
GRADLE_COMMAND_ARGUMENTS: |
-Pnexmark.args=--manageResources=false
--monitorJobs=true
--bigQueryTable=nexmark
--bigQueryDataset=nexmark
--project=apache-beam-testing
--resourceNameMode=QUERY_RUNNER_AND_MODE
--exportSummaryToBigQuery=false
--tempLocation=gs://temp-storage-for-perf-tests/nexmark
--influxDatabase=beam_test_metrics
--influxHost=http://10.128.0.96:8086
--baseInfluxMeasurement=nexmark
--exportSummaryToInfluxDB=true
--influxRetentionPolicy=forever
--region=us-central1
--suite=STRESS
--numWorkers=4
--maxNumWorkers=4
--autoscalingAlgorithm=NONE
--nexmarkParallel=16
--enforceEncodability=true
--enforceImmutability=true
--runner=DataflowRunner
jobs:
beam_PostCommit_Java_Nexmark_Dataflow_V2_Java:
name: ${{matrix.job_name}} (${{matrix.job_phrase_1}} ${{matrix.java_version}} ${{matrix.job_phrase_2}})
runs-on: [self-hosted, ubuntu-20.04, main]
timeout-minutes: 240
strategy:
fail-fast: false
matrix:
job_name: [beam_PostCommit_Java_Nexmark_Dataflow_V2_Java]
job_phrase_1: [Run Dataflow Runner V2 Java]
job_phrase_2: [Nexmark Tests]
streaming: [false, true]
java_version: ['11','17']
if: |
github.event_name == 'workflow_dispatch' ||
github.event_name == 'schedule' ||
(contains(github.event.comment.body, 'Run Dataflow Runner V2 Java') &&
contains(github.event.comment.body, 'Nexmark Tests'))
steps:
- uses: actions/checkout@v3
- name: Setup repository
uses: ./.github/actions/setup-action
with:
comment_phrase: ${{ matrix.job_phrase_1 }} ${{matrix.java_version}} ${{ matrix.job_phrase_2 }}
github_token: ${{ secrets.GITHUB_TOKEN }}
github_job: ${{ matrix.job_name }} ${{ matrix.job_phrase_1 }} ${{matrix.java_version}} ${{ matrix.job_phrase_2 }}
- name: Set up Java${{ matrix.java_version }}
uses: actions/[email protected]
with:
distribution: 'temurin'
java-version: ${{ matrix.java_version }}
- name: run PostCommit Java ${{matrix.java_version}} Nexmark Dataflow V2 (streaming = ${{ matrix.streaming }}) script
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:java:testing:nexmark:run
arguments: |
-PcompileAndRunTestsWithJava${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-Pnexmark.runner.version=V2 \
-Pnexmark.runner=:runners:google-cloud-dataflow-java \
'${{ env.GRADLE_COMMAND_ARGUMENTS }}--influxTags={"runnerVersion":"V2","javaVersion":"${{matrix.java_version}}"}--streaming=${{ matrix.streaming }}'
Loading

0 comments on commit e6ecfac

Please sign in to comment.