Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sync with open source how #118

Draft
wants to merge 5,605 commits into
base: li_trunk
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5605 commits
Select commit Hold shift + click to select a range
9394f85
[JdbcIO] Adding disableAutoCommit flag (#32988)
cwashcraft Nov 13, 2024
a6daf6d
Bump cloud.google.com/go/bigquery from 1.63.1 to 1.64.0 in /sdks (#32…
dependabot[bot] Nov 13, 2024
87e251a
Add Kafka tag to review bot (#32975)
Abacn Nov 13, 2024
f25ac69
Lineage support for JdbcIO (#33062)
Abacn Nov 13, 2024
63fc0db
Update dataframes to PEP 585 typing
jrmccluskey Nov 13, 2024
b62e8c4
[KafkaIO] Fix per-split metric updates for KafkaUnboundedReader and R…
sjvanrossum Nov 13, 2024
ff5feed
Introduce pipeline options to disable user counter and user stringset…
Abacn Nov 13, 2024
c6a7354
SolaceIO write connector (#32060)
iht Nov 13, 2024
c643645
fix imports
jrmccluskey Nov 13, 2024
306c6d7
Share AvgRecordSize and KafkaLatestOffsetEstimator caches among DoFns…
sjvanrossum Nov 13, 2024
50ed69a
Portable Managed BigQuery destinations (#33017)
ahmedabu98 Nov 13, 2024
7f268ac
Disable gradle cache for gcp expansion service (#33099)
claudevdm Nov 13, 2024
0b1b154
[YAML] Add a new StripErrorMetadata transform. (#33094)
robertwb Nov 13, 2024
cb06b1b
Set streaming engine option to fix V1 tests (#33100)
ahmedabu98 Nov 13, 2024
3c664e9
Moving to 2.62.0-SNAPSHOT on master branch.
Nov 13, 2024
965c3c6
Merge pull request #33098 from jrmccluskey/dataframesUpdate
liferoad Nov 13, 2024
38415b8
Make AvroUtils compatible with older versions of Avro (#33102)
damccorm Nov 13, 2024
6a45624
fix JDBC providers (#32985)
Polber Nov 13, 2024
f4d07c4
Add logging to see which topic each split is reading from (#33031)
Naireen Nov 13, 2024
bff3eac
Update names.py (#33107)
tvalentyn Nov 13, 2024
ab5c069
[yaml] Fix examples catalog tests (#33027)
Polber Nov 13, 2024
2d69dde
Merge branch 'master' into update-type-hints
robertwb Nov 14, 2024
c694701
Add 2.62.0 to changes (#33116)
damccorm Nov 14, 2024
7accd8d
Fix small typos in release email (#33118)
damccorm Nov 14, 2024
f38edb7
Adds detailed instructions on how to execute ExampleEchoPipeline.java…
svetakvsundhar Nov 14, 2024
667269a
Add temp location to batch file loads (#33115)
claudevdm Nov 14, 2024
0cafb46
Build docker RC on self-hosted (#33114)
damccorm Nov 14, 2024
df3b6bb
Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#33127)
dependabot[bot] Nov 15, 2024
20fe987
Merge branch 'master' into numpy-2-update-deps
liferoad Nov 15, 2024
23faf28
updated the deps
liferoad Nov 15, 2024
7650a6d
Merge pull request #32756 from liferoad/numpy-2-update-deps
liferoad Nov 15, 2024
cc57bde
Fix vllm typo
damccorm Nov 15, 2024
5bfacc9
Add ability to encode and decode histogram data to portable runners (…
Naireen Nov 15, 2024
bc28ea4
Merge pull request #33131 from apache/users/damccorm/typo
liferoad Nov 15, 2024
8005d34
Disable gradle cache for transform service shadowJar (#33130)
claudevdm Nov 15, 2024
4d6dcd5
Fix the broken flink tests by optimizing conf (#33135)
liferoad Nov 16, 2024
4c3b736
add case study for yelp streaming
hspandher Nov 18, 2024
030c975
Use highmem for beam_PostCommit_Python_Examples_Flink.yml (#33138)
liferoad Nov 18, 2024
8da9bbe
Revert "Merge pull request #32705: fix schema inference for parameter…
Abacn Nov 18, 2024
e6f9d83
Bump google.golang.org/protobuf from 1.35.1 to 1.35.2 in /sdks (#33142)
dependabot[bot] Nov 18, 2024
366d89f
docs: missing word (#33149)
wikey Nov 18, 2024
de6965a
Enable Dataflow managed service for Python tests (#33134)
ahmedabu98 Nov 18, 2024
6edfcd4
Revert "Enable Dataflow managed service for Python tests (#33134)" (#…
damccorm Nov 18, 2024
0d894a7
Update BEAM_DEV_SDK_CONTAINER_TAG (#33153)
claudevdm Nov 18, 2024
22ea62c
numpy.int64 type is not serialized correctly in Python 3.11 and Pytho…
liferoad Nov 18, 2024
a568bb8
Plug transform annotations thru cross-lang interface (#33150)
ahmedabu98 Nov 18, 2024
0871f93
Use fallback reviewers for bot (#33155)
damccorm Nov 18, 2024
bff2cbb
Use random gcs bucket name (#33151)
claudevdm Nov 18, 2024
e939be3
Windowing Support for the Dask Runner (#32941)
alxmrs Nov 18, 2024
d120140
Don't use __getstate__ for pickling for objects with a custom __reduc…
robertwb Nov 18, 2024
3b759f2
Bump avro, fix CVE-2024-47561 (#32770)
damccorm Nov 18, 2024
c57553c
Added replaceGcsFilesWithLocalFiles (#33006)
liferoad Nov 19, 2024
75fd964
Merge pull request #32872 Modernize python type hints for apache_beam.
robertwb Nov 19, 2024
daee3a5
Merge pull request #33143 from hspandher/yelp_streaming_case_study
aaltay Nov 19, 2024
412d8b0
updated the yelp blog
liferoad Nov 19, 2024
2ed33b5
Merge pull request #33161 from liferoad/yelp-fix
aaltay Nov 19, 2024
5229c93
add shutdown and start mechanics to windmill streams (#32774)
m-trieu Nov 19, 2024
210e38c
add highmem22 runners, scale to zero and assign flink job to highmem2…
volatilemolotov Nov 19, 2024
092bc2e
Pin notebook dependencies (#33123)
damccorm Nov 19, 2024
fb93331
Create workflow for republishing released containers (#33167)
damccorm Nov 19, 2024
ca92726
Update vllm colab to work with asyncio requirements (#33122)
damccorm Nov 19, 2024
baa1591
Revert "Update vllm colab to work with asyncio requirements (#33122)"…
damccorm Nov 19, 2024
fd7de78
Fix notebook escaping
damccorm Nov 19, 2024
fd6b133
Fix notebook escaping
damccorm Nov 19, 2024
12a056d
Add BigQuery enrichment bug to changes.md and blogs (#33165)
claudevdm Nov 19, 2024
37c3e6d
Merge pull request #33046 Nested Beam Row descriptions.
robertwb Nov 19, 2024
97de2ae
Publish updated containers regularly (#33169)
damccorm Nov 19, 2024
e91c46c
[flink] #33145 cache StateNamespace#stringKey in FlinkStateInternals
Nov 13, 2024
225b252
Merge pull request #33146: #33145 cache namespace string key in Flink…
je-ik Nov 20, 2024
e1e0934
use the tight range for dask (#33172)
liferoad Nov 20, 2024
3f8fabe
Update dask_runner_tests.yml (#33177)
liferoad Nov 20, 2024
fc9083b
Create Python SDK Distroless variant (#33160)
damondouglas Nov 20, 2024
d2e9928
Revert "Fixed the broken beam python on flink with PortableRunner" (#…
Abacn Nov 20, 2024
5d3088f
Bump zetasql version to 2024.11.1 (#32902)
Abacn Nov 20, 2024
10c7eb3
fix the flink runner doc (#33182)
liferoad Nov 20, 2024
a06454a
Upgrade GCP-BOM to 26.49.0 (#32864)
Abacn Nov 20, 2024
bd1471e
[yaml] Enrichment transform doc improvements (#33186)
Polber Nov 22, 2024
d4a9ca8
Add input boxes for required user inputs (#33183)
claudevdm Nov 22, 2024
4073aba
Bump org.sonarqube from 3.0 to 6.0.0.5145 (#33174)
dependabot[bot] Nov 22, 2024
b70375d
Revert "Bump org.sonarqube from 3.0 to 6.0.0.5145 (#33174)" (#33193)
damccorm Nov 22, 2024
d342dd3
Remove the default machine types (#33191)
liferoad Nov 22, 2024
09c1c9e
Enable Java SDK Distroless container image variants. (#33173)
damondouglas Nov 23, 2024
e5bd69d
Remove leftover print statement in SpannerIO (#33200)
Abacn Nov 24, 2024
60f8793
Use --enable-component-gateway when creating the flink cluster (#33198)
liferoad Nov 25, 2024
78630d1
Add a new precommit workflow to test flink container (#33206)
liferoad Nov 25, 2024
9bdc9a5
Remove sickbay GitHub Actions (#33214)
kennknowles Nov 25, 2024
8f75c61
More complete error message for StripErrorMetadata. (#33110)
robertwb Nov 25, 2024
2e4417f
Update website for 2.61.0 release (#33117)
damccorm Nov 25, 2024
c2f5686
Move MetricAggregator to its only use in BundleBasedDirectRunner.
robertwb Nov 22, 2024
aa939c1
Consolidate implementation of metric cells.
robertwb Nov 22, 2024
ea93ce5
Fixed the new flink container precommit (#33217)
liferoad Nov 25, 2024
c0ab7e5
ZetaSQL test Java version fixes (#33213)
Abacn Nov 25, 2024
3c9a60b
Type hints.
robertwb Nov 25, 2024
d37d141
Merge pull request #33195 Simplify metrics in preparation for adding …
robertwb Nov 26, 2024
ad8545c
[Accenture Baltics] Case Study (#33215)
liferoad Nov 26, 2024
aa21e4a
Integrate direct path with StreamingDataflowWorker code path (#32778)
m-trieu Nov 26, 2024
d745198
Change the Java sdk harness cache timeout for bundle processors to b…
scwhittle Nov 26, 2024
ddfd1a2
Fixed beam_PreCommit_Flink_Container.yml (#33219)
liferoad Nov 26, 2024
aeefa4f
Make the warning about JSON BigQuery column type more precise (#33121)
turb Nov 26, 2024
5f59781
updated the pic (#33222)
liferoad Nov 26, 2024
e1b7b3f
Increase Go test coverage run timeout to 25 minutes
kennknowles Nov 26, 2024
b4673d4
Merge pull request #33223: Increase Go test coverage run timeout to 2…
kennknowles Nov 26, 2024
292da72
Add beam_PostCommit_Python_ValidatesDistrolessContainer_Dataflow (#33…
damondouglas Nov 26, 2024
54eb50d
Changes maxBufferingDuration payload to be taken from millis directly…
acrites Nov 26, 2024
720b824
use direct executor to deflake tests (#33187)
m-trieu Nov 26, 2024
0d55231
Better error messaging for chain-type pipeline (#33210)
mravi Nov 26, 2024
9138697
Cleanup Iceberg test files (#33218)
ahmedabu98 Nov 26, 2024
159c830
Update beam_PreCommit_Flink_Container.yml (#33230)
liferoad Nov 26, 2024
409a31a
Add bounded Trie metric type.
robertwb Nov 22, 2024
f83c683
Bounded Trie proto representation.
robertwb Nov 25, 2024
d8358cb
Bounded Trie translation.
robertwb Nov 26, 2024
5a6c902
Make the model names unique and fix the tox package deps for tensorfl…
liferoad Nov 27, 2024
9696201
Unpin Dataflow legacy worker container for Nexmark test (#33224)
Abacn Nov 27, 2024
9560fe1
SolaceIO: refactor to allow inheritance of BasicAuthSempClient (#32400)
bzablocki Nov 27, 2024
d723216
fixed ML tests (#33234)
liferoad Nov 27, 2024
d014249
Remove (long) obsolete declaration of unsupported features that are n…
robertwb Nov 27, 2024
4d2d2ce
Add Java Distroless Github workflow check (#33233)
damondouglas Nov 27, 2024
e8dd3c6
Fix python distroless workflow (#33232)
damondouglas Nov 27, 2024
95322c6
bump hadoop version (#33011)
Abacn Nov 27, 2024
f28e65f
Update beam_LoadTests_Python_Combine_Flink_Streaming.yml (#33241)
liferoad Dec 2, 2024
58e1625
update 2.60.0 changelog to include a fix in Bigtable (#33053)
mutianf Dec 2, 2024
727a865
Add safe to ignore category to dashboard/alerting (#33168)
damccorm Dec 2, 2024
e279e55
SolaceIO.Read: handle occasional cases when finalizeCheckpoint is not…
bzablocki Dec 2, 2024
0ec76af
Fix java validatesdistrolesscontainer dataflow workflow (#33245)
damondouglas Dec 2, 2024
2481d4b
Bump golang.org/x/net from 0.30.0 to 0.31.0 in /sdks (#33111)
dependabot[bot] Dec 2, 2024
2141dd7
Bump cloud.google.com/go/spanner from 1.70.0 to 1.73.0 in /sdks (#33126)
dependabot[bot] Dec 2, 2024
7e25649
Revert "bump hadoop version (#33011)" (#33257)
Abacn Dec 2, 2024
4356253
Revert "SolaceIO.Read: handle occasional cases when finalizeCheckpoin…
bzablocki Dec 2, 2024
9f16557
Clean up changelog (#33260)
damccorm Dec 2, 2024
afdc5ea
Revert "Revert "Update vllm colab to work with asyncio requirements""…
damccorm Dec 2, 2024
ccffdba
Remove dupe metadata (#33261)
damccorm Dec 2, 2024
abd03fe
Bump org.javacc.javacc from 3.0.2 to 3.0.3 (#33139)
dependabot[bot] Dec 2, 2024
160ffd5
[CHANGES] Add prism note to 2.61 release blog and CHANGES.md (#33262)
lostluck Dec 3, 2024
63b8dd1
Remove unnecessary branches on Python version in typehint code (#33258)
jrmccluskey Dec 3, 2024
a938fa5
Bump github.com/aws/aws-sdk-go-v2 from 1.32.4 to 1.32.6 in /sdks (#33…
dependabot[bot] Dec 3, 2024
532880a
Bump golang.org/x/oauth2 from 0.23.0 to 0.24.0 in /sdks (#33141)
dependabot[bot] Dec 3, 2024
63d89cd
Propagate gcs-connector options to GcsUtil (#32769)
clairemcginty Dec 3, 2024
2fe725d
Update republish workflow with latest beam (#33194)
damccorm Dec 3, 2024
95a0845
jdbcio write batch config changes (#33205)
mravi Dec 3, 2024
697a843
Tour of Beam: formatting/typos (#33249)
thread-sleep Dec 3, 2024
3201c57
[yaml] error_handling normalization follow-up
Polber Nov 8, 2024
edd3528
Add a pointer to java provider example. (#33240)
robertwb Dec 4, 2024
a9a82a1
[yaml] remove PubSubLite from API docs (#33184)
Polber Dec 4, 2024
ed163f6
Merge pull request #33225 Take maxBufferingDuration payload from mill…
robertwb Dec 4, 2024
d6cad34
Merge pull request #33060 [yaml] error_handling normalization follow-up
robertwb Dec 4, 2024
2f788f6
use standard sql (#33278)
ahmedabu98 Dec 4, 2024
78bde63
[yaml] Fix YAML join with upstream ReadFromPubSub (#33273)
Polber Dec 4, 2024
f080be0
[YAML] Make datetime available for jinja templatization.
robertwb Dec 4, 2024
56d6a46
[YAML] Document the yaml provider include syntax.
robertwb Dec 4, 2024
40423bc
add iceberg gcp dependency (#33281)
ahmedabu98 Dec 4, 2024
122368f
Fix TypeError: 'type' object is not subscriptable (#33280)
damondouglas Dec 4, 2024
2e43e29
[#32222] Actually maintain the heap invariant for timers. (#33270)
lostluck Dec 4, 2024
7ef6d36
[YAML] Document jinja templatization features.
robertwb Dec 5, 2024
2bdbbdc
Bump cloud.google.com/go/pubsub from 1.45.1 to 1.45.3 in /sdks (#33289)
dependabot[bot] Dec 5, 2024
d24d838
Bump cloud.google.com/go/storage from 1.45.0 to 1.47.0 in /sdks (#33266)
dependabot[bot] Dec 5, 2024
68e3c0d
Small string fix for error message (#33295)
damccorm Dec 5, 2024
2344fdc
[#32211] Fail job on SDK worker disconnect. (#33298)
lostluck Dec 5, 2024
e509e70
Synchronize JdbcSchemaIO provider and Python transform configuration …
damondouglas Dec 5, 2024
7df49a5
Clarificaiton on yaml invocation.
robertwb Dec 5, 2024
1de636b
Merge pull request #33287 [YAML] Document jinja templatization features.
robertwb Dec 5, 2024
d379968
Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks (#33304)
dependabot[bot] Dec 6, 2024
222086b
Bump golang.org/x/text from 0.20.0 to 0.21.0 in /sdks (#33305)
dependabot[bot] Dec 6, 2024
288c156
Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#33303)
dependabot[bot] Dec 6, 2024
cd02853
[infra] #33309 automatically apply spotless
je-ik Dec 5, 2024
89ae2c1
Enable managed service in Python tests (#33190)
ahmedabu98 Dec 6, 2024
0bf6f69
[yaml] various inline provider doc fixes (#33296)
Polber Dec 6, 2024
d138b75
Add throttling metrics and retries to vertex embeddings (#33311)
damccorm Dec 6, 2024
1712964
Fix FlatMapTuple typehint bug (#33307)
hjtran Dec 6, 2024
0dae9c9
Polish FlatMapTuple and MapTuple docstrings (#33316)
hjtran Dec 7, 2024
85bff0d
Reapply "bump hadoop version (#33011)" (#33312)
Abacn Dec 9, 2024
a806c0e
Add Dataflow Cost Benchmark framework to Beam Python (#33297)
jrmccluskey Dec 9, 2024
e1245d9
Merge pull request #33310: [infra] #33309 automatically apply spotless
je-ik Dec 9, 2024
354e8e3
Replace None metric values with 0.0 in Dataflow Cost Benchmark (#33336)
jrmccluskey Dec 10, 2024
6d8e02e
[java] BQ: add missing avro conversions to BQ TableRow (#33221)
RustedBones Dec 10, 2024
76bcec0
Remove the use of google-github-actions/auth@v1 (#33338)
damondouglas Dec 10, 2024
b4fbf89
Add hadoop-auth (#33342)
ahmedabu98 Dec 10, 2024
7a94b61
Bump golang.org/x/net from 0.31.0 to 0.32.0 in /sdks (#33326)
dependabot[bot] Dec 11, 2024
04b059f
Remove mandatory beam-sdks-io-kafka dependency for dataflow worker ja…
Abacn Dec 11, 2024
b26e2fa
Bump github.com/nats-io/nats-server/v2 from 2.10.22 to 2.10.23 in /sd…
dependabot[bot] Dec 11, 2024
00ec6cc
fix Error processing result file: CData section too big found in publ…
stankiewicz Dec 11, 2024
e2ff659
Reapply "SolaceIO.Read: handle occasional cases when finalizeCheckpoi…
bzablocki Dec 11, 2024
a272823
Update confluent version to fix CVE-2024-26308 CVE-2024-25710 (#32674)
stankiewicz Dec 11, 2024
3dcd104
Update reference.
robertwb Dec 11, 2024
d9092a5
Merge pull request #33196 Add bounded Trie metric type to Python.
robertwb Dec 11, 2024
bb0c910
Bump cloud.google.com/go/profiler from 0.4.1 to 0.4.2 in /sdks (#33351)
dependabot[bot] Dec 12, 2024
224900c
Bump cloud.google.com/go/storage from 1.47.0 to 1.48.0 in /sdks (#33327)
dependabot[bot] Dec 12, 2024
272feb8
Fix custom coder not being used in Reshuffle (global window) (#33339)
shunping Dec 12, 2024
7b88337
Bump golang.org/x/crypto from 0.30.0 to 0.31.0 in /sdks (#33366)
dependabot[bot] Dec 12, 2024
fd17dce
Update numpy requirement from <2.2.0,>=1.14.3 to >=1.14.3,<2.3.0 in /…
dependabot[bot] Dec 12, 2024
2efd124
Switch to use unshaded hive-exec for io expansion service (#33349)
Abacn Dec 12, 2024
e9c0d35
Bump cloud.google.com/go/bigquery from 1.64.0 to 1.65.0 in /sdks (#33…
dependabot[bot] Dec 12, 2024
de5e8eb
[#32211] Support OnWindowExpiration in Prism. (#33337)
lostluck Dec 12, 2024
02f9cb9
Bump google.golang.org/api from 0.210.0 to 0.211.0 in /sdks (#33374)
dependabot[bot] Dec 13, 2024
994d2f0
Fix Apache snapshot repo usage (#33370)
adoroszlai Dec 13, 2024
a6061fe
Batch optimized SparkRunner groupByKey (#33322)
twosom Dec 13, 2024
011ec94
remove flaky tests that terminate before read. (#33371)
lostluck Dec 13, 2024
f92dde1
Clean up Java Tests GitHub workflow (#33354)
damondouglas Dec 13, 2024
88398f3
fix BoundedTrieData get_result return type hint
Polber Dec 13, 2024
9d1eea4
End to end BoundedTrie metrics through worker and runner APIs.
robertwb Nov 27, 2024
0768e10
Add missing pxd definition.
robertwb Dec 14, 2024
2aab9cd
Fix custom coders not being used in Reshuffle (non global window) (#3…
shunping Dec 14, 2024
d7502fa
Merge pull request #33377 Fix BoundedTrieData get_result return type …
robertwb Dec 14, 2024
5918e6d
Skip unsupported runners.
robertwb Dec 14, 2024
049f567
In contributor docs, don't try to download releases from snapshot repo
raboof Dec 16, 2024
f77efba
[yaml] Add use cases for Enrichment transform in YAML (#32289)
reeba212 Dec 16, 2024
0eb82b7
Fix duplicate yml entry causing test result dashboard sync to fail
kennknowles Dec 16, 2024
8884d12
Merge pull request #33387: Fix duplicate yml entry causing test resul…
kennknowles Dec 16, 2024
74b723d
Fix yaml error in Flink Java8 ValidatesRunner
kennknowles Dec 16, 2024
4f70ca1
Merge pull request #33390: Fix yaml error in Flink Java8 ValidatesRunner
kennknowles Dec 16, 2024
0bd6639
Merge pull request #33282 [YAML] Make datetime available for jinja te…
robertwb Dec 16, 2024
002d3fd
Merge pull request #33283 [YAML] Document the yaml provider include s…
robertwb Dec 16, 2024
0f4c647
yapf
robertwb Dec 16, 2024
4bac374
chore: upgrade google_cloud_bigdataoss_version to 2.2.26 in order to …
ldetmer Dec 16, 2024
3dc2b2e
Bump google.golang.org/protobuf from 1.35.2 to 1.36.0 in /sdks (#33397)
dependabot[bot] Dec 17, 2024
c5a5be5
Rag chunking embedding (#33364)
claudevdm Dec 17, 2024
d34409e
Bump google.golang.org/api from 0.211.0 to 0.212.0 in /sdks (#33398)
dependabot[bot] Dec 17, 2024
f7a7bdd
Faster default coder for unknown windows. (#33382)
robertwb Dec 17, 2024
e50a136
Support creating BigLake managed tables (#33125)
ahmedabu98 Dec 17, 2024
c83444d
Docstring fix.
robertwb Dec 17, 2024
9232cd8
Merge pull request #33378 End to end BoundedTrie metrics through work…
robertwb Dec 17, 2024
8e1e124
[#32929] Add OrderedListState support to Prism. (#33350)
lostluck Dec 17, 2024
9ef19a9
Migrate lineage counters to bounded tries.
robertwb Dec 13, 2024
76db105
Add lineage support for local files.
robertwb Dec 13, 2024
30288ae
Remove file-specific lineage bounding.
robertwb Dec 14, 2024
0e37501
Add TF MNIST classification cost benchmark (#33391)
jrmccluskey Dec 17, 2024
e27eced
Do no throw exceptions if commitSync fails in KafkaUnboundedReader. (…
an2x Dec 17, 2024
470d7d6
Add missing to_type_hint to WindowedValueCoder (#33403)
shunping Dec 17, 2024
a9f50fa
Python ExternalTransformProvider improvements (#33359)
ahmedabu98 Dec 17, 2024
286e29c
[Managed Iceberg] Support partitioning time types (year, month, day, …
ahmedabu98 Dec 17, 2024
d704422
Add missing imports.
robertwb Dec 17, 2024
e68a79c
Validate circular reference for yaml (#33208)
mravi Dec 18, 2024
71a5ced
Deserialization fix.
robertwb Dec 18, 2024
4cbf257
Revert three commits related to supporting custom coder in reshuffle
shunping Dec 18, 2024
f476417
Added support for SparkRunner streaming stateful processing (#33267)
twosom Dec 18, 2024
14f7caf
Update lineage query function.
robertwb Dec 18, 2024
d17f77a
Merge pull request #33386: In contributor docs, don't try to download…
kennknowles Dec 18, 2024
330f57b
[YAML] Better docs for Filter and MapToFields. (#33274)
robertwb Dec 18, 2024
116df9f
Fix env variable loading in Cost Benchmark workflow (#33404)
jrmccluskey Dec 18, 2024
e9424b9
Merge pull request #33414 from shunping/revert-reshuffle-changes
chamikaramj Dec 18, 2024
5eed396
fix error on start-build-env.sh (#33401)
marblejenka Dec 18, 2024
ec2e150
Moving to 2.63.0-SNAPSHOT on master branch.
Dec 19, 2024
ca8a35a
Bump Iceberg to v1.6.1 (#33294)
regadas Dec 19, 2024
55b0ce9
Add 2.63.0 section to CHANGES.md
kennknowles Dec 19, 2024
fe6b7aa
Merge pull request #33423: Add 2.63.0 section to CHANGES.md
kennknowles Dec 19, 2024
8fee3ca
Improve existing Python multi-lang SchemaTransform examples (#33361)
ahmedabu98 Dec 19, 2024
e8df26f
Merge pull request #33381 Migrate lineage counters to bounded tries.
robertwb Dec 19, 2024
a51a0e1
Move Dataflow Python Managed tests to prod (#33426)
ahmedabu98 Dec 19, 2024
30f128b
Bump github.com/docker/docker in /sdks (#33420)
dependabot[bot] Dec 20, 2024
7c86bf3
Bump google.golang.org/api from 0.212.0 to 0.214.0 in /sdks (#33429)
dependabot[bot] Dec 20, 2024
7e077dc
[yaml] add RunInference support with VertexAI (#33406)
Polber Dec 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
77 changes: 77 additions & 0 deletions .asf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,89 @@ github:
- pcoet
- olehborysevych
- rshamunov
- andreydevyatkin
- liferoad

enabled_merge_buttons:
squash: true
merge: true
rebase: false

protected_branches:
master: {}
release-2.62.0: {}
release-2.61.0: {}
release-2.60.0: {}
release-2.59.0: {}
release-2.58.1: {}
release-2.58.0: {}
release-2.57.0: {}
release-2.56.0: {}
release-2.55.1: {}
release-2.55.0: {}
release-2.54.0: {}
release-2.53.0: {}
release-2.52.0: {}
release-2.51.0: {}
release-2.50.0: {}
release-2.49.0: {}
release-2.48.0: {}
release-2.47.0: {}
release-2.46.0: {}
release-2.45.0: {}
release-2.44.0: {}
release-2.43.0: {}
release-2.42.0: {}
release-2.41.0: {}
release-2.40.0: {}
release-2.39.0: {}
release-2.38.0: {}
release-2.37.0: {}
release-2.36.0: {}
release-2.35.0: {}
release-2.34.0: {}
release-2.33.0: {}
release-2.32.0: {}
release-2.31.0: {}
release-2.30.0: {}
release-2.29.0: {}
release-2.28.0: {}
release-2.27.0: {}
release-2.26.0: {}
release-2.25.0: {}
release-2.24.0: {}
release-2.23.0: {}
release-2.22.0: {}
release-2.21.0: {}
release-2.20.0: {}
release-2.19.0: {}
release-2.18.0: {}
release-2.17.0: {}
release-2.16.0: {}
release-2.15.0: {}
release-2.14.0: {}
release-2.13.0: {}
release-2.12.0: {}
release-2.11.0: {}
release-2.10.0: {}
release-2.8.0: {}
release-2.8.0: {}
release-2.7.0: {}
release-2.6.0: {}
release-2.5.0: {}
release-2.4.0: {}
release-2.3.0: {}
release-2.2.0: {}
release-2.1.1: {}
release-2.1.0: {}
release-0.6.0: {}
release-0.5.0: {}
release-0.4.0: {}
release-0.4.0-incubating: {}
release-0.3.0-incubating: {}
release-0.2.0-incubating: {}
release-0.1.0-incubating: {}

notifications:
commits: [email protected]
issues: [email protected]
Expand Down
4 changes: 4 additions & 0 deletions .github/ACTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,3 +98,7 @@ Phrases self-assign, close, or manage labels on an issue:
| `.add-labels` | Add comma separated labels to the issue (e.g. `add-labels l1, 'l2 with spaces'`) |
| `.remove-labels` | Remove comma separated labels to the issue (e.g. `remove-labels l1, 'l2 with spaces'`) |
| `.set-labels` | Sets comma separated labels to the issue and removes any other labels (e.g. `set-labels l1, 'l2 with spaces'`) |

## Security Model

For information on the Beam CI security model, see https://cwiki.apache.org/confluence/display/BEAM/CI+Security+Model
3 changes: 3 additions & 0 deletions .github/ISSUE_TEMPLATE/bug.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ body:
- "Priority: 2 (default / most bugs should be filed as P2)"
- "Priority: 1 (data loss / total loss of function)"
- "Priority: 0 (outage / urgent vulnerability)"
default: 1
validations:
required: true
- type: checkboxes
Expand All @@ -63,10 +64,12 @@ body:
- label: "Component: Go SDK"
- label: "Component: Typescript SDK"
- label: "Component: IO connector"
- label: "Component: Beam YAML"
- label: "Component: Beam examples"
- label: "Component: Beam playground"
- label: "Component: Beam katas"
- label: "Component: Website"
- label: "Component: Infrastructure"
- label: "Component: Spark Runner"
- label: "Component: Flink Runner"
- label: "Component: Samza Runner"
Expand Down
3 changes: 3 additions & 0 deletions .github/ISSUE_TEMPLATE/failing_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ body:
- "Priority: 2 (backlog / disabled test but we think the product is healthy)"
- "Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)"
- "Priority: 0 (outage / failing precommit test impacting development)"
default: 1
validations:
required: true
- type: checkboxes
Expand All @@ -69,10 +70,12 @@ body:
- label: "Component: Go SDK"
- label: "Component: Typescript SDK"
- label: "Component: IO connector"
- label: "Component: Beam YAML"
- label: "Component: Beam examples"
- label: "Component: Beam playground"
- label: "Component: Beam katas"
- label: "Component: Website"
- label: "Component: Infrastructure"
- label: "Component: Spark Runner"
- label: "Component: Flink Runner"
- label: "Component: Samza Runner"
Expand Down
3 changes: 3 additions & 0 deletions .github/ISSUE_TEMPLATE/feature.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ body:
options:
- "Priority: 3 (nice-to-have improvement)"
- "Priority: 2 (default / most feature requests should be filed as P2)"
default: 1
validations:
required: true
- type: checkboxes
Expand All @@ -57,10 +58,12 @@ body:
- label: "Component: Go SDK"
- label: "Component: Typescript SDK"
- label: "Component: IO connector"
- label: "Component: Beam YAML"
- label: "Component: Beam examples"
- label: "Component: Beam playground"
- label: "Component: Beam katas"
- label: "Component: Website"
- label: "Component: Infrastructure"
- label: "Component: Spark Runner"
- label: "Component: Flink Runner"
- label: "Component: Samza Runner"
Expand Down
3 changes: 3 additions & 0 deletions .github/ISSUE_TEMPLATE/task.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ body:
- "Priority: 3 (nice-to-have improvement)"
- "Priority: 2 (default / most normal work should be filed as P2)"
- "Priority: 1 (urgent / mostly reserved for critical bugs)"
default: 1
validations:
required: true
- type: checkboxes
Expand All @@ -58,10 +59,12 @@ body:
- label: "Component: Go SDK"
- label: "Component: Typescript SDK"
- label: "Component: IO connector"
- label: "Component: Beam YAML"
- label: "Component: Beam examples"
- label: "Component: Beam playground"
- label: "Component: Beam katas"
- label: "Component: Website"
- label: "Component: Infrastructure"
- label: "Component: Spark Runner"
- label: "Component: Flink Runner"
- label: "Component: Samza Runner"
Expand Down
4 changes: 2 additions & 2 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Thank you for your contribution! Follow this checklist to help us incorporate yo
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).

To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)

Expand All @@ -19,4 +19,4 @@ GitHub Actions Tests Status (on master branch)
[![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)

See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
54 changes: 45 additions & 9 deletions .github/REVIEWERS.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,39 +20,75 @@
labels:
- name: Go
reviewers:
- damccorm
- lostluck
- jrmccluskey
- riteshghorse
exclusionList:
- youngoli
- name: Python
reviewers:
- AnandInguva
- damccorm
- jrmccluskey
- tvalentyn
- pabloem
- liferoad
- shunping
exclusionList: []
- name: Java
reviewers:
- lukecwik
- Abacn
- kennknowles
- robertwb
- kileys
- apilloud
- m-trieu
- damondouglas
exclusionList: []
- name: IO
reviewers:
- chamikaramj
- johnjcasey
- pabloem
- Abacn
- ahmedabu98
- damondouglas
- shunping
exclusionList: []
- name: spanner
reviewers:
- nielm
- name: bigtable
reviewers:
- igorbernstein2
- mutianf
exclusionList: []
- name: healthcare
reviewers:
- svetakvsundhar
exclusionList: []
- name: kafka
reviewers:
- johnjcasey
- fozzie15
- Dippatel98
- sjvanrossum
- name: Build
reviewers:
- damccorm
- Abacn
exclusionList: []
fallbackReviewers: []
- name: website
reviewers:
- rszper
- melap
- damccorm
- liferoad
- kennknowles
exclusionList: []
fallbackReviewers:
- Abacn
- chamikaramj
- damccorm
- damondouglas
- johnjcasey
- jrmccluskey
- kennknowles
- lostluck
- robertwb
- shunping
- tvalentyn
10 changes: 5 additions & 5 deletions .github/actions/common-rc-validation/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,19 +36,19 @@ runs:
shell: bash
run: |
echo "---------------------Downloading Python Staging RC----------------------------"
wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE_VER}/python/apache-beam-${RELEASE_VER}.zip
wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE_VER}/python/apache-beam-${RELEASE_VER}.zip.sha512
if [[ ! -f apache-beam-$RELEASE_VER.zip ]]; then
wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE_VER}/python/apache-beam-${RELEASE_VER}.tar.gz
wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE_VER}/python/apache-beam-${RELEASE_VER}.tar.gz.sha512
if [[ ! -f apache-beam-$RELEASE_VER.tar.gz ]]; then
{ echo "Fail to download Python Staging RC files." ;exit 1; }
fi

echo "--------------------------Verifying Hashes------------------------------------"
sha512sum -c apache-beam-${RELEASE_VER}.zip.sha512
sha512sum -c apache-beam-${RELEASE_VER}.tar.gz.sha512

`which pip` install --upgrade pip
`which pip` install --upgrade setuptools

- name: Installing python SDK
shell: bash
run: pip install apache-beam-${RELEASE_VER}.zip[gcp]
run: pip install apache-beam-${RELEASE_VER}.tar.gz[gcp]

1 change: 0 additions & 1 deletion .github/actions/gradle-command-action
Submodule gradle-command-action deleted from 90ccf0
21 changes: 9 additions & 12 deletions .github/actions/gradle-command-self-hosted-action/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,6 @@ inputs:
required: false
description: 'Gradle options'
default: ''
default-arguments:
required: false
description: 'Default gradle switches' # Copied from CommonJobProperties.groovy'
default: |
--continue -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx6g \
-Dorg.gradle.vfs.watch=false -Pdocker-pull-licenses
max-workers:
required: false
description: 'Max number of workers'
Expand All @@ -41,9 +35,12 @@ runs:
- name: Run Gradle Command
shell: bash
run: |
# Removing settings.xml is a workaround to avoid a decryption issue
# of Beam's gradle-command-action plugin and github's provided
# maven settings.xml file
rm ~/.m2/settings.xml
./gradlew ${{ inputs.gradle-command }} --max-workers=${{ inputs.max-workers }} ${{ inputs.arguments }} \
${{ inputs.default-arguments }}
# This step is a workaround to avoid a decryption issue of Beam's
# net.linguica.gradle.maven.settings plugin and github's provided maven
# settings.xml file
if [ -f ~/.m2/settings.xml ]; then
rm ~/.m2/settings.xml
fi
./gradlew ${{ inputs.gradle-command }} --max-workers=${{ inputs.max-workers }} --continue \
-Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.vfs.watch=false -Pdocker-pull-licenses \
${{ inputs.arguments }}
Loading
Loading