Code

Integrate direct path#31902

Code

Closed

m-trieu wants to merge 26 commits intoapache:masterapache/beam:masterfrom m-trieu:mt-dp-streamsm-trieu/beam:mt-dp-streamsCopy head branch name to clipboard

+2,233-1,217

Commits

Commits on Oct 10, 2024

• add getdata clients and shareable logic for GetData w/ throttling and metric tracking.

  m-trieu

  committed
  15675d0

• add HeartbeatSender interface

  m-trieu

  committed
  8c1ca87

• address PR comments

  m-trieu

  committed
  9f42751

• move side input state API into Work instance w/ keyed state fetching. Future proofs this logic for direct path.

  m-trieu

  committed
  e623fc7

• don't let FanOutWorkRefreshClient exceptions crash the user worker

  m-trieu

  committed
  0735a0c

• address PR comments

  m-trieu

  committed
  998c07a

• address PR comments

  m-trieu

  committed
  659d5aa

• add WorkProvider interface and have StreamingEngine and Appliance impls

  m-trieu

  committed
  554b73e

• change StreamingEngineClient to FanOutStreamingEngineWorkProvider and have it implement WorkProvider interface. Move class to windmill/work/provider directory, update visibility for dependent class…

  Show description for a7a9efa
  m-trieu

  committed
  a7a9efa

• add WorkProvider interface and have StreamingEngine and Appliance impls

  m-trieu

  committed
  52c1fec

• change StreamingEngineClient to FanOutStreamingEngineWorkProvider and have it implement WorkProvider interface. Move class to windmill/work/provider directory, update visibility for dependent class…

  Show description for 83655fa
  m-trieu

  committed
  83655fa

• add WorkProvider interface and have StreamingEngine and Appliance impls

  m-trieu

  committed
  e27fde2

• change StreamingEngineClient to FanOutStreamingEngineWorkProvider and have it implement WorkProvider interface. Move class to windmill/work/provider directory, update visibility for dependent class…

  Show description for 9bf73c4
  m-trieu

  committed
  9bf73c4

• add stream management and correct stream shutdown mechanisms for direct path

  m-trieu

  committed
  9beeec9

• integrate direct path components

  m-trieu

  committed
  0d60a18

• address PR comments

  m-trieu

  committed
  361ec3c

• simplify budgeting logic for GrpcDirectGetWorkStream

  m-trieu

  committed
  fd6549e

• fix stuckness on range movements

  m-trieu

  committed
  36015f8

• address PR comments, fix broken tests

  m-trieu

  committed
  14025db

• address PR comments

  m-trieu

  committed
  2144cd9

• remove thread polling for new metadata; propogate metadata version to WindmillEndpoints and don't process any version that is older than the current version in FanOutStreamingEngineWorkerHarness

  m-trieu

  committed
  92414f7

• deflake FanOutStreamingEngineWorkerHarnessTest.testOnNewWorkerMetadata_redistributesBudget()

  m-trieu

  committed
  b09ba8f

• address PR comments

  m-trieu

  committed
  3db3b7f

• address PR comments

  m-trieu

  committed
  06836da

• remove MAX_WAIT_SECONDS and just use the deadline passed in, its already 600 seconds anyway. Move DEFAULT_STREAM_RPC_DEADLINE_SECONDS to where it is being used and remove references in tests

  m-trieu

  committed
  6df1adf

Commits on Oct 11, 2024

• address PR comments, remove unused IPv6 WindmillServiceAddress, rename StreamingEngineConnectionsState to StreamingEngineBackends

  m-trieu

  committed
  4f5b381