You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Dynamic destinations is a feature that allows users to write to multiple destinations using one IO instance. This is usually done via some sort of function that inspects the record and determines its destination. Currently, this feature is only supported with native use-cases (e.g. Python can use callables; Java can use SerializableFunctions or DynamicDestinations).
Of course, we can’t simply pass a whole function from one language to another, but we can extract the important bits and articulate them in a language-agnostic way
After discussion in the mailing list and design document, the agreed upon approach (proposal (1) in the doc) looks something like this:
PR #31807 is a proof of concept that includes the necessary utilities and a real implementation for portable dynamic destinations in IcebergIO. This PR will be broken down into multiple PRs to decouple concepts and make it easier to review.
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components
Component: Python SDK
Component: Java SDK
Component: Go SDK
Component: Typescript SDK
Component: IO connector
Component: Beam YAML
Component: Beam examples
Component: Beam playground
Component: Beam katas
Component: Website
Component: Infrastructure
Component: Spark Runner
Component: Flink Runner
Component: Samza Runner
Component: Twister2 Runner
Component: Hazelcast Jet Runner
Component: Google Cloud Dataflow Runner
The text was updated successfully, but these errors were encountered:
What needs to happen?
Dynamic destinations is a feature that allows users to write to multiple destinations using one IO instance. This is usually done via some sort of function that inspects the record and determines its destination. Currently, this feature is only supported with native use-cases (e.g. Python can use callables; Java can use SerializableFunctions or DynamicDestinations).
Of course, we can’t simply pass a whole function from one language to another, but we can extract the important bits and articulate them in a language-agnostic way
After discussion in the mailing list and design document, the agreed upon approach (proposal (1) in the doc) looks something like this:
PR #31807 is a proof of concept that includes the necessary utilities and a real implementation for portable dynamic destinations in IcebergIO. This PR will be broken down into multiple PRs to decouple concepts and make it easier to review.
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components
The text was updated successfully, but these errors were encountered: