Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Task]: Add utilities to easily implement portable dynamic destinations #32365

Open
2 of 17 tasks
ahmedabu98 opened this issue Aug 29, 2024 · 0 comments
Open
2 of 17 tasks

Comments

@ahmedabu98
Copy link
Contributor

ahmedabu98 commented Aug 29, 2024

What needs to happen?

Dynamic destinations is a feature that allows users to write to multiple destinations using one IO instance. This is usually done via some sort of function that inspects the record and determines its destination. Currently, this feature is only supported with native use-cases (e.g. Python can use callables; Java can use SerializableFunctions or DynamicDestinations).

Of course, we can’t simply pass a whole function from one language to another, but we can extract the important bits and articulate them in a language-agnostic way

After discussion in the mailing list and design document, the agreed upon approach (proposal (1) in the doc) looks something like this:

image

PR #31807 is a proof of concept that includes the necessary utilities and a real implementation for portable dynamic destinations in IcebergIO. This PR will be broken down into multiple PRs to decouple concepts and make it easier to review.

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant