Duet AI Transform service SDK Harness prompts #30499
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
March 5, 2024 23:55
dariabezkorovaina
suggested changes
Mar 7, 2024
There was a problem hiding this comment.
@olehborysevych looks good, I improved the wording and fixed minor issues with formatting.
learning/prompts/documentation-lookup-nolinks/48_sdk_harness.md
Outdated
Show resolved
Hide resolved
learning/prompts/documentation-lookup-nolinks/48_sdk_harness.md
Outdated
Show resolved
Hide resolved
learning/prompts/documentation-lookup-nolinks/48_sdk_harness.md
Outdated
Show resolved
Hide resolved
learning/prompts/documentation-lookup-nolinks/48_sdk_harness.md
Outdated
Show resolved
Hide resolved
learning/prompts/documentation-lookup-nolinks/48_sdk_harness.md
Outdated
Show resolved
Hide resolved
Comment on lines
17
to
18
| Currently the following transforms are supported: | ||
|
|
There was a problem hiding this comment.
Suggested change
| Currently the following transforms are supported: | |
| Currently, the Transform service can upgrade the following transforms: |
Comment on lines
19
to
23
| * BigQuery read transform: `beam:transform:org.apache.beam:bigquery_read:v1` | ||
| * BigQuery write transform: `beam:transform:org.apache.beam:bigquery_write:v1` | ||
| * Kafka read transform: `beam:transform:org.apache.beam:kafka_read_with_metadata:v2` | ||
| * Kafka write transform: `beam:transform:org.apache.beam:kafka_write:v2` | ||
|
|
There was a problem hiding this comment.
Suggested change
| * BigQuery read transform: `beam:transform:org.apache.beam:bigquery_read:v1` | |
| * BigQuery write transform: `beam:transform:org.apache.beam:bigquery_write:v1` | |
| * Kafka read transform: `beam:transform:org.apache.beam:kafka_read_with_metadata:v2` | |
| * Kafka write transform: `beam:transform:org.apache.beam:kafka_write:v2` | |
| * BigQuery read: `beam:transform:org.apache.beam:bigquery_read:v1` | |
| * BigQuery write: `beam:transform:org.apache.beam:bigquery_write:v1` | |
| * Kafka read: `beam:transform:org.apache.beam:kafka_read_with_metadata:v2` | |
| * Kafka write: `beam:transform:org.apache.beam:kafka_write:v2` | |
| * Kafka read transform: `beam:transform:org.apache.beam:kafka_read_with_metadata:v2` | ||
| * Kafka write transform: `beam:transform:org.apache.beam:kafka_write:v2` | ||
|
|
||
| Transform service implement Beam expansion API. This means you can use the Transform service to construct and execute multi-language pipelines. For example, you can build a Python pipeline that uses Java `KafkaIO` transform and execute in without installing Java locally. |
There was a problem hiding this comment.
Suggested change
| Transform service implement Beam expansion API. This means you can use the Transform service to construct and execute multi-language pipelines. For example, you can build a Python pipeline that uses Java `KafkaIO` transform and execute in without installing Java locally. | |
| The Transform service implements the Beam expansion API, enabling multi-language pipelines to leverage it for expanding supported transforms. This feature allows you to create and run multi-language pipelines without additional language runtimes. For instance, you can build a Python pipeline that utilizes a Java `KafkaIO` transform without the need to install Java locally. |
|
|
||
| Transform service implement Beam expansion API. This means you can use the Transform service to construct and execute multi-language pipelines. For example, you can build a Python pipeline that uses Java `KafkaIO` transform and execute in without installing Java locally. | ||
|
|
||
| Transform service can be started automatically by Apache Beam SDK or manually by users: |
There was a problem hiding this comment.
Suggested change
| Transform service can be started automatically by Apache Beam SDK or manually by users: | |
| In some cases, Apache Beam SDKs can start the Transform service automatically, provided that Docker is available locally. You can also start the Transform service manually by running the following command: |
Comment on lines
32
to
57
| Beam transform service includes a number of transforms implemented in the Apache Beam Java and Python SDKs: | ||
|
|
||
| * Java transforms: Google Cloud I/O connectors, the Kafka I/O connector, and the JDBC I/O connector | ||
| * Python transforms: all portable transforms implemented within the Apache Beam Python SDK, such as RunInference and DataFrame transforms. | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
There was a problem hiding this comment.
Suggested change
| Beam transform service includes a number of transforms implemented in the Apache Beam Java and Python SDKs: | |
| * Java transforms: Google Cloud I/O connectors, the Kafka I/O connector, and the JDBC I/O connector | |
| * Python transforms: all portable transforms implemented within the Apache Beam Python SDK, such as RunInference and DataFrame transforms. | |
| The Beam Transform service includes several portable transforms implemented in the Apache Beam Java and Python SDKs: | |
| * Java transforms: Google Cloud I/O connectors, the Kafka I/O connector, and the JDBC I/O connector. | |
| * Python transforms: all portable transforms implemented within the Apache Beam Python SDK, such as RunInference and DataFrame transforms. |
Co-authored-by: Daria Bezkorovaina <[email protected]>
olehborysevych
changed the title
|
Hey @damccorm this one is also ready for review |
|
Assigning reviewers. If you would like to opt out of this review, comment R: @AnandInguva added as fallback since no labels match configuration Available commands:
The PR bot will only process comments in the main thread (not review comments). |
hjtran
pushed a commit
to hjtran/beam
that referenced
this pull request
Apr 4, 2024
* transform service prompt added * SDK harness added * fix whitespace * Apply suggestions from code review Co-authored-by: Daria Bezkorovaina <[email protected]> --------- Co-authored-by: Daria Bezkorovaina <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Transform service lookup prompts
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.