[Go SDK]: Create natsio.Write transform for writing to NATS #29184

johannaojeling · 2023-10-29T16:21:29Z

Addresses #29000 and implements a Write transform for writing to NATS.

The Read transform is more involved and will be in a separate PR.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

codecov · 2023-10-29T16:30:54Z

Codecov Report

Merging #29184 (78c85ad) into master (c2816c8) will increase coverage by 0.03%.
Report is 92 commits behind head on master.
The diff coverage is 68.05%.

@@            Coverage Diff             @@
##           master   #29184      +/-   ##
==========================================
+ Coverage   38.38%   38.41%   +0.03%     
==========================================
  Files         687      691       +4     
  Lines      101745   101876     +131     
==========================================
+ Hits        39051    39137      +86     
- Misses      61113    61151      +38     
- Partials     1581     1588       +7

Flag	Coverage Δ
go	`53.50% <68.05%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
sdks/go/pkg/beam/io/natsio/write_option.go	`0.00% <0.00%> (ø)`
sdks/go/pkg/beam/io/natsio/write.go	`86.95% <86.95%> (ø)`
sdks/go/pkg/beam/io/natsio/common.go	`40.90% <40.90%> (ø)`

... and 12 files with indirect coverage changes

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

github-actions · 2023-10-29T17:05:47Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @riteshghorse for label go.
R: @ahmedabu98 for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

riteshghorse

Go changes looks fine to me and the rest as well. I noticed the different package name for example_test.go but I don't have strong opinion on that. In fact that makes sense to separate out the test functions.

I would let @ahmedabu98 do a pass for IO side and should be good to merge then!

johannaojeling · 2023-10-31T07:19:35Z

Go changes looks fine to me and the rest as well. I noticed the different package name for example_test.go but I don't have strong opinion on that. In fact that makes sense to separate out the test functions.

Thanks for reviewing! Regarding the <package>_test name, it is the convention for writing testable examples to show up as code examples in the go doc (the user should be guided to use natsio.Write rather than Write)

riteshghorse · 2023-10-31T12:52:33Z

Go changes looks fine to me and the rest as well. I noticed the different package name for example_test.go but I don't have strong opinion on that. In fact that makes sense to separate out the test functions.

Thanks for reviewing! Regarding the <package>_test name, it is the convention for writing testable examples to show up as code examples in the go doc (the user should be guided to use natsio.Write rather than Write)

that's great! thanks for clarifying!

ahmedabu98 · 2023-11-01T15:48:36Z

@lostluck I'm not really familiar with Go, do you mind taking a look for the IO parts of this PR?

lostluck

LGTM for Go (although Ritesh already did that).

Just the one IO style thing. @ahmedabu98 can confirm if that remains an IO best practice though.

lostluck · 2023-11-01T21:53:11Z

sdks/go/pkg/beam/io/natsio/write.go

+		opt(option)
+	}
+
+	beam.ParDo0(s, newWriteFn(uri, option), col)


Style-wise, even for sinks/writes, it's best to return a PCollection anyway and emit something when the write has been persisted. This allows composition of writing transforms in a pipeline.

Eg. One could want to do something in the same pipeline after the writes have been made. Having an output allows something downstream to use a side input to block until that Write is finished for a given window.

That's right, we strongly recommend against using PDone in Java for example. In general, we're leaning towards having transforms output something.

Thanks, updated

johannaojeling · 2023-11-03T16:56:44Z

sdks/go/pkg/beam/io/natsio/write.go

+type ProducerMessage struct {
+	Subject string
+	ID      string
+	Headers nats.Header


This nats.Header is a map[string][]string type which comes with some additional utility methods, similar to http.Header. It just hit me it might be preferable to change

- Headers nats.Header + Headers map[string][]string

to have the user only deal with Beam code and not leak out that of third-party libraries used in the implementation. I recall reading that in the I/O standards.

Thoughts on this?

Decided to follow the standards

It looks like this change wasn't pushed? We still have nats.Header there.

It was pushed in 78c85ad so should be visible if refreshing the files

lostluck

LGTM. Thank you kindly. Nats is neat.

Create natsio.Write transform for writing to NATS

2bceea4

github-actions bot added go io labels Oct 29, 2023

johannaojeling changed the title ~~Create natsio.Write transform for writing to NATS~~ [Go SDK]: Create natsio.Write transform for writing to NATS Oct 29, 2023

github-actions bot added the Next Action: Reviewers label Oct 29, 2023

riteshghorse approved these changes Oct 30, 2023

View reviewed changes

lostluck approved these changes Nov 1, 2023

View reviewed changes

Emit representation of acknowledged message from writeFn

b6d277f

johannaojeling commented Nov 3, 2023

View reviewed changes

Use type map[string][]string for ProduceMessage headers

78c85ad

lostluck approved these changes Nov 6, 2023

View reviewed changes

lostluck merged commit cf560ca into apache:master Nov 6, 2023
10 checks passed

johannaojeling deleted the feat/natsio-write branch November 7, 2023 17:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Go SDK]: Create natsio.Write transform for writing to NATS #29184

[Go SDK]: Create natsio.Write transform for writing to NATS #29184

johannaojeling commented Oct 29, 2023

codecov bot commented Oct 29, 2023 •

edited

Loading

github-actions bot commented Oct 29, 2023

riteshghorse left a comment

johannaojeling commented Oct 31, 2023

riteshghorse commented Oct 31, 2023

ahmedabu98 commented Nov 1, 2023

lostluck left a comment

lostluck Nov 1, 2023

ahmedabu98 Nov 3, 2023

johannaojeling Nov 3, 2023

johannaojeling Nov 3, 2023

johannaojeling Nov 3, 2023

lostluck Nov 3, 2023

johannaojeling Nov 3, 2023

lostluck left a comment

[Go SDK]: Create natsio.Write transform for writing to NATS #29184

[Go SDK]: Create natsio.Write transform for writing to NATS #29184

Conversation

johannaojeling commented Oct 29, 2023

GitHub Actions Tests Status (on master branch)

codecov bot commented Oct 29, 2023 • edited Loading

Codecov Report

github-actions bot commented Oct 29, 2023

riteshghorse left a comment

Choose a reason for hiding this comment

johannaojeling commented Oct 31, 2023

riteshghorse commented Oct 31, 2023

ahmedabu98 commented Nov 1, 2023

lostluck left a comment

Choose a reason for hiding this comment

lostluck Nov 1, 2023

Choose a reason for hiding this comment

ahmedabu98 Nov 3, 2023

Choose a reason for hiding this comment

johannaojeling Nov 3, 2023

Choose a reason for hiding this comment

johannaojeling Nov 3, 2023

Choose a reason for hiding this comment

johannaojeling Nov 3, 2023

Choose a reason for hiding this comment

lostluck Nov 3, 2023

Choose a reason for hiding this comment

johannaojeling Nov 3, 2023

Choose a reason for hiding this comment

lostluck left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 29, 2023 •

edited

Loading