[Parent issue] Support for Apache Pulsar

[hpvd]

Since there is a continious demand for it,
just an overview for current state of Support for Apache Pulsar:

• when looking into beams i/o connector matrix pulsar is not listed yet:
  https://beam.apache.org/documentation/io/connectors/

• but there is a sdk package
  Package org.apache.beam.sdk.io.pulsar
  https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/pulsar/package-summary.html

• in v2.39 of beam a relatively basic pulsar io was introduced:
  Implemented Apache PulsarIO https://beam.apache.org/blog/beam-2.39.0/
  for details see:
  https://issues.apache.org/jira/browse/BEAM-8218

• there are some open issues in beam for pulsar:
  https://github.com/apache/beam/issues?q=is%3Aissue+pulsar+is%3Aopen

  • [Bug]: PulsarIO.read() is failing with java.lang.RuntimeException: Could not find a way to create AutoValue class class com.idfy.beam.pulsar.PulsarSourceDescriptor #30688
  • [Bug]: PulsarIO write() complains of missing ClientUrl even when provided with withClientUrl #30152
  • [Feature Request]: Set option for acknowledge the messages in PulsarIO reader #22054
  • [Feature Request]: Pulsar Connector should accept auth parameters to connect #22027
  • [Feature Request]: Make Pulsar IO connector publish messages asynchronously #22025
  • PulsarIOTest.testReadFromSimpleTopic is very flaky #21714
  • org.apache.beam.sdk.io.pulsar.PulsarIOTest.testReadFromSimpleTopic flaky #21560
  • [Doc Request]: current state of Pulsar support not in doc #31079

• there were also some non merged pull request marked as stale and automatically closed, which may be worth to have a look on
  https://github.com/apache/beam/pulls?q=is%3Apr+pulsar+is%3Aclosed+label%3Astale

  • Bump Jackson dependency due to CVE-2022-1471 #26410
  • fix pulsar reader calls #22375
  • pulsar connector accepts auth params #22028
  • pulsar connector produces message asynchronously #22026
  • [BEAM-14269] PulsarIOTest.testReadFromSimpleTopic flaky test #17473
  • Validate timestamps when creating windowed or timestamped values #16749

• in general it would be a good longterm target, to reach the same quality of integration in Beam as there is for Kafka

[hpvd]

regarding the beginning of pulsar integration
from https://www.youtube.com/watch?v=L_Pm2yKgA6g

[damccorm]

Hey @hpvd - have you been able to make any progress on this issue? I agree that the current state of the IO isn't great right now, so there's some good contribution opportunity :)