Skip to content

Latest commit

 

History

History
82 lines (59 loc) · 3.01 KB

rocketmq.md

File metadata and controls

82 lines (59 loc) · 3.01 KB
Raw

RocketMQ connector

Parent document: connectors

BitSail RocketMQ connector supports writing in batch mode.

Maven dependency

<dependency>
   <groupId>com.bytedance.bitsail</groupId>
   <artifactId>bitsail-connector-rocketmq</artifactId>
   <version>${revision}</version>
</dependency>

RocketMQ writer

Supported data types

  • int, bigint
  • float, double, decimal
  • timestamp, date
  • string, char
  • boolean
  • binary

Parameters

The following mentioned parameters should be added to job.writer block when using, for example:

{
  "job": {
    "writer": {
      "class": "com.bytedance.bitsail.connector.legacy.rocketmq.sink.RocketMQOutputFormat",
      "name_server_address": "127.0.0.1:9876",
      "producer_group": "test_producer_group",
      "topic": "test_topic"
    }
  }
}

Necessary parameters

Param name Required Optional value Description
class Yes Class name of RocketMQ writer, com.bytedance.bitsail.connector.legacy.rocketmq.sink.RocketMQOutputFormat
name_server_address Yes Name server address of rocketmq
topic Yes Topic to write
columns Yes Describing fields' names and types

Optional parameters

Param name Required Optional value Description
writer_parallelism_num No Writer parallelism num
producer_group No Producer group for the task. If not defined, it will use a random string
tag No Tags of the produced messages
enable_batch_flush No true: send a batch of messages at once;
false: send single message at once.
Default true.
batch_size No The number of messages to send in a batch. Default 100
log_failures_only No When send failure happens, it will:
true: only log this failure,
false: throw exception.
Default false.
enable_sync_send No If use sync send. Default false.
access_key No Access key for authorization.
secret_key No Secret key for authorization.
send_failure_retry_times No Max retry times for a send failure, default 3
send_message_timeout_ms No Timeout for sending a message, default 3000 ms
max_message_size_bytes No Max message size, default 4194304 bytes
key No Specify which field(s) is used as the message key.
partition_fields No Specify which field(s) is used to select queue.

Related document

Configuration examples: rocketmq-connector-example