Announcing: The Pulsar PMC Published The 2020 Apache Pulsar User Survey Report!

Overview
Get started
Install and upgrade
Configure
Secure
Manage and monitor
Connect
Process
Overview
Pulsar Flink Connector
Overview
Link
Read data from Pulsar (source)
Write data to Pulsar (sink)
Configure
Secure
Use Pulsar catalog
Tutorial
Pulsar Spark Connector
Pulsar Functions
Release notes

Configure Pulsar Flink connector

You can set the following configurations for the Pulsar Flink connector (both source and sink).

Pulsar source configuration

For Pulsar source, you can set the following configurations for both batch and streaming queries.

Configuration Value Required / Optional Default Description
topic A topic name string Required N/A The topic to be consumed.

Only one of topic, topics or topicsPattern options can be specified for Pulsar source.
topics A comma-separated list of topics Required N/A The topic list to be consumed.

Only one of topic, topics or topicsPattern options can be specified for Pulsar source.
topicsPattern A Java regex string Required N/A The pattern used to subscribe to topic(s).

Only one of topic, topics or topicsPattern options can be specified for Pulsar source.
partitionDiscoveryIntervalMillis A discovery interval millis long as string Optional -1 Partition discovery is used for long running Flink job listen to partitions add during execution.

If a positive number is specified, Pulsar Flink connector checks periodically for newly added partitions every interval time.

If set to -1, the partition auto discovery is shutdown.
pulsar.reader.* Corresponding setting values Optional N/A Topic data in Pulsar are actually read out using Pulsar Reader API.

Therefore, you can set reader related settings by specifying pulsar.reader.*.

Pulsar sink configuration

For Pulsar sink, you can set the following configurations for both batch and streaming queries.

Configuration Value Required / Optional Default Description
flushOnCheckpoint "true"

"false"
Optional "true" Each time Flink checkpoint is called for the connector, all messages write through the sink are flushed to Pulsar directly.
failOnWrite "true"

"false"
Optional "false" Once a record failed to write to Pulsar, whether the connector fails directly or continues processing and just log the problem encountered.
pulsar.producer.* Corresponding setting values Optional N/A Topic data in Pulsar are actually written out using Pulsar Producer API.

Therefore, you can set reader related settings by specifying pulsar.producer.*.