Announcing: The Pulsar PMC Published The 2020 Apache Pulsar User Survey Report!

Overview
Get started
Install and upgrade
Configure
Secure
Manage and monitor
Connect
Process
Overview
Pulsar Flink Connector
Overview
Link
Read data from Pulsar (source)
Write data to Pulsar (sink)
Configure
Secure
Use Pulsar catalog
Tutorial
Pulsar Spark Connector
Pulsar Functions
Release notes

Tutorial

This tutorial guides you through every step of using Pulsar Flink Connector to write data to Pulsar and read from Pulsar.

How to use Pulsar Flink connector to write data to Pulsar and read from Pulsar?

This example contains two flink streaming jobs, StreamWrite to populate data into Pulsar and StreamRead to read data out in a streaming fashion.

  1. Start StreamNative Platform using Supervisord.

    ${PLATFORM_HOME}/bin/streamnative-standalone-start.sh
  2. Start Flink locally.

    You can follow the instructions to download and start Flink.

    ${FLINK_HOME}/bin/start-cluster.sh
  3. Download and build the example.

    The example project is available at here. After downloading, you could unzip it and build the example as a jar.

    cd flink-connector-test-1.0
    mvn clean package -DskipTests
  4. Run StreamWrite to produce NASA data to Pulsar topic TOPIC_NAME.

    ${FLINK_HOME}/bin/flink run -c com.example.StreamWrite ${Example_project}/target/flink-connector-test-1.0-SNAPSHOT.jar [TOPIC_NAME]
  5. Run StreamRead to read NASA data out from TOPIC_NAME.

    ${FLINK_HOME}/bin/flink run -c com.exmple.StreamRead ${Example_project}/target/flink-connector-test-1.0-SNAPSHOT.jar [TOPIC_NAME]

    You will see sample output for above application as follows:

    1,Mercury program,1959
    2,Apollo program,1961
    3,Gemini program,1963
    4,Skylab,1973
    5,Apollo–Soyuz Test Project,1975