Announcing: The Pulsar PMC Published The 2020 Apache Pulsar User Survey Report!

Overview
Get started
Install and upgrade
Configure
Secure
Manage and monitor
Connect
Overview
Client
Pulsar connector
KoP
Overview
Get started
Limitation
Process
Release notes

Get started

In this guide, you will learn how to run KoP plugin in StreamNative Platform to serve requests from a Kafka client.

Prerequisite

Check the following requirements before using KoP.

This example assumes that you will use local machine to start StreamNative Platform.

Quick start

Configure and start StreamNative Platform

KoP is a plugin with protocol name "kafka" in Pulsar and it is loaded when Pulsar broker starts.

  1. Decompress the downloaded .tar.gz or .zip file, and you can see the following directories.

    ├── README
    ├── bin // Store executable files provided by StreamNative Platform.
    ├── etc // Store related configuration file information.
    ├── lib // Provide systemd services. When the service is down, it restarts automatically. 
    └── share // Store dependent files.
  2. Set the environment variable for the StreamNative Platform directory.

    $ export PLATFORM_HOME=<path-to-streamnative>
  3. Check the configuration of KoP.

    As mentioned previously, KoP module is loaded along with the Pulsar broker. You need to configure the Pulsar broker to run the KoP protocol handler as a plugin, that is, add the following properties in the Pulsar configuration file, in ${PLATFORM_HOME}/etc/pulsar/standalone.conf.

    Property Set it to the following value Default value
    messagingProtocols kafka null
    protocolHandlerDirectory Location of KoP NAR file ${PLATFORM_HOME}/share/java/pulsar/protocols

    Example

    messagingProtocols=kafka
    protocolHandlerDirectory=share/java/pulsar/protocols

    By default, these values are already set in StreamNative Platform, and kop.nar has already been placed under ${PLATFORM_HOME}/share/java/pulsar/protocols.

  4. Add the StreamNative Platform bin directory to your path.

    $ export PATH=${PLATFORM_HOME}/bin:$PATH
  5. Start StreamNative Platform using Supervisord.

    ${PLATFORM_HOME}/bin/streamnative-standalone-start.sh

    In the log of broker, after the StreamNative Platform is started successfully, you should find a log similar to this:

    12:22:22.945 [main] INFO  org.apache.pulsar.broker.protocol.ProtocolHandlers - Successfully loaded protocol handler for protocol `kafka`

Verify with Kafka client.

  1. Download the Kafka 2.0.0 release and untar it.

    tar -xzf kafka_2.11-2.0.0.tgz
    cd kafka_2.11-2.0.0
  2. Use a Kafka producer and a Kafka consumer to verify.

    In Kafka’s binary, there is a command-line producer and consumer. These steps verify Kafka producer send message to Pulsar broker, and Kafka consumer get message from Pulsar broker.

    Run the command-line producer and send a few messages to the server.

    > bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
    This is a message
    This is another message

    Kafka has a command-line consumer dumping out messages to standard output.

    > bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test --from-beginning
    This is a message
    This is another message

More information