Announcing: The Pulsar PMC Published The 2020 Apache Pulsar User Survey Report!

Overview
Get started
Overview
Local
Docker
Helm
Install and upgrade
Configure
Secure
Manage and monitor
Connect
Process
Release notes

Quick start(local)

This section guides you through every step of installing and running StreamNative Platform on your local machine quickly, including downloading, installing, and starting StreamNative Platform, creating topics using pulsarctl, generating sample data with Pulsar clients, publishing messages from Kafka to Pulsar using KoP, monitoring cluster status with Prometheus and Grafana, handling alerts using Alertmanager, stopping StreamNative Platform, and so on.

Prerequisite

Before deploying the StreamNative Platform, make sure the following items are installed on your machine.

  • JDK 1.8+

  • Nginx

  • j2cli

  • Supervisord

Tip

For the following steps, step 2 to step 5 are for developers and step 6 to step 8 are for administrators.

Step 1: Download and start StreamNative Platform

  1. Go to the downloads page and choose .tar.gz or .zip based on your needs.

  2. Decompress the file and you can see the following directories.

    ├── README
    ├── bin // Store executable files provided by StreamNative Platform.
    ├── etc // Store related configuration file information.
    ├── lib // Provide systemd services. When the service is down, it restarts automatically. 
    └── share // Store dependent files.
  3. Set the environment variable for the StreamNative Platform directory.

    $ export PLATFORM_HOME=<path-to-streamnative>
  4. Add the StreamNative Platform bin directory to your path.

    $ export PATH=${PLATFORM_HOME}/bin:$PATH
  5. Start StreamNative Platform using Supervisord.

    ${PLATFORM_HOME}/bin/streamnative-standalone-start.sh

    Output

    Initialized the pulsar data source.
    2020-03-03 15:48:29,709 INFO RPC interface 'supervisor' initialized
    2020-03-03 15:48:29,709 CRIT Server 'inet_http_server' running without any HTTP authentication checking
    2020-03-03 15:48:29,709 INFO supervisord started with pid 72286
    2020-03-03 15:48:30,715 INFO spawned: 'pulsar' with pid 72288
    2020-03-03 15:48:30,717 INFO spawned: 'pulsar-manager' with pid 72289
    2020-03-03 15:48:30,720 INFO spawned: 'sn-alertmanager' with pid 72290
    2020-03-03 15:48:30,723 INFO spawned: 'sn-grafana' with pid 72291
    2020-03-03 15:48:30,726 INFO spawned: 'sn-node-exporter' with pid 72292
    2020-03-03 15:48:30,729 INFO spawned: 'sn-prometheus' with pid 72294
    2020-03-03 15:48:33,722 INFO success: sn-alertmanager entered RUNNING state, process has stayed up for > than 3 seconds (startsecs)
    2020-03-03 15:48:33,722 INFO success: sn-grafana entered RUNNING state, process has stayed up for > than 3 seconds (startsecs)
    2020-03-03 15:48:33,768 INFO success: sn-node-exporter entered RUNNING state, process has stayed up for > than 3 seconds (startsecs)
    2020-03-03 15:48:33,768 INFO success: sn-prometheus entered RUNNING state, process has stayed up for > than 3 seconds (startsecs)
    2020-03-03 15:48:35,712 INFO success: pulsar entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
    2020-03-03 15:48:35,716 INFO success: pulsar-manager entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)

    We can use the supervisorctl status command to check the status.

    Output

    pulsar                           RUNNING   pid 72288, uptime 0:00:08
    pulsar-manager                   RUNNING   pid 72289, uptime 0:00:08
    sn-alertmanager                  RUNNING   pid 72290, uptime 0:00:08
    sn-grafana                       RUNNING   pid 72291, uptime 0:00:08
    sn-node-exporter                 RUNNING   pid 72292, uptime 0:00:08
    sn-prometheus                    RUNNING   pid 72294, uptime 0:00:08

Step 2: Use pulsarctl to create Pulsar tenants/namespaces/topics

pulsarctl is a CLI tool for Pulsar. In this step, you can use pulsarctl to create resources including tenants, namespaces, and topics.

  1. Create a tenant named example-tenant.

    $ ${PLATFORM_HOME}/bin/pulsarctl tenants create example-tenant -c standalone

    Then you can see that the tenant example-tenant is created successfully.

    Output

    Create tenant example-tenant successfully
  2. Create a namespace named example-ns in the tenant example-tenant.

    $ ${PLATFORM_HOME}/bin/pulsarctl namespaces create example-tenant/example-ns -c standalone

    Then you can see that the namespace example-ns is created successfully.

    Output

    Created example-tenant/example-ns successfully
  3. Create a topic test-topic with 1 partition in the namespace example-ns.

    $ ${PLATFORM_HOME}/bin/pulsarctl topics create example-tenant/example-ns/test-topic 1

    Then you can see that the topic test-topic is created successfully.

    Output

    Create topic persistent://example-tenant/example-ns/test-topic with 1 partitions successfully
  4. List all the topics under the namespace example-ns.

    $ ${PLATFORM_HOME}/bin/pulsarctl topics list example-tenant/example-ns

    Then you can see all topics under the namespace example-ns.

    Output

    +---------------------------------------------------------------+---------------+
    |                          TOPIC NAME                           | PARTITIONED ? |
    +---------------------------------------------------------------+---------------+
    | persistent://example-tenant/example-ns/test-topic             | Y             |
    | persistent://example-tenant/example-ns/test-topic-partition-0 | N             |
    +---------------------------------------------------------------+---------------+

Step 3: Use Pulsar client to produce and consume events

StreamNative Platform supports all the official Pulsar clients. You can use the Pulsar client to create producers and consumers to simulate a simple production and consumption model.

  1. Create a subscription to consume messages from the topic test-topic.

    $ ${PLATFORM_HOME}/bin/pulsar-client consume -t Failover -s subname test-topic  -n 0
  2. Create a producer and send messages to the topic test-topic.

    $ ${PLATFORM_HOME}/bin/pulsar-client produce test-topic  -m "---------hello streamnative platform-------" -n 10
  3. Send 10 messages to the topic test-topic.

    The message is ---------hello streamnative platform-------.

    Output

    • From the producer side.

      23:04:25.433 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x7660f1ad, L:/127.0.0.1:56942 - R:/127.0.0.1:6650]] Connected to server
      23:04:25.534 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {
      "topicName" : "test-topic",
      "producerName" : null,
      "sendTimeoutMs" : 30000,
      "blockIfQueueFull" : false,
      "maxPendingMessages" : 1000,
      "maxPendingMessagesAcrossPartitions" : 50000,
      "messageRoutingMode" : "RoundRobinPartition",
      "hashingScheme" : "JavaStringHash",
      "cryptoFailureAction" : "FAIL",
      "batchingMaxPublishDelayMicros" : 1000,
      "batchingPartitionSwitchFrequencyByPublishDelay" : 10,
      "batchingMaxMessages" : 1000,
      "batchingMaxBytes" : 131072,
      "batchingEnabled" : true,
      "compressionType" : "NONE",
      "initialSequenceId" : null,
      "autoUpdatePartitions" : true,
      "multiSchema" : true,
      "properties" : { }
      }
      23:04:25.536 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {
      "serviceUrl" : "pulsar://127.0.0.1:6650/",
      "authPluginClassName" : null,
      "authParams" : null,
      "operationTimeoutMs" : 30000,
      "statsIntervalSeconds" : 60,
      "numIoThreads" : 1,
      "numListenerThreads" : 1,
      "connectionsPerBroker" : 1,
      "useTcpNoDelay" : true,
      "useTls" : false,
      "tlsTrustCertsFilePath" : "",
      "tlsAllowInsecureConnection" : false,
      "tlsHostnameVerificationEnable" : false,
      "concurrentLookupRequest" : 5000,
      "maxLookupRequest" : 50000,
      "maxNumberOfRejectedRequestPerConnection" : 50,
      "keepAliveIntervalSeconds" : 30,
      "connectionTimeoutMs" : 10000,
      "requestTimeoutMs" : 60000,
      "initialBackoffIntervalNanos" : 100000000,
      "maxBackoffIntervalNanos" : 60000000000
      }
      23:04:25.544 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test-topic-partition-0] [null] Creating producer on cnx [id: 0x7660f1ad, L:/127.0.0.1:56942 - R:/127.0.0.1:6650]
      23:04:25.557 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test-topic-partition-0] [standalone-0-8] Created producer on cnx [id: 0x7660f1ad, L:/127.0.0.1:56942 - R:/127.0.0.1:6650]
      23:04:25.565 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.PartitionedProducerImpl - [test-topic] Created partitioned producer
      23:04:25.580 [main] WARN  com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
      23:04:25.623 [main] INFO  org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://127.0.0.1:6650/
      23:04:25.637 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test-topic-partition-0] [standalone-0-8] Closed Producer
      23:04:25.638 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.PartitionedProducerImpl - [test-topic] Closed Partitioned Producer
      23:04:25.642 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0x7660f1ad, L:/127.0.0.1:56942 ! R:/127.0.0.1:6650] Disconnected
      23:04:25.652 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 10 messages successfully produced
    • From the consumer side.

      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------

Step 4: Use Kafka client to produce and consume events

KoP (Kafka on Pulsar) supports Kafka protocol and it is backed by Pulsar, which means you can use Pulsar as the infrastructure without modifying various applications and services based on Kafka API.

KoP, implemented as a Pulsar protocol handler plugin with protocol name "kafka", is loaded when Pulsar broker starts.

Currently, KoP supports Kafka Client 2.0.0.

  1. Download Kafka client 2.0.0 and untar it.

    $ tar -xzf kafka_2.11-2.0.0.tgz 
    $ cd kafka_2.11-2.0.0
  2. Run the command-line producer and send a few messages to the server.

    $ bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
  3. Kafka has a command-line consumer dumping out messages to standard output.

    $ bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test --from-beginning
  4. Publish messages to the Kafka producer.

    $ bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test-topic
    >hello
    >streamnative
    >platform
  5. The Kafka consumer receives messages.

    Output

    hello
    streamnative
    platform

Step 5: Verify interoperability between Pulsar and Kafka

As shown in the step 3 and step 4, Pulsar producer, Pulsar consumer, Kafka producer, and Kafka consumer run normally. Pulsar consumer can get the message from Pulsar producer and Kafka consumer can receive the message from Kafka producer. This section further verifies the interoperability between Pulsar and Kafka.

To verify interoperability between Pulsar and Kafka, follow these steps:

  1. Publish messages to Pulsar producer.

    $ ${PLATFORM_HOME}/bin/pulsar-client produce test-topic  -m "---------hello this is pulsar producer-------" -n 2
  2. Pulsar consumer and the Kafka consumer receive messages simultaneously.

    Output

    • Pulsar consumer

      ----- got message -----
      ---------hello this is pulsar producer-------
      ----- got message -----
      ---------hello this is pulsar producer-------
    • Kafka consumer

      ---------hello this is pulsar producer-------
      ---------hello this is pulsar producer-------
  3. Publish message to Kafka producer.

    $> hello this is kafka producer
  4. Pulsar consumer and Kafka consumers receive the message simultaneously.

    Output

    • Kafka comsumer

      ----- got message -----
      ---------hello this is kafka producer-------
    • Pulsar consumer

      ---------hello this is kafka producer-------

Step 6: Use Pulsar Manager to manage cluster

Pulsar Manager is a web-based GUI management tool for managing and monitoring Pulsar. Here, we use Pulsar Manager to create topics as an example to show how to manage clusters.

  1. Navigate to the Pulsar Manager website at http://localhost:9527/.

    Tip

    If you are deploying Pulsar Manager using the latest code, you can create a super-user using the following commands.

    $ curl -H "Content-Type: application/json" -X PUT > http://localhost:7750/pulsar-manager/users/superuser -d '{"name": "pulsar", "password": "pulsar", "description": "test", "email": "test@test.org"}'
  2. Log into the Pulsar Manager website using the default credentials.

    • Account: pulsar

    • Password: pulsar

  3. Create a new environment with an environment name and service URL, and click Confirm. In this example, we set the environment name and service URL to test-platform and http://127.0.0.1:8080 respectively for your reference.

  4. Click Topics and New Topic, select Persistent in Domain, name it with test-topic, and specify the partition number 1.

  5. Click Confirm and the topic is successfully created. The newly-created topic looks like the one shown as below.

    Tip

    You can use pulsarctl to double check that the topic is created successfully with Pulsar Manager.

    ${PLATFORM_HOME}/bin/pulsarctl topics list public/default

    Output

    +----------------------------------------------------+---------------+
    |                     TOPIC NAME                     | PARTITIONED ? |
    +----------------------------------------------------+---------------+
    | persistent://public/default/test-topic             | Y             |
    | persistent://public/default/test-topic-partition-0 | N             |
    +----------------------------------------------------+---------------+

Step 7: Use Prometheus and Grafana to monitor status

Prometheus

Prometheus is an open-source system monitoring and alerting toolkit, which is used to collect and store Pulsar related metrics.

Navigate to the Prometheus website at http://localhost:9090/targets, where you can see Prometheus shows metrics of all Pulsar components.

Grafana

Apache Pulsar Grafana dashboard is an open-source visualization tool, containing a unique Graphite target parser that enables easy metrics and function editing, which is used to visualize time series data of different monitoring indexes.

Navigate to the Grafana website at http://localhost:3000 and log into using the following credentials.

  • Account: pulsar

  • Password: pulsar

Grafana dashboard shows detailed metrics of components, such as bookie, JVM, messaging, node, proxy, Zookeeper, Pulsar topics, and so on.

General

Overview

JVM metrics

Messaging metrics

Pulsar topics

ZooKeeper

Step 8: Use Alertmanager to handle alerts

StreamNative Platform integrates the Alertmanager to handle alerts sent by its components, such as the Prometheus server.

Navigate to the Alertmanager website at http://localhost:9093 and log into the website without credentials.

If you want to customize your alert rules, edit the ${PLATFORM_HOME}/share/sn-alert-manager/alertmanager.yml file.

Step 9: Stop StreamNative Platform

Use supervisorctl to stop processes.

$ supervisorctl stop <name> // Stop a process

$ supervisorctl stop <name> <name> // Stop multiple processes or groups

$ supervisorctl stop all // Stop all processes

For more information about supervisord commands, see here.

Configure

If you run StreamNative Platform in standalone mode, (that is, run StreamNative Platform in local environment or on Docker) and want to change the configuration of StreamNative Platform, edit the ${PLATFORM_HOME}/etc/pulsar/standalone.conf file. The standalone configuration file is embedded with configurations of KoP, as shown below. For more information about this standalone configuration file, see here.

# For kop config
messagingProtocols=kafka
protocolHandlerDirectory=share/java/pulsar/potocols

If you run StreamNative Platform in cluster mode and want to change the configuration of StreamNative Platform, you can change the configuration of each component of Pulsar core. For more information, see here.

Troubleshoot

When you run into problems, try the following solutions.

Issue: operating system does not recognize PLATFORM_HOME environment variable

  • Issue

    Apache Pulsar standalone started
    Starting Grafana...
    Initialized the pulsar data source.
    Error: Format string '%(ENV_PLATFORM_HOME)s/share/supervisor/pulsar.log' for 'stdout_logfile' contains names which cannot be expanded
    For help, use /usr/bin/supervisord -h
  • Resolution

    Set the environment variable for the StreamNative Platform directory.

    $ export PLATFORM_HOME=<path-to-streamnative>