10 Useful Pulsarctl Commands to Manage Your Cluster

Apache Pulsar is a cloud-native, distributed messaging and streaming platform. Originally created at Yahoo! and currently maintained by the Apache Software Foundation, Pulsar provides a scalable and durable messaging platform with a multitude of features like multi-tenancy, geo-replication, and persistent storage.

Pulsar comes with a command-line utility pulsar-admin that is used to perform administrative operations on Pulsar.

Pulsarctl by StreamNative is an alternative to the traditional pulsar-admin. Pulsarctl provides numerous improvements over pulsar-admin, including a unified interface for managing partitioned and non-partitioned topics; outputs in text, JSON, and YAML; detailed documentation; and many more.

This article will go over ten pulsarctl commands that are useful for Pulsar cluster administration. The commands here are chosen based on how frequently they are used and how vital they are in configuring, managing, and utilizing various aspects of Apache Pulsar.

1. clusters create

A Pulsar cluster is one of the highest level components of a Pulsar instance. A Pulsar cluster consists of one or more brokers, one or more BookKeeper servers, and a ZooKeeper server. A Pulsar instance can consist of one or more clusters. Since clusters sit at the heart of a Pulsar instance, it is crucial to be able to create new clusters, which can be achieved through the clusters create command. This command takes the following arguments:

1. The cluster name.
2. A broker service URL using the --broker-url flag.
3. A Pulsar cluster web service URL using the --url flag.

You can find other flags accepted by this command in the reference.

Usage:

pulsarctl clusters create --url http://localhost:8080 --broker-url pulsar://localhost:6650 my-cluster

2. topics create

A topic in Pulsar is a named channel used to transmit messages from producers to consumers. As this is a vital part of Pulsar, it’s important to know how to create a topic. Pulsar supports two types of topics: persistent and non-persistent. It also supports partitioned topics. You can create a topic with the topics create command. The command takes the following arguments:

1. The topic name in the format {persistent|non-persistent}://tenant/namespace/topic. If the type of the topic is omitted, it defaults to persistent. If the tenant and namespaces are omitted, the topic is created in the default tenant and namespace.
2. An integer partition number. If the partition number is 0, the resultant topic is non-partitioned.

Usage:

1. Creating a persistent topic

pulsarctl topics create persistent://public/default/example-topic 0

2. Creating a non-persistent topic

pulsarctl topics create non-persistent://public/default/example-topic-2 0

3. Creating a partitioned topic in the default tenant and namespace

pulsarctl topics create example-topic-3 2

3. topics list

Just like creating a new topic, listing all the existing topics is a job you need to perform frequently as a Pulsar cluster administrator. The topics list command provides a quick overview of all the topics in a specified namespace. The command takes as argument the namespace in the format tenant/namespace. The output is shown in a table format where you get the names of the topics as well as whether they’re partitioned. You can also get the output in JSON or YAML format by using the --output=json or --output=yaml flags, respectively.

Usage:

1. List all the topics in chosen namespace / topic

pulsarctl topics list public/default

2. List all the topics for chosen namespace / topic in JSON format

pulsarctl topics list public/default --output=json

3. List all the topics for chosen namespace / topic in YAML format

pulsarctl topics list public/default --output=yaml

4. subscriptions create

A subscription, just like topics, is an integral part of a messaging system. It defines a named configuration that dictates how messages are delivered to consumers. You can create a subscription on a topic with the subscriptions create command. You can also subscribe to a topic from the latest position or specify a position manually. The command takes the following arguments:

1. The topic name. The format is the same as described in the topic create command.
2. The subscription name.
3. The --messageId flag that determines the position. The argument to this flag can be latest, earliest, or a message-id in the format ledgerId:entryId.

Usage:

1. Create a subscription from the latest position

pulsarctl subscriptions create my-topic my-subscription

2. Create a subscription from a specific position

pulsarctl subscriptions create --messageId 656:1 my-topic my-subscription-2

5. subscriptions list

The subscriptions list command can be used to list all subscriptions of a topic. This command was included because it’s a frequent operation when dealing with subscriptions. It takes the name of the topic as argument and displays the subscriptions in a table format.

Usage:

pulsarctl subscriptions list my-topic

6. functions create

Pulsar Functions are lightweight compute processes that consume messages from one or more topics, apply a user-specified processing logic to each message, and publish the result to another topic. They are similar in concept to serverless cloud platforms like AWS Lambda or Google Cloud Functions. Pulsar Functions can be written in Java, Go, or Python and deployed to the cluster using the functions create command. This command accepts a number of flags, such as:

1. --name - The name of the function.
2. --jar - Path to the JAR file of the function if it’s written in Java.
3. --go and --py - Path to the main Go executable binary, or the main Python file of the function if it’s written in Go or Python, respectively.
4. --class - The class name of the function.
5. --inputs - A comma-separated list of input topics.
6. --output - The output topic.
7. --tenant - The tenant of the function.
8. --namespace - The namespace of the function.

Usage:

1. Create a function in Java

pulsarctl functions create \
    --tenant public \
    --namespace default \
    --name my-function \
    --inputs my-input-topic \
    --output persistent://public/default/my-output-topic \
    --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
    --jar /examples/api-examples.jar

2. Create a function in Python

pulsarctl functions create \
    --tenant public \
    --namespace default \
    --name my-function \
    --inputs my-input-topic \
    --output persistent://public/default/my-output-topic \
    --classname my_function.MyFunction \
    --py my_function.py

7. functions putstate

Pulsar integrates with Apache BookKeeper to store states of functions. These states are stored as simple key/value pairs that can be accessed or modified by the function or via the admin API. This can be extremely useful in cases where you need to persist state across restarts. For example, a counter that keeps track of the number of words in messages processed by a function can be stored as a state. The functions putstate command allows you to put a key/value pair to the state associated with a function.

To put a key/value pair, you need to pass the key and value in key - value format, and to put a key/file path pair, the format is key = filepath.

The command takes the following flags:

1. --tenant and --namespace - The tenant and namespace of the function. If omitted, the default tenant and namespace are used.
2. --name - The name of the function.
3. --fqfn - The fully qualified function name.

Usage:

1. Put a key/value pair

pulsarctl functions putstate --name my-function \
            pulsar - hello pulsar

The string hello pulsar is stored under the key pulsar.

2. Put a key/file path pair

pulsarctl functions putstate --name my-function \
        fileKey = some_file

The contents of the file some_file is stored under the key fileKey as a ByteValue.

8. functions querystate

Closely related to the previous command, the functions querystate command can be used to query the state associated with a function. Just like the last command, this one is also frequently used while working with functions. The command takes the following flags:

1. --name - The name of the function.
2. --tenant and --namespace - The tenant and namespace of the function. If omitted, the default tenant and namespace are used.
3. --fqfn - The fully qualified function name.
4. --key - The key of the state to be queried.
5. --watch - Whether to watch for changes in the value associated with the key.
6. --output - The output format (text, JSON, YAML).

Usage:

1. Query a state

pulsarctl functions querystate --name my-function --key pulsar

2. Query a state and watch for changes

pulsarctl functions querystate --name my-function --key pulsar --watch

9. context set

Context in pulsarctl is instrumental if you’re working with a multi-cluster setup. Using contexts, you can cache the information of multiple clusters and seamlessly switch between them. The context set command creates a new context and stores it in $HOME/.config/pulsar. The command takes the following arguments:

1. The name of the context.
2. The broker service URL using the --admin-service-url flag.
3. The bookie service URL using the --bookie-service-url flag.

Usage:

Create a context named my-context-1

pulsarctl context set my-context-1 --admin-service-url="http://localhost:8080" --bookie-service-url="http://localhost:6650"

10. context use

Once you have created the contexts, you can use context use to switch between them. The command takes the name of the context as argument and switches to that context. Since this command is used heavily in a multi-cluster setup, it undoubtedly makes the list of top pulsarctl commands.

Usage:

Switch to the context named my-context-1

pulsarctl context use my-context-1

Conclusion

Pulsarctl can be an indispensable tool for managing Apache Pulsar. Compared to pulsar-admin, pulsarctl commands provide a more user-friendly, intuitive interface. The ten pulsarctl commands listed here are used often and provide a good starting point for learning how to use pulsarctl. For a complete reference of all the commands, check out the pulsarctl documentation.

If you’re looking for a robust messaging and streaming solution, check out StreamNative. StreamNative, from the creators of Apache Pulsar, provides offerings like StreamNative Cloud—Apache Pulsar as a Service—and StreamNative Platform, a cloud-native streaming platform. With its global 24/7 support and unparalleled expertise in Apache Pulsar, it is an excellent choice for a messaging platform.