leveraging separate Pulsar clusters that use separate BookKeeper clusters,
leveraging separate Pulsar clusters that share one BookKeeper cluster, and
using a single Pulsar cluster with a single BookKeeper cluster. Each of these approaches and their specific use cases are discussed at length in the subsequent blogs.
Pulsar Isolation Part II: Separate Pulsar Clusters shows you how to achieve isolation between separate Pulsar clusters that use separate BookKeeper clusters. This shared-nothing approach offers the highest level of isolation and is suitable for storing highly sensitive data, such as personally identifiable information or financial records.
Pulsar Isolation Part III: Separate Pulsar Clusters Sharing a Single BookKeeper Cluster demonstrates how to achieve Pulsar isolation using separate Pulsar clusters that share one BookKeeper cluster. This approach uses separate Pulsar broker clusters in order to isolate the end-users from one another and allows you to use different authentication methods based on the use case. As a result, you gain the benefits of using a shared storage layer, such as a reduced hardware footprint and the associated hardware and maintenance costs.
In this fourth and final blog of the series, we provide a step-by-step tutorial on how to use a single cluster to achieve broker and bookie isolation. This more traditional approach takes advantage of Pulsar’s built-in multi-tenancy and removes the need to manage multiple broker and bookie clusters.
This tutorial is based on docker 20.10.10, docker-compose 1.29.2, and MacOS 12.3.1.
Get the docker-compose configuration files.
git clone https://github.com/gaoran10/pulsar-docker-compose
Start the cluster.
Check the pods.
Name Command State Ports
bk1 bash -c export dbStorage_w ... Up
bk2 bash -c export dbStorage_w ... Up
bk3 bash -c export dbStorage_w ... Up
bk4 bash -c export dbStorage_w ... Up
broker1 bash -c bin/apply-config-f ... Up
broker2 bash -c bin/apply-config-f ... Up
broker3 bash -c bin/apply-config-f ... Up
proxy1 bash -c bin/apply-config-f ... Up 0.0.0.0:6650->6650/tcp, 0.0.0.0:8080->8080/tcp
pulsar-init bin/init-cluster.sh Exit 0
zk1 bash -c bin/apply-config-f ... Up
After the cluster initiation completes, we can begin setting the broker isolation policy.
Download a Pulsar release package to execute the pulsar-admin command.
tar -txvf apache-pulsar-2.10.0-bin.tar.gz
// we can execute pulsar-admin command in this directory
zk1 is up-to-date
bk1 is up-to-date
bk2 is up-to-date
bk3 is up-to-date
broker1 is up-to-date
broker2 is up-to-date
broker3 is up-to-date
Creating broker4 ... done
proxy1 is up-to-date
The default value of the configuration bookkeeperClientRackawarePolicyEnabled is true, so the RackawareEnsemblePlacementPolicy is the default bookie isolation policy, we'll set the rack name like this /rack.
Check ledger metadata for the newly added ledgers [5,6,7,8,9]. Because bookie1 is not usable and the configuration bookkeeperClientEnforceMinNumRacksPerWriteQuorum is false, we should find that the secondary bookies are used. Bookie3 is in the primary group so bookie3 is always used.
Learn Pulsar Fundamentals with StreamNative Academy: If you are new to Pulsar, we recommend taking the self-paced Pulsar courses developed by the original creators of Pulsar.
Spin up a Pulsar cluster in minutes: Sign up for StreamNative Cloud today. StreamNative Cloud is the simple, fast, and cost-effective way to run Pulsar in the public cloud.
Save your spot at the Pulsar Summit San Francisco: The first in-person Pulsar Summit is taking place this August! Sign up today to join the Pulsar community and the messaging and event streaming community.
Ran Gao is a software engineer at StreamNative. Before that, he was responsible for the development of search service at Zhaopin.com. Prior to that, he worked on the development of the logistics system at JD Logistics. Being interested in open source and messaging systems, Ran is an Apache Pulsar committer.