Blog

Apr 3, 2023

5 min read

Install StreamNative Platform on minikube

Vikas Dadhich

Customer Reliability Engineer, StreamNative

Install StreamNative Platform on minikube

Text Link

StreamNative Cloud

StreamNative Cloud Tutorials

StreamNative Platform is a cloud-native messaging and event-streaming platform that enables you to build a real-time application and data infrastructure for both real-time and historical events. In this article, I will demonstrate how to install StreamNative Platform on minikube on a local machine to explore it with the lightweight Kubernetes distribution, and deploy Pulsar on it. I will perform all the steps on a Mac computer with Apple silicon.

This setup mainly required DFD (Docker for Desktop). You can download the dmg file for Mac.

Install minikube

minikube is a lightweight Kubernetes distribution that allows you to quickly create a local Kubernetes cluster. For educational purposes, we will use minikube in this blog to set up our environment with Kubernetes 1.23 as an example.

1. Download and install minikube using the following commands.

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-arm64
sudo install minikube-darwin-arm64 /usr/local/bin/minikube

2. Start Docker Desktop. You can test minikube by running _{minikube start --kubernetes-version=v1.23.0}. Once it is started, you can stop it.

Install Helm

Use the following command to install Helm 3 with _brew. We will use Helm to install StreamNative Platform later.

brew install helm

Install StreamNative Platform

Once all the above dependencies are installed and configured, follow the steps below to install StreamNative Platform on minikube.

1. Start Docker Desktop.

2. Start minikube with Kubernetes version 1.23.

minikube start -p minikube --kubernetes-version=v1.23.0

Pulsar Operators will be installed later, which support Kubernetes versions between v1.16 (inclusive) and v1.26 (exclusive).

3. Set the default context to _minikube.

kubectl config use-context minikube

4. Create a Kubernetes namespace called _pulsar. As we are going to install everything under the same namespace, we can set the default context to the _pulsar namespace. This allows us to perform operations in the _pulsar namespace by default without using the _-n option every time.

kubectl create ns pulsar
kubectl config set-context --current --namespace=pulsar

5. Add the _streamnative repository using _helm.

helm repo add streamnative https://charts.streamnative.io
helm repo update

6. Install Pulsar Operators.

helm upgrade --install pulsar-operator streamnative/pulsar-operator

To fully leverage the power of StreamNative Platform, you can choose to install the Vault operator, cert-manager, and Function Mesh operator. For more information, see the StreamNative Platform documentation.

Install Apache Pulsar

Now that we have installed Pulsar Operators, we can deployed the custom resources (CRs) of ZooKeeper, BookKeeper, and brokers on minikube.

To deploy these CRs, download the YAML files from this GitHub repository and apply them. Alternatively, follow the examples below to install and create Pods for each component. Install ZooKeeper first, then BookKeeper, and finally the Pulsar broker.

Create the ZooKeeper custom resource

1. Create a _{zookeeper.yaml} manifest file.

apiVersion: zookeeper.streamnative.io/v1alpha1
kind: ZooKeeperCluster
metadata:
  name: my
  namespace: pulsar
spec:
  image: streamnative/pulsar:2.11.0.1
  replicas: 1
  pod:
    resources:
      requests:
        cpu: "200m"
        memory: "512Mi"
  persistence:
    reclaimPolicy: Retain
    data:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: "10Gi"
    dataLog:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: "20Gi"

2. Run the following command to create the ZooKeeper Pod.

kubectl apply -f zookeeper.yaml

3. Verify if the ZooKeeper Pod has been created. Once it is up and running, you can install BookKeeper.

kubectl get pods

Install the BookKeeper custom resource

1. Create a _{bookkeeper.yaml} manifest file.

apiVersion: bookkeeper.streamnative.io/v1alpha1
kind: BookKeeperCluster
metadata:
  name: my
  namespace: pulsar
spec:
  image: streamnative/pulsar:2.11.0.1
  replicas: 1
  pod:
    resources:
      requests:
        cpu: "200m"
        memory: "512Mi"
  storage:
    reclaimPolicy: Retain
    journal:
      numDirsPerVolume: 1
      numVolumes: 1
      volumeClaimTemplate:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: "8Gi"
    ledger:
      numDirsPerVolume: 1
      numVolumes: 1
      volumeClaimTemplate:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: "16Gi"
  zkServers: my-zk-headless:2181

2. Run the following command to create the BookKeeper Pod.

kubectl apply -f bookkeeper.yaml

3. Verify if the BookKeeper Pod has been created. Once it is up and running, you can install the broker.

kubectl get pods

If you are running three bookie Pods, you must set anti-affinity to _false.

Install the Pulsar broker custom resource

1. Create a _broker.yaml manifest file.

apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarBroker
metadata:
  name: my
  namespace: pulsar
spec:
  image: streamnative/pulsar:2.11.0.1
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
    terminationGracePeriodSeconds: 30
  config:
    custom:
  replicas: 1
  zkServers: my-zk-headless:2181

2. Run the following command to create the broker Pod.

kubectl apply -f broker.yaml

3. Verify if the broker Pod has been created.

kubectl get pods

4. Expected output:

NAME                                                             READY   STATUS             RESTARTS      AGE
my-bk-0                                                          1/1     Running            0             2m30s
my-bk-auto-recovery-0                                            1/1     Running            0             70s
my-broker-0                                                      1/1     Running            0             30s
my-zk-0                                                          1/1     Running            0             3m23s
pulsar-operator-bookkeeper-controller-manager-76948555c6-xcw4r   1/1     Running            0             5m9s
pulsar-operator-pulsar-controller-manager-694c8974-kcgfq         1/1     Running            0             5m9s
pulsar-operator-zookeeper-controller-manager-69c6d-2qjml         1/1     Running            0             5m9s

5. To access the cluster, you can exec into the broker Pod and run the producer or consumer client.

kubectl exec my-broker-0 -it bash

I have no name!@my-broker-0:/pulsar/bin$ ./pulsar-admin --admin-url http://localhost:8080 tenants list
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/pulsar/lib/org.slf4j-slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
public
pulsar
sn

For the MacBook Pro M1 chip, make sure IPV6 is disabled. Always use the latest version of Docker Desktop and ensure the following settings are enabled before creating CRs.

‍

‍

Now, you can use _pulsar-admin to manage clusters, tenants, namespaces, topics, and more.

*☁️ Happy Learning ☁️*

More resources

Powered by Apache Pulsar, StreamNative Platform makes it easy to build mission-critical messaging and streaming applications and real-time data pipelines by integrating data from multiple sources into a single, central messaging and event streaming platform for your company. See the following resources for more details:

Vikas Dadhich

Vikas Dadhich is a Customer Reliability Engineer at StreamNative. With more than ten years in customer operations and technical support engineering roles, he is able to work independently and in a team, with excellent leadership, communication, organizational, interpersonal, and time management abilities.