Pulsar Operators Tutorial Part 4: Use kpack to Streamline the Build Process

Note: StreamNative now offers a unified approach to managing Pulsar clusters on Kubernetes systems, transitioning from two distinct versions of operators—Pulsar Operators (Basic Version) and StreamNative Operator (Advanced Version)—to a single, consolidated operator, StreamNative Operator, effective from the start of 2024. As part of this change, we will cease the release of new versions of Pulsar Operators, with future updates and enhancements being exclusively available through the StreamNative Operator, accessible only via StreamNative's paid services.

In the previous blog, I demonstrated how to containerize Pulsar client apps (producer and consumer) using Dockerfiles in VS Code. This is probably the most common way for the cloud-native build process. However, as Pulsar supports many languages, maintaining different Dockerfiles for Pulsar consumer/producer/function apps can be difficult as your system grows. For example, specifying dependency versions, changing base build and run images, mounting new ConfigMaps and Secrets (externalizing configurations), and adding TLS certificates can become more complicated. Using Dockerfiles forces developers to maintain those items while writing cloud-native apps.

In this blog, I will demonstrate how to streamline this process using kpack so that developers can focus on writing Pulsar producers, consumers, or functions with different languages.

Install and configure kpack

kpack is a Kubernetes operator implementing Cloud Native Buildpacks. If you like Google Cloud Build and want to implement it in your Kubernetes clusters, kpack is an ideal tool. For kpack/buildpacks details, you can find their concepts here and here.

1. kpack provides a Kubernetes operator. First, you must install the kpack operator in the Kubernetes namespace kpack.

kubectl create namespace kpack
kubectl apply -n kpack -f https://github.com/pivotal/kpack/releases/download/v0.5.4/release-0.5.4.yaml

2. Once the operator is installed, store the pull Secret of the Docker registry so the kpack operator can store the images. You can create a Secret to store your Docker registry pull credential or robot token.

kubectl create secret -n kpack docker-registry mydocker \
                 --docker-username= \
                 --docker-password= \
                 --docker-server=https://index.docker.io/v1/

3. Create a service account in the kpack namespace and associate the Secret with this service account. Note that you need secrets and imagePullSecrets in this service account.

kubectl apply -f - 
4. Create a custom resource ClusterStore to store the necessary buildpacks. Here, I list some buildpacks for Python (cpython, python-start, pip-install, pip, procfile and ca-certificates). Refer to the [kpack doc](https://github.com/pivotal/kpack) for more details. 

kubectl apply -f - 5. Create a cluster stack which defines the build and run images. From this custom resource, you may find it is similar to “multi-stage build” in a Dockerfile.

kubectl apply -f - 
6. Define a builder. A kpack builder is similar to “docker build, tag, push”.

kubectl apply -f -

Build the client app

After you deploy a ClusterStore, a ClusterStack and a Builder, you are ready to build some images. These images are defined as Custom Resources too.

1. Create the producer image.

kubectl apply -f - 
2. Create the consumer image.

kubectl apply -f - 3. Once those two image CRs are applied, you can use kp (kpack cli) or kubectl to check the build status. After “Steps Completed” reaches “export”, you can find that the image is pushed to the Docker registry you specified in the “image.spec.tag”.

kubectl describe -n kpack build pulsar-consumer-image-build-1
  …
  Steps Completed:
    prepare
    analyze
    detect
    restore
    build
    export
   …

4. You can reuse the ConfigMap and Deployment in Part 3 to test the container images. The following code is the same as the one in Part 3.

kubectl apply -f - 
5. Create a Deployment and a ConfigMap for the consumer.

kubectl apply -f - Once these two containers are deployed, you should find that the messages have been delivered. Then, you can follow the same ArgoCD project in Part 2. You can git push the kpack CRs to a GitHub repository and create an ArgoCD app to automate the image build process.

Conclusion

This blog shows how we can automate the container build process with two Pulsar Python client apps. As you can see, the Python code is just a GitHub repository tag in this tutorial. Whenever developers push their code to GitHub, the kpack build process will kick in and rebase the image.

You can find the example in my GitHub repositories.

• Pulsar Python client code
• kpack Python example

More on Apache Pulsar

Pulsar has become one of the most active Apache projects over the past few years, with a vibrant community driving innovation and improvements to the project. Check out the following resources to learn more about Pulsar.

• Pulsar Virtual Summit Europe 2023 will take place on Tuesday, May 23rd, 2023! See this blog to submit your session or become a community sponsor (no fee required).
• Start your on-demand Pulsar training today with StreamNative Academy.
• Spin up a Pulsar cluster in minutes with StreamNative Cloud. StreamNative Cloud provides a simple, fast, and cost-effective way to run Pulsar in the public cloud.
• Blog Pulsar Operators Tutorial Part 1: Create an Apache Pulsar Cluster on Kubernetes
• Blog Pulsar Operators Tutorial Part 2: Manage Pulsar Custom Resources with ArgoCD
• Blog Pulsar Operators Tutorial Part 3: Create and Deploy a Containerized Pulsar Client
• Blog StreamNative’s Pulsar Operators Certified as Red Hat OpenShift Operators
• Blog Introducing Pulsar Resources Operator for Kubernetes