In the previous blog, I demonstrated how to containerize Pulsar client apps (producer and consumer) using Dockerfiles in VS Code. This is probably the most common way for the cloud-native build process. However, as Pulsar supports many languages, maintaining different Dockerfiles for Pulsar consumer/producer/function apps can be difficult as your system grows. For example, specifying dependency versions, changing base build and run images, mounting new ConfigMaps and Secrets (externalizing configurations), and adding TLS certificates can become more complicated. Using Dockerfiles forces developers to maintain those items while writing cloud-native apps.
In this blog, I will demonstrate how to streamline this process using kpack so that developers can focus on writing Pulsar producers, consumers, or functions with different languages.
Install and configure kpack
kpack is a Kubernetes operator implementing Cloud Native Buildpacks. If you like Google Cloud Build and want to implement it in your Kubernetes clusters, kpack is an ideal tool. For kpack/buildpacks details, you can find their concepts here and here.
1. kpack provides a Kubernetes operator. First, you must install the kpack operator in the Kubernetes namespace kpack.
2. Once the operator is installed, store the pull Secret of the Docker registry so the kpack operator can store the images. You can create a Secret to store your Docker registry pull credential or robot token.
3. Create a service account in the kpack namespace and associate the Secret with this service account. Note that you need secrets and imagePullSecrets in this service account.
4. Create a custom resource ClusterStore to store the necessary buildpacks. Here, I list some buildpacks for Python (cpython, python-start, pip-install, pip, procfile and ca-certificates). Refer to the kpack doc for more details.
5. Create a cluster stack which defines the build and run images. From this custom resource, you may find it is similar to “multi-stage build” in a Dockerfile.
6. Define a builder. A kpack builder is similar to “docker build, tag, push”.
Build the client app
After you deploy a ClusterStore, a ClusterStack and a Builder, you are ready to build some images. These images are defined as Custom Resources too.
1. Create the producer image.
2. Create the consumer image.
3. Once those two image CRs are applied, you can use kp (kpack cli) or kubectl to check the build status. After “Steps Completed” reaches “export”, you can find that the image is pushed to the Docker registry you specified in the “image.spec.tag”.
4. You can reuse the ConfigMap and Deployment in Part 3 to test the container images. The following code is the same as the one in Part 3.
5. Create a Deployment and a ConfigMap for the consumer.
Once these two containers are deployed, you should find that the messages have been delivered. Then, you can follow the same ArgoCD project in Part 2. You can git push the kpack CRs to a GitHub repository and create an ArgoCD app to automate the image build process.
This blog shows how we can automate the container build process with two Pulsar Python client apps. As you can see, the Python code is just a GitHub repository tag in this tutorial. Whenever developers push their code to GitHub, the kpack build process will kick in and rebase the image.
You can find the example in my GitHub repositories.
More on Apache Pulsar
Pulsar has become one of the most active Apache projects over the past few years, with a vibrant community driving innovation and improvements to the project. Check out the following resources to learn more about Pulsar.