Pulsar Newbie Guide for Kafka Engineers (Part 7): Pulsar Security for Kafka Admins

‍TL;DR

This post covers how authentication and authorization work in Pulsar, drawing parallels to Kafka’s security. Pulsar supports pluggable auth mechanisms (TLS, OAuth2, JWT tokens, Kerberos, etc.) and fine-grained authorization (permissions at topic or namespace level). We’ll explain how to set up common auth methods (e.g., mutual TLS or token-based auth, akin to Kafka SASL/SSL), and how Pulsar’s multi-tenant design makes ACL management both powerful and a bit different from Kafka’s. In short: Pulsar can do everything Kafka can (SSL encryption, SASL auth, ACLs) and more, like multi-tenant isolation and token-based auth built in.

Authentication in Pulsar

Pulsar, like Kafka, can be run in plaintext (no auth) or with authentication enabled. By default, Pulsar has no authentication required (open cluster). In production you’ll want to enable one or more auth methods.

Supported auth providers in Pulsar include:

• TLS authentication (using client certificates – similar to Kafka’s SSL client auth).
  
• Token-based authentication (using JWTs or arbitrary tokens).
  
• OAuth2 (good for cloud identity integration).
  
• Kerberos (yes, Pulsar can use Kerberos SASL via JAAS, which is analogous to Kafka’s GSSAPI).
  
• Basic (username/password, typically not used in production unless over TLS).
  
• Athenz (Yahoo’s system) for those in specific environments.
  

These correspond to Kafka’s SASL mechanisms:

• Kafka SASL_SSL with PLAIN or SCRAM -> Pulsar’s “Basic” or token can cover this (though token is more robust because it’s not plaintext creds each time, it’s signed tokens).
  
• Kafka SASL_SSL with GSSAPI (Kerberos) -> Pulsar’s Kerberos option.
  
• Kafka SSL client certs -> Pulsar TLS auth.
  

Configuring Authentication:
In Pulsar, you enable authentication on brokers (and proxies if using). For example, in broker.conf:

This would enable both token and TLS auth providers. Pulsar can allow multiple auth methods at once (it will try to authenticate clients using each provider in turn).

Clients then must authenticate using one of these:

• For token: they provide a token (usually a JWT) on connect (e.g., using AuthenticationFactory.token("tokenString") in Java client).
  
• For TLS: they present a client certificate signed by a trusted CA to the broker.
  
• For OAuth2: they perform the OAuth flow (Pulsar client library supports it by obtaining an access token and passing it).
  
• For Kerberos: similar to Kafka, you’d configure JAAS and such, and the Pulsar client will do SASL handshake (if using the Kafka-on-Pulsar protocol, or perhaps in proxy).
  

A common modern approach is token auth, which is easier to manage than setting up a Kerberos infrastructure or distributing certs. You create a token (JWT) for a role (like “app1”) and the broker uses a secret or public key to verify it. Pulsar has utilities to create tokens (e.g., bin/pulsar tokens create --subject app1 --private-key my-sec.key). The broker config has the public key to validate.

Encryption in Transit: Pulsar can (and should) run on TLS. You’d configure brokerServicePortTls=6651, etc., to have brokers use TLS for their internal comms and for client connections (set up TLS listeners). This is analogous to Kafka’s SSL encryption. So you can have pulsar+ssl:// URLs for clients. Typically, enable tlsAllowInsecure=false and provide certs. Pulsar supports both a plaintext and TLS port simultaneously if desired (similar to Kafka’s multiple listeners).

One difference: Pulsar’s multi-tenancy means you might issue different credentials per tenant or role, and those roles are part of the token or cert CN. In Kafka, an ACL entry might refer to a Kafka principal like User:alice. In Pulsar, a “role” is similar to a principal. For instance, a token might carry “subject”:”alice”, making alice the role.

Authorization in Pulsar

Authorization (which resources a client can access) in Pulsar is integrated with tenants and namespaces:

• Pulsar supports ACLs at the namespace (and topic) level. You grant a role with produce/consume permission on a namespace or a specific topic.
  
• For example, to allow role “sensor-app” to produce to topics in iot/sensors namespace:

• And perhaps another for consumption.
  
• These permissions are stored in ZK/metadata and enforced by brokers. If a client with role sensor-app tries to produce to a topic in that namespace and it has produce permission, fine. If not, it gets an unauthorized error.
  

Important: Pulsar’s tenants add an admin layer. A role can be made a tenant admin (which allows creating namespaces, etc.). Also, Pulsar has the concept of superusers (configured in broker.conf) who have carte blanche (like Kafka’s super.users setting).

Kafka’s authorization works with topics, cluster, group resources, etc., via ACLs. Pulsar’s is a bit simpler in that it’s mostly produce/consume on topics (or namespaces) and admin operations on namespaces/tenants. Kafka’s granular actions (Describe, Create, Delete) have equivalents like Pulsar admin API can be controlled if you integrate with something like function-worker but by default, Pulsar doesn’t have separate ACL for “can this role create a topic” aside from being tenant admin.

Default Authorization: If you enable authorization (authorizationEnabled=true on broker), and no permission is set for a role on a resource, the access is denied. You must populate grants as needed. You can automate that or use Pulsar’s REST/CLI to do it.

Example for a Kafka ACL scenario:
In Kafka, you might do:

In Pulsar:

This would allow alice to both produce and consume on all topics in that namespace. If you want to restrict to a single topic, Pulsar CLI has topics grant-permission too, but typically namespace-level is used for manageability.

Role token vs role name: If using token auth, the token maps to a role (subject) internally. If using TLS, the client certificate’s CN or SAN is the role. If using Kerberos, the short Kerberos principal becomes the role (like “alice@EXAMPLE.COM” might map to “alice” depending on config). Ensure consistency so the role in the auth method matches what you use in grant-permission.

Multi-tenancy means you likely segregate roles by tenant. For example, tenant “finance” might have roles “financeApp1”, etc. Pulsar’s authorization is cluster-wide but often you manage per tenant.

Kafka vs Pulsar Security Quick Comparison

• TLS Encryption: Both Kafka and Pulsar can use TLS for encryption. Setup is similar (keystores, truststores or PEM files, configuring listeners). Pulsar documentation provides steps to enable TLS on brokers and clients.
  
• Client Auth: Kafka’s options are SASL (with PLAIN, SCRAM, GSSAPI, OAUTHBEARER, etc.) or mTLS. Pulsar offers similar range: you can do TLS + token (somewhat like SASL + SSL), or just TLS with client certs, or OAuth2 (similar to SASL/OAUTHBEARER concept), or Kerberos.
  
• Token vs SASL/PLAIN: Pulsar’s JWT token is a stateless way of auth, which many prefer to storing passwords on brokers (like Kafka’s SCRAM). Tokens can have expiry, and Pulsar brokers can auto-expire auth sessions or revalidate (token auth can be configured to not require re-auth each time or to periodically require).
  
• Authorization Domains: Kafka’s ACLs are global to cluster (no concept of tenant). Pulsar’s ACLs logically separate by tenant/namespace – which is nice because e.g. role “alice” can be allowed in tenantA without affecting tenantB. Kafka would need you to prefix resources or run separate clusters.
  
• Superuser/Admin Rights: Kafka ACL has CLUSTER action for cluster-wide operations. Pulsar uses a superuser list in config (e.g., admin roles) to allow all actions. Also, tenant admins can be designated to manage their tenant.
  

Setting Up a Simple Auth Scenario

Imagine you have a Pulsar cluster and you want:

• Clients to authenticate via token.
  
• One tenant per team.
  

Steps:

1. Enable TLS on broker (so token isn’t sniffable): Set up broker certificates and enable TLS listener.
   
2. Enable authentication with AuthenticationProviderToken. Provide a secret key to broker to verify tokens (e.g., a public key if tokens are signed with private key).
   
3. Create tokens for roles: Use Pulsar token tool to generate JWTs for roles like “team1-producer”, “team1-consumer”. Distribute those securely to apps (like how you’d distribute Kafka credentials).
   
4. Enable authorization and create a tenant “team1”, assign allowed clusters.
   
5. Grant permissions: For namespace team1/ns1, grant produce to team1-producer role, consume to team1-consumer role:
   

6. Now those roles can only do those actions in that namespace. If team1-consumer tries to produce, it fails authorization.
   
7. Client config: Producer passes its token (with role team1-producer) to Pulsar client config, uses service URL with TLS. Consumer similarly uses its token.
   

From that point, the broker will enforce:

• Only connections with valid tokens signed by our secret key are accepted (others get auth error).
  
• Those connections can only do what they’re permitted. If someone with team1-producer token tries to subscribe (consume), broker returns authorization error.
  
• Multi-tenancy: If someone tries to use the wrong tenant (say team1-producer sending to tenant2’s topic), it’s not authorized because team1-producer role likely has no perms on tenant2.
  

Kerberos scenario: If you have Kafka using Kerberos, Pulsar can integrate too. Pulsar’s Kafka-on-Pulsar (KoP) can even accept SASL GSSAPI and map to a token internally. But natively, you can enable AuthenticationProviderKerberos. The setup is similar to a Hadoop or Kafka Kerberos setup (JAAS config, keytabs for broker and client). Many find tokens simpler nowadays.

OAuth2 scenario: Pulsar can delegate auth to an OAuth2 server (like Auth0, Azure AD, etc.), which is analogous to Kafka’s OAUTHBEARER SASL where clients present a bearer token from some IdP. It’s a modern approach for cloud. StreamNative’s docs have details on enabling that. Essentially, broker trusts a certain JWT issuer, clients get a JWT from that issuer and present to broker.

Auditing and Best Practices

• Use secure connections (TLS) so that authentication tokens or credentials aren’t intercepted.
  
• Limit superuser access; ideally only infrastructure accounts are superusers.
  
• Each tenant can have admins – you might grant a team lead a token that allows them to manage their tenant (create namespaces, etc.) without making them cluster superuser.
  
• Rotate tokens or credentials periodically. Pulsar tokens can have expiration, and brokers can require refresh. Pulsar’s broker supports token revocation lists as well (so you can invalidate a token before expiry by configuring a callback or cache clear).
  
• Monitor auth failures in broker logs or metrics – repeated failures could indicate an attack or misconfig.
  

Key Takeaways

• Pulsar supports a wide array of authentication mechanisms – from TLS certs to OAuth2 and Kerberos. This gives parity with Kafka’s SASL/SSL capabilities, and then some (built-in JWT support which Kafka doesn’t have out-of-the-box).
  
• Multi-tenancy and roles: Pulsar’s security model is built around roles and tenants. Roles (identities) can be granted permissions on resources (like produce/consume on a namespace). Tenants group those permissions logically. Kafka’s ACLs are global and need to be managed for each topic; Pulsar can cut down on admin by granting at namespace (group of topics) level.
  
• Token-based auth (JWT) in Pulsar provides a simple way to issue credentials without a heavy infrastructure like Kerberos. It’s analogous to SASL/PLAIN or OAUTHBEARER but with cryptographic verification and optional expiration.
  
• Authorization (ACLs) in Pulsar is straightforward: specify role, actions, and resource (namespace or topic). Kafka ACLs have more operations but Pulsar covers the main ones (produce, consume, admin).
  
• Because of multi-tenancy, isolation is stronger: one tenant’s admin can’t affect another tenant. In Kafka, an admin could theoretically create topics or ACLs anywhere if they had rights. In Pulsar, you’d typically give each tenant a scoped admin.
  
• Setting up TLS and auth in Pulsar is a bit of a learning curve but very similar conceptually to Kafka – certificates, trust, config files. If you’ve secured Kafka, securing Pulsar is very achievable.
  

Coming up, Part 8 will move into operations: Load Balancing with ExtensibleLoadManager, where we discuss how Pulsar’s brokers distribute load – something we touched on with bundles, now more on the load manager mechanism and how it compares to Kafka’s static partition assignment.

‍

-----------------------------------------------------------------------------------------------------------

‍

Want to go deeper into real-time data and streaming architectures? Join us at the Data Streaming Summit San Francisco 2025 on September 29–30 at the Grand Hyatt at SFO.

30+ sessions | 4 tracks | Real-world insights from OpenAI, Netflix, LinkedIn, Paypal, Uber, AWS, Google, Motorq, Databricks, Ververica, Confluent & more!

[Explore the Full Agenda]

[Register Now]