Information Regarding Log4j Security Vulnerabilities

Updated on Dec. 21st to add details of how the vulnerability impacts other Pulsar ecosystem tools.

We wanted to provide an update on the Log4Shell critical vulnerability (known as log4j). Below we provide a status of the vulnerability in the open-source Apache Pulsar and in the StreamNative products, as well as any actions that need to be taken to mitigate the vulnerability.  

If there are any additional questions, please open a support ticket at https://support.streamnative.io or email support@streamnative.io.

Apache Pulsar Open Source Update

A total of three CVEs impacting log4j have been discovered. Of the three vulnerabilities, only two impact Apache Pulsar by default and the Pulsar community has been working to patch all three CVEs.

As of the publish date, Pulsar releases 2.9.1 and 2.8.2 have updated log4j versions and mitigate all known vulnerabilities. Additional releases for Pulsar 2.7 and 2.6 are in progress.

The following table provides a summary of the impact and mitigation options for these vulnerabilities.

StreamNative Products Update

StreamNative has worked to mitigate and patch these issues quickly for all customers. The following table provides details on the current status and recommended actions.

Non standard log4j / Pulsar functions and process runtime

As seen above, log4j, even when configured with the mitigating no message lookup mitigation, can be exploited if the user has configured either customer log4j template strings or is using Pulsar functions with the process runtime.

As a summary, if you have configured your log4j template strings to contain references either to a context object ($${ctx:}, %x,%mdc, etc) or are using Pulsar functions with the process runtime, then you potentially are at risk. We encourage you to remove the custom settings, or if using pulsar functions with process runtime, to upgrade to 2.8.2 or 2.9.1.

Mitigating Pulsar Functions

Pulsar functions written in Java will need to be re-deployed in order to get the updated values. StreamNative Cloud Managed customers will be reached out directly in the coming weeks for any functions that are not mitigated. This should be as simple as using the pulsar-admin functions update command. For an example, please see https://pulsar.apache.org/docs/en/functions-deploying/#updating-cluster-mode-functions

How to mitigate using StreamNative Platform

If you are using StreamNative Platform, you can edit your values file accordingly to deploy the formatMsgNoLookups mitigation. We currently do not recommend updating your chart version, as the latest release contains other unrelated changes.

Shown below are the new values that need to be added or changed in your values file. NOTE: you may have previously seen a mitigation using the -Dlog4j2.formatMsgNoLookups=true being added to PULSAR_EXTRA_OPTS. These two mitigations are equivalent and either is sufficient.

<script>
bookkeeper:
  configData:
    LOG4J_FORMAT_MSG_NO_LOOKUPS: “true”
broker:
  configData:
    LOG4J_FORMAT_MSG_NO_LOOKUPS: “true”
proxy:
  configData:
    LOG4J_FORMAT_MSG_NO_LOOKUPS: “true”
streamnative_console:
  configData:
    LOG4J_FORMAT_MSG_NO_LOOKUPS: “true”
zookeeper:
  configData:
    LOG4J_FORMAT_MSG_NO_LOOKUPS: “true”
<script> 

Upgrading the version can be done as follows:

<script>
images:
  autorecovery:
    repository: streamnative/sn-platform
    tag: 2.8.1.30
  bookie:
    repository: streamnative/sn-platform
    tag: 2.8.1.30
  broker:
    repository: streamnative/sn-platform
    tag: 2.8.1.30
  functions:
    repository: streamnative/sn-platform
    tag: 2.8.1.30
  presto:
    repository: streamnative/sn-platform
    tag: 2.8.1.30
  proxy:
    repository: streamnative/sn-platform
    tag: 2.8.1.30
  zookeeper:
    repository: streamnative/sn-platform
    tag: 2.8.1.30
  streamnative_console:
    repository: streamnative/sn-platform-console
    tag: "1.10-rc2"
<script> 

Both mitigations can be deployed simultaneously, and it is in fact recommended.

How to mitigate via open-source Pulsar

If you are using open-source Pulsar, please see the blog post here for instructions on mitigation. Also we still encourage you to upgrade to 2.8.2 or 2.9.1 when possible.

One additional note for users of either the open-source or StreamNative Helm charts also do not need to wait for images to be updated in the helm chart and new versions can be specified similar to platform charts shown above.

Ecosystem Tools

There have also been questions regarding other tools in the Pulsar ecosystem.

• Pulsar Manager - Pulsar manager is not directly affected by the log4j vulnerability. Pulsar manager is built using the spring framework, which uses logback for logging. Logback is also impacted, but unlike the log4j exploit, it requires access to directly edit logback configuration file which drastically lowers the severity of the issue. We will release a new version once spring releases a new version with the fix.
• Pulsar Spark/Flink Connector - Both the Apache Flink and Apache Spark connectors for Pulsar are not directly impacted. Neither connector directly includes log4j, and instead uses the log4j included in your Flink or Spark distribution. Upgrading your Flink or Spark deployments will mitigate any issues
• Pulsar IO Connectors - Connectors within Pulsar IO also (by default) do not include their own log4j, and instead rely on the log4j provided by the Pulsar IO framework. Following the instructions above for functions and redeploying connectors will mitigate the issue. This is true for all connectors included in Pulsar as well as StreamNative supported connectors. Custom connectors may need to be validated independently.

FAQ

How serious is this issue?

The underlying issue is potentially very serious, with a Remote Code Execution (RCE) vulnerability allowing for arbitrary code to severely expose a system to additional damage by an attacker, potentially gaining complete system access.

At this time, there are no known RCE exploits against Apache Pulsar. However, we recommend still treating this issue as critical, as the mechanism for executing code is complex and many potential vectors exist. We do know that attacks which could expose environment variables or other environment data is possible.

We strongly recommend taking action ASAP.

Why has it taken so long to get releases in open-source?

Releasing open-source Pulsar involves following a well-defined process in the community. As additional exploits were found over the past week, the community / Pulsar PMC decided to restart the process multiple times in order to select the latest version of log4j which mitigates all open issues.

Was there any known impact to StreamNative customers and their data?

At this time, we are not aware of any usage of this exploit that resulted in loss of secrets or customer data. There have been some automated reports via cloud vendors of attackers attempting to leverage the exploits, but no successful attempts have been found so far. For customers where we received warning reports, we are in the process of rotating the credentials.

‍