Patching the IBM Cognos Server to Mitigate Apache Log4j Security Vulnerabilities

[#1]

Summary

This document provides alternative ways to configure the Flexera Analytics Cognos server to mitigate security vulnerabilities caused by Apache Log4j.  Apache Log4j is used by IBM Cognos Analytics as part of its logging infrastructure.  These fixes address the exposure to the Apache Log4j vulnerabilities: CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 and CVE-2021-44832.  

Note: The configuration change is applicable to FlexNet Manager Suite on-premises versions from 2017 R2 to 2021 R1 (which included Cognos Analytics versions 11.0.6 – 11.0.13).

Method

Broadly speaking, you may choose any one of these three ways to remove the security vulnerabilities:

1. Run-time patch: This approach uses a file sourced from IBM to modify the behavior of Cognos Analytics, mitigating the Apache Log4j vulnerability without requiring any upgrade to Flexera Analytics (powered by Cognos).
   
   The log4jSafeAgent file provided for Cognos Analytics modifies the class byte code at Java startup time. It disables the vulnerable JndiLookup class, and enforces the StrSubstitutor recursion limit, without altering the installed product. If you choose this option, see Downloading the resources and 1. Run-time patch to Analytics (Cognos) Server below for step-by-step instructions.
2. Removal/Replacement: Two options for removal or replacement of the JndiLookup class have been proposed by IBM. The original advice from IBM (for removal) was made available to our Flexera Community customers in https://community.flexera.com/t5/FlexNet-Manager-Knowledge-Base/Flexera-Analytics-Cognos-mitigation-for-Apache-Log4j-2/ta-p/217655. If you wish to follow the removal path, please refer to that article for instructions.  Since then, IBM has released a patch that replaces the JndiLookup class with updated code that does not include the vulnerability. Replacement is achieved by running a provided installation patch on your existing Cognos server within your FlexNet Manager Suite implementation. If you choose this path, see Downloading the resources and 2. Replacing the JndiLookup file below for step-by-step instructions.
3. Reinstallation: This approach is more thorough-going, in that it replaces the entire installation of Flexera Analytics (powered by Cognos). This includes running the Flexera installation script in PowerShell, where the script installs an updated version of Log4j that removes the vulnerability. If you choose this path, see Downloading the resources and 3. Reinstalling Flexera Analytics below. In that set of step-by-step instructions, you may also choose either of the following fully-tested scenarios:
   • Remove and replace: This follows the standard upgrade path of removing the previous installation and installing the patched version of IBM Cognos.
   • Over-the-top installation: This is a slightly simplified approach where you leave the current installation untouched, and run the installer to replace it.

Choose the approach that best aligns with your security policies and practices. Next, collect the necessary resources.

Downloading the resources

All files required for any of the above approaches are included in one zip archive available from the Product and License Center. Follow these steps to secure the materials you need.

To download the resource file for log4j mitigation of Flexera Analytics:

1. In your preferred web browser, on your Flexera Analytics server (or a convenient alternative), navigate to the Flexera Product and License Center (available through the Product Access drop-down in the top right corner of the Flexera Community web page).
   Tip: Log in with your Flexera Community account is required.
2. Locate and download Flexera Analytics (Cognos) Log4j mitigation update.zip, saving to a working folder such as C:\temp\Log4jFix.
3. Unzip the archive to reveal the three folders inside:
   • 1 Run-time patch
   • 2 Replace JndiLookup
   • 3 Reinstall Flexera Analytics

Each of the following approaches uses the resources within the matching folder. You may ignore folders that do not match your chosen approach.

Now step forward to the approach you have selected from the introduction to Method. Each approach assumes that you are logged into your Cognos server using an account with administrator privileges.

1. Run-time patch to Analytics (Cognos) Server

 This approach installs the log4jSafeAgent file (provided by IBM) that modifies the class byte code at Java startup time. It disables the vulnerable JndiLookup class, and enforces the StrSubstitutor recursion limit, without altering the installed product.

To install the Analytics (Cognos) Server run-time patch:

1. Launch the IBM Cognos Configuration tool as an administrator, and stop the Cognos service if it is running.
2. Navigate to the Cognos installation directory.
   Note: This is typically located in C:\ProgramFiles\ibm\cognos\analytics.  Your path is identified as “install-path” through this process.
3. Take a backup of the files that need to be modified in this procedure:
   1. install-path\bin64\bootstrap_wlp_os_version.xml
   2. install-path\configuration\xqe.config.custom.xml.
4. In your downloaded and unzipped archive, navigate into the 1 Run-time patch folder.
5. From this folder, copy the log4jSafeAgent2021.jar file, and save the copy to the folder  install-path\webapps\p2pd\WEB-INF\lib.
6. Navigate to the install-path\bin64 folder, and in your preferred plain text editor, open the bootstrap_wlp_os_version.xml file.
7. In the <start> section of the file only, add a new javaagent parameter that refers to the log4jSafeAgent2021.jar file. You may copy the sample code shown below this process, and be sure to replace three instances of {install-path} with the installation path on your system.
8. Navigate to the install-path\configuration folder:
   1. If there is no xqe.config.custom.xml file present, save a copy of the xqe.config.xml as xqe.config.custom.xml in the configuration folder.
   2. In your preferred plain text editor, edit the xqe.config.custom.xml file to specify the javaagent parameter that refers to the log4jSafeAgent2021.jar file in the IBM Java Runtime Environment (JRE), as follows (some line wrapping has been applied here):
      <vendor name="IBM Corporation" options="-Xscmx100m
      -Xshareclasses:cachedir=
          ../javasharedresources^name=cognos10%u^nonfatal
      -Xmso512K -XX:+HeapManagementMXBeanCompatibility
      -Xjit:dontInline={gnu/trove/*}^{gnu/trove/*}
          (disableLoopVersioner)
      -Dcom.ibm.jsse2.overrideDefaultTLS=true
      -javaagent:../webapps/p2pd/WEB-INF/lib/log4jSafeAgent2021.jar"
      compressedrefs="-Xcompressedrefs" max="27000"/>
      
      
9. Save the modified file, and restart the Cognos service.
10. Verify that the mitigation process worked:
    1. From the install-path/logs directory, open the p2pd_messages.log file.
    2. Search for the following two entries. If these two entries are logged, the patch is applied and bootstrap_wlp_os_version.xml is configured correctly:
       • SystemOut O Modified class org.apache.logging.log4j.core.lookup.JndiLookup to empty method lookup
         Note: This entry applies to CVE-2021-44228 and CVE-2021-45046.
       • SystemOut O Modified class org.apache.logging.log4j.core.lookup.StrSubstitutor to check currentRecursionCount
         Note: This entry applies to CVE-2021-44832 and CVE-2021-45105.
    3. From the install-path/logs directory, open the dq_messages.log file, and search for the following two entries. If these entries are logged, the patch is applied and the xqe.config.custom.xml is configured correctly:
       • SystemOut O Modified class org.apache.logging.log4j.core.lookup.JndiLookup to empty method lookup
         Note: This entry applies to CVE-2021-44228 and CVE-2021-45046.
       • SystemOut O Modified class org.apache.logging.log4j.core.lookup.StrSubstitutor to check currentRecursionCount
         Note: This entry applies to CVE-2021-44832 and CVE-2021-45105.
11. If the log files don't show the expected entries, use the following steps to verify the process:
    1. From the Windows Start menu, run PowerShell as administrator.
    2. Run the following command (all on one line):
       wmic path win32_process get commandline | findstr "log4jSafeAgent2021"
    3. For the Analytics (Cognos) server, two processes should be returned. Each process must include one of the following entries:
       • -javaagent:install_path/webapps/p2pd/WEB-INF/lib/log4jSafeAgent2021.jar
       • -javaagent:../webapps/p2pd/WEB-INF/lib/log4jSafeAgent2021.jar

Here is the sample code for step 7 in the above process.

Important: Ensure that the new javaagent parameter is added within the <start> section of the file only, and not anywhere else in the file. Otherwise, the process will not be successful even if the verification step (10 b) returns the expected entry.

<start>........
 <param>"-javaagent:${install-path}/webapps/p2pd/WEB-INF/lib/log4jSafeAgent2021.jar"</param>
 <param>"-javaagent:${install-path}/wlp/bin/tools/ws-javaagent.jar"</param>
 <param>-jar</param>
 <param>"${install-path}/wlp/bin/tools/ws-server.jar"</param>
 <param>cognosserver</param>
 </spawn>
</start>

2. Replace the JndiLookup file 

This approach removes the affected contents of the JndiLookup file by running a downloaded installation patch on your existing Cognos server within your FlexNet Manager Suite implementation.

To install the IBM Cognos Server patch that replaces the JndiLookup file:

1. Launch the IBM Cognos Configuration tool as an administrator, and stop the Cognos service if it is running.
2. Navigate to the Cognos installation directory.
   Note: This is typically located in C:\ProgramFiles\ibm\cognos\analytics. Your path is identified as “install-path” through this process.
3. In a separate directory, make a backup copy of the Cognos installation folder, and name it as cognos_backup.
4. In SQL Server, make a backup copy of the Content Store database.
5. Without restarting the Cognos service, close the IBM Cognos Configuration tool.
   Important! Do not reopen the IBM Cognos Configuration tool until instructed to do so.
6. In your downloaded and unzipped archive (from Downloading the resources), navigate into the 2 Replace JndiLookup folder.
7. Copy and replace the preserve.txt file under your Cognos installation directory, by default:
   C:\Program Files\ibm\cognos\analytics\configuration\preserve
8. Also from your 2 Replace JndiLookup folder, extract the ca_srv-11.0.13-2201052300-winx64h.zip file to a convenient working directory, and identify the following three components:
• A lightweight installer (.exe)
• A server repository (.zip) somewhat confusingly with the same name
• An executable batch file (.bat) used to begin the installation process.
9. Run the ca_srv_win64_11.0.13.2201052300.bat file as an administrator. The .bat file runs the installer, upgrading your existing Analytics environment by installing Cognos 11.0.13 over the top.
10. A GUI interface to install IBM Cognos Analytics launches. Click Next to proceed with the installation.
11. Ensure that the installation location is the same as the previous installation. This will overwrite files in the existing installation directory.
12. Click Install to install the application.
    Note: A message may pop up indicating some folders are in use. Ensure that the Cognos Configuration tool is still closed. Click OK to continue the installation.
13. Once the upgrade is complete, click Done.
14. Navigate to the following Cognos installation directory and launch the Cognos Configuration tool (if you have used a custom installation path, adjust as required):
    C:\ProgramFiles\ibm\cognos\analytics\bin64\cogconfig.exe
15. Restart Cognos services. Click Yes to save the configuration when prompted.

3. Reinstalling Flexera Analytics

This process includes the steps both for a complete reinstallation, and also for a simplified, over-the-top installation.  To complete this process, you must use an account on your Flexera Analytics server that has administrator permissions (to install a service).

Remember: The Flexera Analytics server must be accessible by its host name, rather than just its IP address. Do not use IP addresses anywhere in the Flexera Analytics settings.

This article assumes that you are reinstalling on an existing system, such that you already have:

• Your chosen web server, correctly configured
• The SQL Server login account, which does not contain any of the greater-than, less-than, or ampersand characters (< > &)
• Microsoft® SQL Server® 2012 Native Client installed on the host server (where this is separate from the Cognos database server)
• The C:\FNMSCognosAnalytics\Support directory on the Flexera Analytics server
• Up-to-date license terms for Flexera Analytics.

To reinstall Flexera Analytics: 

1. Best practice is to take a full backup of your existing Flexera Analytics content store database and save it securely. (If there are any database problems in your upgrade, you can restore this protective copy to recover.)
2. Stop the IBM Cognos service, and ensure that the IBM Cognos Configuration program is not running.
3. From the downloaded and unzipped archive (from Downloading the resources), navigate into the 3 Reinstall Flexera Analytics folder.
4. From this folder, copy the following files into your C:\FNMSCognosAnalytics\Support folder (if prompted that existing files will be overwritten, accept this, as the replacements are required):
   1. analytics-installer-1.2.2-win
   2. ca_srv_11.0.13-2201052300-winx64h.zip
   3. InstallCognos.ps1
   4. CognosFlipperConfig.psm1
   5. ConfigureCognos.psm1
   6. CognosIISConfig.template.xml
   7. CognosConfigProperties.xml

Tip: The installation script automatically handles extracting the contents of the archive at b.; and the executable that is extracted automatically installs 32-bit software on 32-bit systems, and 64-bit software on 64-bit operating systems.

5. Locate your copy of CognosConfigProperties.xml from your previous installation. (Typically, this is in the C:\FNMSCognosAnalytics\Support folder; but it may be stored elsewhere if it contains plain-text passwords.)  This already contains all the required properties configured for your implementation. If required, you may review in Notepad, or your preferred text editor.  If you need guidance on the properties, refer back to your Installation Guide, available through docs.flexera.com for your installed release.
6. Prepare to run the PowerShell script:
   1. Open a PowerShell command-line window with administrator privileges.
   2. If you have not done so already, set the PowerShell permissions with the following command:
      set-ExecutionPolicy AllSigned -Force
      Respond to the warning text with the default Y.
7. For an over-the-top installation of Cognos Analytics version 11.0.13:
   1. In your PowerShell window, navigate to the Cognos installation directory.
      Tip: The typical path is C:\ProgramFiles\ibm\cognos\analytics.
   2. In a separate directory, make a backup copy of the Cognos installation folder, and name it cognos_backup.
   3. Run the following command in the PowerShell window:
      .\InstallCognos.ps1
      This script updates the build containing the log4j mitigation.
      (Skip the next step for a complete installation.)
8. For a complete upgrade:
   1. Optionally, uninstall Cognos from your Cognos server.
   2. In your PowerShell window, navigate to the directory where you copied the support directory. For example: C:\FNMSCognosAnalytics\Support
   3. Run the following command in the PowerShell window:
      .\InstallCognos.ps1
   4. A dialog box opens, prompting you to run the installer. Click Run to proceed with the installation.

Tip: Installation in either process may take some time to complete. After updating the configuration, the PowerShell script restarts the IBM Cognos service. If the script reports any difficulties restarting the service, it may be because of environmental issues, such as memory pressure. In this case, it is not necessary to run the PowerShell script again: you can try restarting the IBM Cognos service manually in Windows Service Manager.

9. Navigate to the root install-path of the Cognos Analytics installation directory and open the cmplst.txt file in a text editor. Verify that the kit_version has changed to 11.0.13.2201052300.
10. Confirm that you are able to access Flexera Analytics with FlexNet Manager Suite.

[DiannaB]

@pwesthorp have these options been tested against an installation where the Jdnilookup class was already removed per IBM's initial direction?  We want a fix that will completely clear this up in SVM, which the manual deletion of that class does not accomplish.  Option 1 might fail if that lookup class isn't present, so option 3 may be the only choice.  Thanks!

[darshanar]

@DiannaB 
Yes, Case 2 and Case 3 (over the top install) would work against an installation where JndiLookup class was removed. This was tested internally.
We will always recommend our customers to take full backups - Content Store DB backup and Cognos install directory full backup, before applying any updates/ patches.

[dbeckner]

Is there documentation available from Apache (who is the authoritative source for Log4j) that provides backing for the runtime patch provided by IBM? We have applied the recommended runtime patch but our cyber team is requiring documentation stating that Apache recognizes the patch as an approved mitigation for this CVE.

[darshanar]

@dbeckner 
We follow the course on what IBM sets in their website. Per the link, https://www.ibm.com/blogs/psirt/an-update-on-the-apache-log4j-cve-2021-44228-vulnerability/, IBM has provided remediation, which I believe should work as stated.
This link provides additional information on what IBM security has analyzed and provided remediation on.
https://securityintelligence.com/posts/apache-log4j-zero-day-vulnerability-update/?_ga=2.10012661.158993233.1650905466-958351102.1649788078

If you need further information, let us know.

[dbeckner]

@darshanar Can you confirm what version of Log4j is used in FNMS 2021 R1. Looking at the links you provided above as well as the official Apache documentation here: https://logging.apache.org/log4j/2.x/security.html the CVEs do not apply to Log4j 1.x.

Looking through my analytics server files I see the presence of Log4j 1.2.x, Log4j 1.3.x, Log4jx.1.4x, and Log4j-x-2.7 files.

Our security team is raising concerns about exactly what version of Log4j Flexera Analytics is using because Log4j 1.x is End of Life.

Additionally, if Log4j 1.x is used in Analytics what are Flexera's plans to update this to the latest approved version since it is EOL?

[darshanar]

@dbeckner 
For FNMS 2021 R1 version, IBM Cognos uses log4j-1.x and 2.x jars. 
FNMS 2021 R1 is on IBM Cognos version 11.0.13, that was identified to be impacted by the security vulnerabilities posed for log4j.
https://www.ibm.com/blogs/psirt/an-update-on-the-apache-log4j-cve-2021-44228-vulnerability/

I believe your security team is correct. Log4j.1.x is end of life. 
Flexera is working to upgrade analytics which will upgrade log4j version as well as include any log4j mitigations. We do not have an ETA at the moment, but are aiming for FNMS 2022 R2.

Just so you know, the FNMS 2022 R1 release contains the updated installers for log4j mitigation for Cognos version, 11.0.13 (this version is same as FNMS 2021 R1, but includes log4j mitigation).

[dbeckner]

@darshanar Thanks for this info. Will the upgraded analytics be backwards compatible with older versions of FNMS or will the customer be required to upgrade to 2022 R2?

[darshanar]

@dbeckner 
To my knowledge, it should be.  We would however need to test this on our end as well to confirm. 

[dbeckner]

@darshanar Thanks for the info!