CVE-2021-44228: Log4j vulnerability impact on FlexNet Operations On-Premises

[#1]

Summary:

A vulnerability identified as CVE-2021-44228 has been reported in the Apache Log4j library. This vulnerability may allow for remote code execution in susceptible products.

Problem Description:

Upon analysis, CVE-2021-44228 has been determined to impact the core module of FlexNet Operations On-Premises.   

Resolution:

A hotfix for FlexNet Operations On-Premises 2021 R1 is available for download from the Product and License Center. (Note: A community login with Product and License Center is required.)

We advise customers on earlier versions of FlexNet Operations On-Premises, who cannot upgrade to FlexNet Operations On-Premises 2021 R1, to use the mitigation steps described in the 'Workaround' section.

Workaround:

At this time, we discourage customers from upgrading Log4j files to a later Log4j version. Doing so may result in runtime problems or unexpected issues.

Until the new FlexNet Operations On-Premises patch can be delivered, we advise customers to remove the JndiLookup.class from the classpath using the platform-specific instructions below:

For Windows:

1. Stop the server.
2. Rename the "log4j-core-2.*.jar" to "log4j-core-2.*.jar.zip" to enable Windows Explorer to Open the file. (Note: FlexNet Operations On-Premises has 10 log4j-core files with version 2.8)
3. Drill-down into the "log4j-core-2.*.jar.zip" using Windows Explorer to select the "org/apache/logging/log4j/core/lookup/JndiLookup.class".
4. Delete the "JndiLookup.class" by right-clicking to select "Delete" from the Context-menu.
5. Click "Yes" on the "Delete File" dialog and the JndiLookup.class will be deleted from the selected "log4j-core-2.*.jar.zip".
6. Rename the "log4j-core-2.*.jar.zip" back to "log4j-core-2.*.jar".
7. Start the server.

For Linux:

Locate the log4j-core files using the command below:

find <path to deployment root> -name log4j-core-*.jar

e.g. find /root/FlexNet-Operations/ -name log4j-core-*.jar

Delete the JndiLookup.class file from all located log4j-core.jar files using the command below:

zip -q -d ./components/wildfly/standalone/deployments/flexnet.ear/flexnet-data/site/reporting/talend/lib/log4j-core-2.8.2.jar  org/apache/logging/log4j/core/lookup/JndiLookup.class

Additional Information:

• CVE Definition: https://nvd.nist.gov/vuln/detail/CVE-2021-44228
• Expanded CVE Definition: https://www.cve.org/CVERecord?id=CVE-2021-44228
• Apache Security Site for CVE severity, score, and vector string: https://logging.apache.org/log4j/2.x/security.html

[jefflaing]

We have applied the deletion mitigation and found that our server is still functional, so thank you.

One issue that arose during the fixups related to our blocking of all outgoing connections from that server.  Incoming activation requests seemed to work fine, but attempting to create a new entitlement resulted in an application failure, which the logs reported as a connection timeout (that later resulted in another exception relating to an inability to close a transaction.

Are there any additional requirements on network connectivity that are known for an FNO host?  At the moment, we know that you need

- to be able to verify our license of the FNO software

- to be able to contact our SMTP server

Is it also necessary to be able to connect out to any other external host?  Or even the current host by IP address rather than loopback (127.0.0.1)?  Our blocking was done in a VM host outside the FNO server itself in an attempt to ensure it was complete.

I am suspicious, from looking at the trace, that it may have been doing something like looking up an XSD from somewhere?

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_102]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_102]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_102]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_102]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[?:1.8.0_102]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_102]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_102]
at java.net.Socket.connect(Socket.java:538) ~[?:1.8.0_102]
at sun.net.NetworkClient.doConnect(NetworkClient.java:180) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) ~[?:?]
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[?:?]
at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[?:?]
at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) ~[?:?]
at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLEntityManager.startEntity(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLEntityManager.startDTDEntity(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLDTDScannerImpl.setInputSource(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLDocumentScannerImpl$DTDDispatcher.dispatch(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at com.opensymphony.xwork2.util.DomHelper.parse(DomHelper.java:118) ~[struts2-core-2.5.17.jar:2.5.17]

[cvirata]

@jefflaing I'm glad the deletion mitigation was able to help. Let me follow up with our Engineering team to see if they can provide guidance on what you are observing.

[mrathinam]

Update: 31/Dec/2021:
FNO Onprem 2021.R1 Log4j version has been upgraded to 2.17.0 and 2021_R1_HotFix_log4j_Upgrade_2.17_and_SWM-9817.zip is now available in the  Product and License Center.
Note: The same will be updated in the document soon. 

[jefflaing]

While it's good to see that FNO 2021 has been updated, those of us stuck on FNO 2018 (due to our use of Oracle as our storage) will need to continue to have the mitigations in place (removing the JNDI class completely).

Is there any further word on exactly what the minimum external internet access is for an FNO to operate?

ie, our experience was that blocking *everything* prevented the creation of new entitlements, so we have opened more ports than my security team are completely comfortable with...

[mrathinam]

@jefflaing When the workaround is implemented successfully then I don't think there is any need to block everything, is the FNO port for example 8888 also blocked?. 

Best Regards,

[jefflaing]

We blocked everything, except:

- SMTP (so emails could get out)

- Flexera themselves, so the FNO product licensing checks could be made

Port 8888 is for INCOMING http requests and we left it as incoming only, nothing allowed out. Doing this allowed license activation, etc to function correctly but we found that it blocked entitlement creation for some reason.

We do have incoming RDP allowed, but only from our internal network so anytime we need to get new software onto it, we need to retrieve it from the internet somewhere else and then PUSH it onto the server via the very controlled RDP.  It's a pain not being able to Google directly from the FNO, but that's the price you pay for proper isolation.

I am suspicious that "Create Entitlement" must be doing an out-bound request via port 8888 and that is not being done using 'localhost' but rather the fqdn of the server, which results in a full out-of-machine request, rather than looping back internally and thus the severe firewall blocks it - ie, the TCP/IP stack doesn't short-circuit the connection to stay inside the same VM.

If this is the case, that's fine and I can get the network security types to allow the machine to "outbound connect to itself", I believe.  But I want to double-check that this is actually what I require.  It's all about having to justify on paper every security exception we make. And the fact that incoming activation works fine, but new entitlement doesn't is just a bit to squirrely for my liking...

[jefflaing]

So, over on Re: Flexera’s response to Apache Log4j vulnerabilities CVE-2021-4104, CVE-2021-45046, CVE-2021-45105 and CVE-2021-44228 Flexera say that they are "... reviewing and testing firewall configurations ..."

Is Revenera doing the same testing?

(Obviously, this is just my way of repeating my previous question - apologies)

[mrathinam]

Hi, @jefflaing Thanks for the query and glad that you have asked it (no need for any apologies I think 🙂 )let me check with the concerned team and come back with a Revenera plan for the same. 

Best Regards,

[jefflaing]

Hi @mrathinam , it's now the end of February and I haven't heard anything on this yet.  Has any progress been made on determining how strict a firewall can be on an FNO site?