FlexNet Inventory Agent upgraded automatically when this had not been requested

Report Inappropriate Content · ‎Nov 07, 2018

Summary

Identifying the cause, impact assessment, and remediation processes if required.

Symptoms

For cloud-based implementations, the FlexNet inventory agent performed an automatic upgrade to the latest version when this had not been requested. (On-premises implementations are not affected.)

After the self-update, the majority of affected inventory devices continue to operate normally using the 2018 R2 FlexNet inventory agent. This version is backward compatible with existing infrastructure (such as earlier versions of inventory beacons). Some inventory devices were not updated, as explained below.

Cause

An update to FlexNet Manager Suite released to US and EU UAT on October 4, and released to both the US and EU production cloud environments at 1am on October 15, 2018 (US CDT), inadvertently authorized updates of the FlexNet inventory agent to version 13.1.1 without first receiving your request for that change. We apologise for this inconvenience. This change was modified shortly before 8pm on October 15 to prevent further unexpected updates to FlexNet inventory agent deployed in your environments. The effects are as follows:

For those few customers who had previously requested that a specific version of FlexNet inventory agent be authorized for their use, there is no change. The patch did not override your settings, and there is no automatic upgrade to FlexNet inventory agent in your environment.
For customers whose distribution of policy has been delayed, the modification to restore original policy means that automatic upgrades of FlexNet inventory agent may be avoided.
For customers whose device policy updates occurred in the 19 hours between release of the patch and reverting policy, some FlexNet inventory agents (where updated as normal by policy) may have self-updated to version 13.1.1. Each installed FlexNet inventory agent chooses a random time within each 24 hours to request any policy updates, so that some installations may have updated and others in the same environment may not. You may wish to validate which installations of FlexNet inventory agent have self-updated to 13.1.1, and monitor to assure yourself that operations are continuing as normal. Notice that reverting the policy meant that the next policy distribution stops further auto-updates, leaving all installed FlexNet inventory agents at their current version: those that have been already updated to 13.1.1 stay at that version (there is no automated downgrade in this case), and any that have not yet been updated stay on their previous version.

For steps to help diagnose and repair impacts, please see the Steps to Reproduce section. For responses and changes from Flexera, please see the Resolution section.

Steps To Reproduce

To identify installed versions of FlexNet inventory agent

You may use this process to identify installations of FlexNet inventory agent that have self-upgraded to 2018 R2 on WIndows and UNIX-like operating systems (other than MacOS, where a known issue excludes those devices from this check).

In the web interface for FlexNet Manager Suite, navigate to License Compliance > All Applications.
Filter the Name column for FlexNet inventory agent, and identify the entry for FlexNet inventory agent 2018 R2. Remember that this page shows both applications in your environment, and also applications in the ARL that have not yet been identified in your environment.
In the Installed column for this row, check the number of installations for this version. If this number is zero, or is the value you expect because you have already authorized use of the 2018 R2 version, this procedure is complete.
For an unexpected, non-zero result in the Installed column, click the hyperlinked Name value to open the Application Properties page for FlexNet inventory agent 2018 R2.
Click the Devices tab to identify all inventory devices that have reported this version of FlexNet inventory agent. If this is a large number of devices, you may also wish to use the download button to copy the list into a spreadsheet for easier manipulation.
To finally check on an individual device, click its name to open its inventory device properties, and then select the History tab. An entry saying Linked to Software Application: FlexNet Inventory Agent 2018 R2 dated October 15 (US CDT, or local equivalent) is definite confirmation that the auto-upgrade has occurred on this device.

The Devices tab of the application properties shows the inventory devices that have satisfactorily upgraded to FlexNet inventory agent 2018 R2. One option you may consider is to accept this upgrade where successful, and simply monitor results as usual from these devices to ensure everything is as you expect. FlexNet inventory agent 2018 R2 is backwards compatible, and functions normally with your existing infrastructure (for example, with earlier versions of inventory beacons). This version also provides you the latest functionality, such as the ability to gather inventory from instances you may be running in AWS EC2.

If, instead, you prefer to switch to an earlier release of FlexNet inventory agent, see the appropriate process outlined further below.

To identify orphaned inventory devices

An inventory device is said to be 'orphaned' when its installed FlexNet inventory agent can no longer communicate with an inventory beacon.

Please note: No class of devices is known to be orphaned as a result of this incident. Specifically, some early concerns were raised by customers because Microsoft Windows Server 2003 had been removed from the list of 'supported platforms for FlexNet inventory agent' in the Release Notes. However, this end of support life does not mean that FlexNet inventory agent no longer runs on Windows Server 2003. Current versions, including FlexNet inventory agent for 2018 R2 (13.1.1), are running normally on Windows Server 2003 SP2. See also the Resolution section below for resulting changes in support life. During any upgrade process, individual inventory devices, regardless of the version of Windows, may be orphaned if the upgrade process fails at a particular point where the MSI cannot roll back. The process below can help to identify any individual orphaned devices, regardless of operating system.

You can use the Out-of-Date Inventory page to watch for devices that have stopped reporting, but that has a minimum 30-day time window. Assuming that your normal inventory upload schedule is more frequent (this example uses daily), you can check for impacts like this:

In the web interface for FlexNet Manager Suite, navigate to Discovery & Inventory - All Inventory.
Expose the header filter row, and filter Last inventory source to FlexNet Manager Suite. Only inventory from the FlexNet inventory agent (which is identified in that way) is relevant here.
Filter the Last inventory date to a period of 2-3 times your inventory collection schedule cadence on either side of October 15, 2018. For example, if you collect FlexNet inventory daily, you could enter the following (if you are using US date formats) into the filter field in the header: 10/13/2018 - 10/17/2018
Optionally, click the Last inventory date header to order the results by date, and look for devices whose last inventory upload was prior to October 15, and who are missing an inventory upload when next expected. Such devices should be individually checked to identify the reason, which may be unrelated to this incident.

To repair orphaned devices, see the summary further below.

To repair custom installation paths on UNIX-like devices

If you have used custom installation paths for FlexNet inventory agent on UNIX-like devices, any auto-update to 2018 R2 has (most likely) reverted to the default installation path. If the mgsft_rollout_response file was left in the temporary path for use during installation (/var/tmp), the upgraded FlexNet inventory agent may have collected the custom settings and completed the upgrade in your custom location, and be operating as normal. Without that temporary file, the upgrade likely completed and the FlexNet inventory agent may be operating as normal, but in the default installation path (the package technology would then have removed the installation from your custom path). If you had made other custom changes (such as customizing both the installation and data paths), this device may now be orphaned. You are likely to be familiar already with the topic Agent third-party deployment: Installing FlexNet inventory agent on UNIX in the Gathering FlexNet Inventory PDF, available through the title page of online help. As documented there, you need to uninstall the agent from the default location, and once again use your third-party deployment method if you need to reinstall the FlexNet inventory agent in a custom path on your UNIX-like device (and you may choose any version, including 2018 R3, as suits your corporate policies).

To restore an earlier version of FlexNet inventory agent

Once again, may we suggest that one possibility is to accept the 2018 R2 FlexNet inventory agent where it has been upgraded, as it is backward compatible, works in your existing infrastructure, and provides the latest functionality. If that is not acceptable, use this process both to recover an orphaned inventory device, or to return an inventory device that is successfully operating with FlexNet inventory agent 2018 R2 to an earlier release (where that is required by your company policies).

In the web interface for FlexNet Manager Suite, navigate to the system menu (in the top right corner), choose Data Inputs, and select the Inventory Data tab.
Click Download inventory agent to expose the appropriate page where you can collect your preferred version of the FlexNet inventory agent. For more information about the process, including how to configure your installation, refer to chapter 4, Agent third-party deployment: Details, in the Gathering FlexNet Inventory PDF, available through the title page of online help.
Use your preferred deployment technology to uninstall FlexNet inventory agent 2018 R2 from each target device (see the topics about uninstallation for the various platforms in the previous reference).
Also use that deployment technology to install your preferred version of FlexNet inventory agent that you downloaded and prepared under step 2.

The installation process is a standard installation of your preferred version, as in the documentation. When you have completed the process of removing the 2018 R2 version that you do not want, and installing the earlier version that you do want, there is no further requirement to advise Flexera of your preference or ask for any database intervention to set your preferred version.

Resolution

Flexera is responding to this incident in the following ways.

Modified code

The code framework used for updates to the cloud infrastructure prior to this incident contained a setting to allow for FlexNet inventory agent self-updates to the latest available version. This area of the code also inserts any new versions of FlexNet inventory agent into the list of available versions from which you may select your choice for deployment. When version 13.1.1 of FlexNet inventory agent was (correctly) added to this list of available agents from all cloud customers, the switch allowing automatic upgrades was inadvertently and incorrectly turned on.

Change: The ability to trigger global automatic updates has been completely removed from this tool. It is no longer possible to repeat this error in the current code framework. This does not affect our ability to enable, for your enterprise only, automatic upgrades to a version you identify, at your individual request.

Modified support

To provide assurance following this incident, Flexera is restoring official support for Microsoft Windows Server 2003 SP2 as a platform for FlexNet inventory agent, extending official support life for current installations on this platform. An updated version of the Release Notes including this support will be available in late October.

Impact assessment

The processes summarized in the Steps to Reproduce section allow you to locally validate the impact of this incident for your environment.

In addition, Flexera continues to analyze the central databases to assess the numbers of inventory devices for each customer that appear to be impacted by this incident. Your Customer Success Manager (CSM) / Strategic Success Manager (SSM) can provide you with some numerical insights:

Your total number of inventory devices that have FlexNet inventory agent installed
The number of these that have not attempted to self-upgrade, and continue to operate with the previously-installed versions
The number where an attempted self-upgrade failed, and the previously-installed versions of FlexNet inventory agent continue to operate
The number that report having successfully self-upgraded, and now have version 13.1.1 of FlexNet inventory agent reporting inventory
The number that have not reported since this incident, which fall into three possible classes: those whose reporting schedule means that are not yet due to upload inventory (for example, those that were on a weekly schedule); those that were decommissioned (or traveling, etc) in the last six weeks or so; and those few where the self-upgrade process may have orphaned the device.

Note that it is not currently possible to centrally reset the individual versions of FlexNet inventory agent for all your inventory devices, nor to centrally trigger downgrades. Please review the Steps to Reproduce section for guidance about local remediation.

Once you have restored your preferred version of FlexNet inventory agent (or accepted the 2018 R2 upgrade, if that suits your policies), normal operations are restored. Thereafter, once normal operations are restored and at any future time, you can (as always) request that an individual setting to upgrade to a particular version of FlexNet inventory agent be recorded for you in the FlexNet Manager Suite database. That setting and version then persist until you request a subsequent changed setting.

If we can be of further assistance in this matter, please contact Flexera Support or your Customer Success Manager (CSM) / Strategic Success Manager (SSM).

Additional Information

To see current and historic incidents affecting FlexNet Manager Suite Cloud please see the Flexera System Status Dashboard at https://status.flexera.com/