Known Issue: ADDM staging tool may fail to stage data and report the error: System.IO.IOException: Stream was too long (IOJ-2077955)

[#1]

Summary

The BMC Atrium Discovery and Dependency Mapping (ADDM) staging tool may fail to stage data and report an error like the following:

2019-09-20 10:43:13.727 : ERROR : *** ERROR: An error occurred when attempting to read data from ADDM or when trying to write data to SQL Server: System.IO.IOException: Stream was too long.

Symptoms & Cause

The following is an example of logging that can be seen associated with this error:

2019-09-20 9:40:35.557 : INFO : ======== NetworkInterface Nodes
2019-09-20 9:41:12.974 : INFO : Response from ADDM in 0 seconds.
2019-09-20 9:41:13.146 : INFO : Staging ADDM query to file './Files/NetworkInterface.xml' complete
2019-09-20 9:41:13.146 : INFO : Transferred 27997259 bytes of data from ADDM
2019-09-20 9:41:13.146 : INFO : Import completed in 37.573 seconds
2019-09-20 9:41:13.146 : INFO : ======== Package Nodes
2019-09-20 10:43:13.727 : ERROR : *** ERROR: An error occurred when attempting to read data from ADDM or when trying to write data to SQL Server: System.IO.IOException: Stream was too long.
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
   at ADDMSource.ADDMQueryResultStream..ctor(Stream stream)
   at ADDMSource.ADDMDataReader.HTTPStream(String protocol, String server, String username, String password, Int32 timeout, String query, Boolean httpsCheckServerCertificate)
   at ADDMSource.Program.Main(String[] args)
2019-09-20 10:43:13.742 : ERROR : *** ERROR: Failed to stage ADDM query to file './Files/DiscoveredPackages.xml'
2019-09-20 10:43:13.742 : INFO : Retrying query with 2 attempts remaining for this mapping.
2019-09-20 10:43:13.742 : INFO : Import completed in 3720.276 seconds
2019-09-20 10:43:13.742 : INFO : ======== Package Nodes

This error can occur when too much data is returned from a query used when extracting data from ADDM.

Workaround

Within the FnmpADDMSettings.xml file, the “Package Node” query (or any other queries that results in this error occurring) can be split into multiple parts. This will look like the following:

<!--
Package nodes
Raw installer evidence gathered by ADDM from the operating system
supported installer technologies. This query can take a very long
time to run and return a very large number of records.
-->
<mapping description="Package Nodes A-M">
	<query>
		<![CDATA[
		SEARCH Host
		STEP IN Host:HostedSoftware:InstalledSoftware:Package
		WHERE #:Host:Host.key MATCHES '^(?i)[a-m]' 
		SHOW
		#:InstalledSoftware:Package.name AS 'Name',
		#:InstalledSoftware:Package.version AS 'Version',
		#:InstalledSoftware:Package.vendor AS 'Vendor',
		#:Host:Host.key as 'HostKey'
		]]>
	</query>
	<table filename="DiscoveredPackages.xml" db-name="DiscoveredPackages_ci">
		<column src="Name" dest="Package" data-size="255" data-type="System.String"/>
		<column src="Version" dest="Version" data-size="255" data-type="System.String"/>
		<column src="Vendor" dest="Vendor" data-size="255" data-type="System.String"/>
		<column src="HostKey" dest="HostKey" data-size="64" data-type="System.String"/>
	</table>
</mapping>

<mapping description="Package Nodes N-Z,0-9">
	<query>
		<![CDATA[
		SEARCH Host
		STEP IN Host:HostedSoftware:InstalledSoftware:Package
		WHERE #:Host:Host.key MATCHES '^(?i)[n-z,0-9]' 
		SHOW
		#:InstalledSoftware:Package.name AS 'Name',
		#:InstalledSoftware:Package.version AS 'Version',
		#:InstalledSoftware:Package.vendor AS 'Vendor',
		#:Host:Host.key as 'HostKey'
		]]>
	</query>
	<table filename="DiscoveredPackages.xml" db-name="DiscoveredPackages_ci" append="true">
		<column src="Name" dest="Package" data-size="255" data-type="System.String"/>
		<column src="Version" dest="Version" data-size="255" data-type="System.String"/>
		<column src="Vendor" dest="Vendor" data-size="255" data-type="System.String"/>
		<column src="HostKey" dest="HostKey" data-size="64" data-type="System.String"/>
	</table>
</mapping>

<mapping description="Package Nodes NOT A-Z,0-9">
	<query>
	<![CDATA[
		SEARCH Host
		STEP IN Host:HostedSoftware:InstalledSoftware:Package
		WHERE #:Host:Host.key NOT MATCHES '^(?i)[a-z,0-9]' 
		SHOW
		#:InstalledSoftware:Package.name AS 'Name',
		#:InstalledSoftware:Package.version AS 'Version',
		#:InstalledSoftware:Package.vendor AS 'Vendor',
		#:Host:Host.key as 'HostKey'
		]]>
	</query>
	<table filename="DiscoveredPackages.xml" db-name="DiscoveredPackages_ci" append="true">
		<column src="Name" dest="Package" data-size="255" data-type="System.String"/>
		<column src="Version" dest="Version" data-size="255" data-type="System.String"/>
		<column src="Vendor" dest="Vendor" data-size="255" data-type="System.String"/>
		<column src="HostKey" dest="HostKey" data-size="64" data-type="System.String"/>
	</table>
</mapping>

Fix Details

The above example divides the Package Nodes query into 3 parts, each of which retrieve a portion of the overall data set to be returned. The specific queries to use may need some modifications depending on what the data looks like.

The split query would have the following components:

1. The MATCHES clause where it matches the specified letters(case insensitive) and numbers at the start of the Host.Key field.
2. The other very important part is the Package Nodes NOT A-Z,0-9 where it will catch everything that the first two queries didn’t capture. Having the NOT MATCHES query at last is very important as it will catch any Host.Keys that start with symbols, punctuation or other non-alphanumeric characters.
3. The final important part is the append="true" attribute on all but the very first query of a particular type; failure to use that attribute correctly will result in the table being cleared out when that queries data is imported.

We may need to break this up into multiple queries depending on data size and specific requirements. This can be done by adding more queries and updating the MATCHES clause to narrow the matching criteria further. For example, the first query will just work on [A-M], then the next work on [N-Z], same can be done with numbers [0-1], [1-2].. etc.

Fix status

There are no plans to address this issue.

Other information

Affected components: Integration: BMC Discovery/ADDM

Master issue ID: IOJ-2077955

Also known as: ITAM-638