Excessive growth of unrecognized file evidences

[christian_ludwig]

Dear Community,

in one of our Systems, we discovered a growing number of unrecognzied file evidences (>9 Million entries). We are using the FNM Agent on all platforms. At the Moment, we do not exclude any folders in the inventory process. I am pretty sure that this causes the raising number of evidences.

Do you have experiences & best practices which folders should be excluded on Windows Clients, Windows Servers, UNIX & Linux Servers? We will start with excluding the Windows Directory and the recycle bin...

Best, Christian

@mfranz 

[JanPie]

Hello Christian,

We experienced similar problem and after a review decided to exclude some paths from inventory collection.

I cannot confirm if this is 100% safe to exclude those path, but in our case those were not useful in any license calculations.

All those folders contained were some details regarding updates:

%:\WSUS\

%:\WSUSDATA\

%:\WSUSdownloadedPatches\

This folder also contained some useless files.

%:\SCCMContentLib\FileLib

Only these 4 added to 50% of total files evidences we were importing.

For Linux I've also excluded /var/lib/docker/ because it was growing rapidly.

[wernerb]

as JanPie highlighted,

For our clients, we do this as an onboarding process to identify folder on both wintel and non-wintel systems which will have 'junk'.

we identify directories, Backup folders, install folder (manual install and or for tools like SCCM)

hope this helps

[marius]

Hi,

Keep in mind that deleting old not reporting and ignored inventory from FNMS Inventory database also helps to reduce number of records in underlying evidence related tables.

Marius

[ChrisG]

One environment I was working in found it useful to exclude the /var/spool and /var/log paths on UNIX computers to reduce the size of data collection.

However similar to @JanPie's comments, ultimately what it is useful to exclude will best be driven by your particular data. I've not found that much similarity across different companies in relation to directories that are helpful to exclude.

[marius]

Hi,

It was some time since I have performed analysis. But if I remember correctly I used this query to find paths with most evidences:

  SELECT TOP 1000 * FROM (SELECT SFP.Path, COUNT(*) [cnt]

  FROM [FNMSInventory].[dbo].[SoftwareFile_MT] SF

  LEFT JOIN [FNMSInventory].[dbo].[SoftwareFilePath] SFP ON SFP.SoftwareFilePathID = SF.SoftwareFilePathID

  GROUP BY SFP.Path) C

  ORDER BY C.cnt DESC

If you use on premise FNMS and have access to DB you can run it to get paths from your environment.

Then I used another SQL to validate if evidences from selected paths trigger any commercial products recognition. We don’t track usage so I haven’t checked impact on usage tracking.

Marius

[mrichardson]

On the Compliance database, one of the scripts I've found is:

-- File Counting SQL Script for Compliance DB.
-- Returns a count of the number of rows in the DB for a particular OS
DECLARE @OSType INT
--
-- Configurable options
-- Use 1 for Windows and 2 for Unix / OSX
SET @OSType = 2
-- End of configurable options
--
DECLARE @FirstChar VARCHAR(3), @SecondChar VARCHAR(1)
IF (@OSType = 1)
BEGIN
    SET @FirstChar = '%:\'
    SET @SecondChar = '\'
END
ELSE IF (@OSType = 2)
BEGIN
    SET @FirstChar = '/'
    SET @SecondChar = @FirstChar
END
SELECT SUBSTRING(IFE.FilePath, 1, CHARINDEX(@SecondChar, IFE.FilePath, LEN(@FirstChar) + 1)) AS 'Directory',
    COUNT(IFE.FilePath) AS 'Count'
FROM ImportedFileEvidence AS IFE
WHERE LEN(IFE.FilePath) <> LEN(@FirstChar)
    AND LEFT(IFE.FilePath, LEN(@FirstChar)) LIKE @FirstChar
    AND IFE.FileName NOT IN (
        SELECT FE.FileName
        FROM FileEvidence AS FE
        INNER JOIN SoftwareTitleFileEvidence AS STFE
            ON FE.FileEvidenceID = STFE.FileEvidenceID
    )
GROUP BY SUBSTRING(IFE.FilePath, 1, CHARINDEX(@SecondChar, IFE.FilePath, LEN(@FirstChar) + 1))
ORDER BY 'Count' DESC

The above will show the most common UNIX file paths containing file evidence that is not linked to an application hence will not be used by the ARL.