Excessive growth of unrecognized file evidences
in one of our Systems, we discovered a growing number of unrecognzied file evidences (>9 Million entries). We are using the FNM Agent on all platforms. At the Moment, we do not exclude any folders in the inventory process. I am pretty sure that this causes the raising number of evidences.
Do you have experiences & best practices which folders should be excluded on Windows Clients, Windows Servers, UNIX & Linux Servers? We will start with excluding the Windows Directory and the recycle bin...
We experienced similar problem and after a review decided to exclude some paths from inventory collection.
I cannot confirm if this is 100% safe to exclude those path, but in our case those were not useful in any license calculations.
All those folders contained were some details regarding updates:
This folder also contained some useless files.
Only these 4 added to 50% of total files evidences we were importing.
For Linux I've also excluded /var/lib/docker/ because it was growing rapidly.
as JanPie highlighted,
For our clients, we do this as an onboarding process to identify folder on both wintel and non-wintel systems which will have 'junk'.
we identify directories, Backup folders, install folder (manual install and or for tools like SCCM)
hope this helps
Keep in mind that deleting old not reporting and ignored inventory from FNMS Inventory database also helps to reduce number of records in underlying evidence related tables.
One environment I was working in found it useful to exclude the /var/spool and /var/log paths on UNIX computers to reduce the size of data collection.
However similar to @JanPie's comments, ultimately what it is useful to exclude will best be driven by your particular data. I've not found that much similarity across different companies in relation to directories that are helpful to exclude.
It was some time since I have performed analysis. But if I remember correctly I used this query to find paths with most evidences:
SELECT TOP 1000 * FROM (SELECT SFP.Path, COUNT(*) [cnt]
FROM [FNMSInventory].[dbo].[SoftwareFile_MT] SF
LEFT JOIN [FNMSInventory].[dbo].[SoftwareFilePath] SFP ON SFP.SoftwareFilePathID = SF.SoftwareFilePathID
GROUP BY SFP.Path) C
ORDER BY C.cnt DESC
If you use on premise FNMS and have access to DB you can run it to get paths from your environment.
Then I used another SQL to validate if evidences from selected paths trigger any commercial products recognition. We don’t track usage so I haven’t checked impact on usage tracking.
On the Compliance database, one of the scripts I've found is:
-- File Counting SQL Script for Compliance DB. -- Returns a count of the number of rows in the DB for a particular OS DECLARE @OSType INT -- -- Configurable options -- Use 1 for Windows and 2 for Unix / OSX SET @OSType = 2 -- End of configurable options -- DECLARE @FirstChar VARCHAR(3), @SecondChar VARCHAR(1) IF (@OSType = 1) BEGIN SET @FirstChar = '%:\' SET @SecondChar = '\' END ELSE IF (@OSType = 2) BEGIN SET @FirstChar = '/' SET @SecondChar = @FirstChar END SELECT SUBSTRING(IFE.FilePath, 1, CHARINDEX(@SecondChar, IFE.FilePath, LEN(@FirstChar) + 1)) AS 'Directory', COUNT(IFE.FilePath) AS 'Count' FROM ImportedFileEvidence AS IFE WHERE LEN(IFE.FilePath) <> LEN(@FirstChar) AND LEFT(IFE.FilePath, LEN(@FirstChar)) LIKE @FirstChar AND IFE.FileName NOT IN ( SELECT FE.FileName FROM FileEvidence AS FE INNER JOIN SoftwareTitleFileEvidence AS STFE ON FE.FileEvidenceID = STFE.FileEvidenceID ) GROUP BY SUBSTRING(IFE.FilePath, 1, CHARINDEX(@SecondChar, IFE.FilePath, LEN(@FirstChar) + 1)) ORDER BY 'Count' DESC
The above will show the most common UNIX file paths containing file evidence that is not linked to an application hence will not be used by the ARL.
If the solution provided has helped, please mark it as such as this helps everyone to know what works.