Code Insight Custom Detector Framework

Yes

1. What is the Custom Detector Framework

A generic framework within the Codeinsight that allows users to implement their own custom file content parsing logic and drop it into Code Insight for automatic creation of inventory to aid in building of the Software Bill of Materials (SBOM)
Supports both standard (XML, JSON, YAML etc.) and non-standard manifest file types
Enables automated discovery of components with license and vulnerability mappings
Creates automated inventory for top-level and direct and transitive dependencies
Processes the JSON rule configurations provided for each of the manifests

1.1 Users

The Custom Detector Framework can be used by any of the following groups to extend current detection functionality of Code Insight based on their own set of requirements .

Customers
Code Insight Service/Support teams
Code Insight Partners

1.2 Providing Custom Detector Rules to Code Insight

The users of the framework will provide JSON rules configuration files at the specified location in the Update-Service update folder at

<Codeinsight_installed_folder>/config/.codeaware/updates

For each of the manifest files, user must read the section “how to write manifest rules?” in order to create a new json rule configuration for that manifest.

1.3 Supported Manifest Type and Inventory Types

XML manifest rules for top-level and first-level dependencies
JSON manifest rules for top-level and first-level dependencies

2. How to Write Rules for XML Manifest Files

This section covers rules needed to identify both top-level and first level dependencies from an XML manifest file.

Definition of attributes used within the rule:

package-indicator: This attribute holds the value for manifest file extension or indicator file. Example: .csproj, .nuspec
package-type: this attribute defines the what type of package-manager a manifest file belongs to. Example: C-Sharp package, Conda package
format: the attribute holds the supported manifest format. Example: xml, json
packageRules: this element will have list of the rules, which define for top-level and first-level dependencies.
ruleType: this element holds either the values top-level or first-level
packageName: this attribute holds the name of the package name attribute in the manifest file. Also, this value should be given starting from the hierarchy. Example:

"packageName":

{

"name": "Project.PropertyGroup.RootNamespace",

"valueType": "XMLElement"

}

Note: In case of XML manifest, the element name should follow hierarchies starting from the root element. In the above example, the root element of the XML file referred is “Project” and name is derived as “Project.PropertyGroup.RootNamespace”

version: this attribute holds the name of the version attribute in the manifest file. Example:

"version":

{

"name": "Project.PropertyGroup.ProductVersion",

"valueType": "XMLElement"

}

valueType: this attribute holds either XMLElement or XMLAttribute.
license: this attribute is optional and holds the name of the license attribute in the manifest rule. Example:

"license":

{

"name": "Project.PropertyGroup.License",

"valueType": "XMLElement"

}

2.1 XML File Example: CSharp Package Detection Using Custom Rules

The following CSharp Package example demonstrates how a custom detector can be created to process the 'csproj' manifest file type to create automated inventory. Code Insight currently uses this custom detector to create automated inventory for CSharp. Users can further extend the rule to catch additional inventory if needed.

2.1.1 Sample CSharp Package Using a 'csproj' Using the XML Format

<?xml version="1.0" encoding="utf-8"?>

<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>

<AppDesignerFolder>Properties</AppDesignerFolder>

<RootNamespace>Sample</RootNamespace>

<AssemblyName>Sample</AssemblyName>

<TargetFrameworkProfile>Client</TargetFrameworkProfile>

</PropertyGroup>

</ItemGroup>

</Project>

2.1.2 Sample XML Manifest Rule for CSharp Package 'csproj' File

The below rule identifies both the top-level and first level components.

{

"package-indicator": ".csproj",

"package-type" :"cSharp package",

"format": "xml",

"packageRules": [

{

"ruleType": "top-level",

"packageName":

{

"name": "Project.PropertyGroup.RootNamespace",

"valueType": "XMLElement"

}

,

"version":

{

"name": "Project.PropertyGroup.ProductVersion",

"valueType": "XMLElement"

}

},

{

"ruleType": "first-level",

"packageName":

{

"name": "Project.ItemGroup.PackageReference.Include",

"valueType": "XMLAttribute"

},

"version":

{

"name": "Project.ItemGroup.PackageReference.Version",

"valueType": "XMLAttribute"

}

},

{

"ruleType": "first-level",

"packageName":

{

"name": "Project.ItemGroup.Reference.Include",

"valueType": "XMLAttribute"

},

"version":

{

"name": "Project.ItemGroup.Reference.Version",

"valueType": "XMLAttribute"

}

]

}

3. How to Write Rules for JSON Manifest Files

This section covers the rules needed to identify both top-level components from a JSON manifest file.

Definition of attributes used within the rule:

package-indicator: This attribute holds the value for manifest file extension or indicator file. Example: Conda package index.json
package-type: this attribute defines the what type of package-manager a manifest file belongs to. Example: Conda package
format: the attribute holds the supported manifest format. Example: xml, json
packageRules: this element will have list of the rules, which define for both top-level and first-level dependencies.
ruleType: this element holds either the values top-level or first-level.
packageName: this attribute holds the name of the package name attribute in the manifest file. Also, this value should be given starting from the hierarchy. Example:

"packageName":

{

"name": "name",

"valueType": "JSONElement"

}

In case of JSON manifest file type, the element name should follow hierarchies starting from the root element. In the above example, the root element of the file referred is “Project” and the name is derived as “Project.PropertyGroup.RootNamespace”

version: this attribute holds the name of the version attribute in the manifest file. Example:

"version":

{

"name": "version",

"valueType": "JSONElement"

}

valueType: this attribute holds either XMLElement or XMLAttribute.
license: this attribute is optional and holds the name of the license attribute in the manifest rule. Example:

"license":

{

"name": "license",

"valueType": "JSONElement"

}

packageDependency: this attribute is used only for first-level rules, to specify dependency details. Example:

"packageDependency":

{

"name": "depends",

"valueType": "JSONStringArrayElement"

}

In the packageDependency attribute the valueType can be either of the below 2 types

JSONStringArrayElement – This is used when the dependencies are represented as Arrays in JSON: Example:

"depends": [

"terminado 0.3.3",

"tornado 4.1.0",

"traitlets 4.3.1"]

JSONMapElement – This is used when the dependencies are represented as Map in JSON: Example:

“dependencies”: {

"terminado": "0.3.3",

"tornado": "4.1.0",

"traitlets": "4.3.1"}

3.1 JSON File Example: Conda Package Detection Using Custom Rules

This section covers rules needed to identify both top-level and first level dependencies from a JSON manifest file.

3.1.2 Sample Conda Package 'index.json' File Which Follows JSON Format

{

"app_cli_opts": [

{

"args": "--port %s",

"default": "8080",

"name": "port",

"summary": "Server port ..."

}

],

"app_entry": "jupyter-notebook",

"app_type": "web",

"arch": "x86_64",

"build": "py27h3661c2b_2",

"build_number": 2,

"depends": [

"ipykernel",

"ipython_genutils",

"jinja2",

"jupyter_client",

"jupyter_core",

"nbconvert",

"nbformat",

"python >=2.7,<2.8.0a0",

"terminado >=0.3.3",

"tornado >=4",

"traitlets >=4.3"

],

"icon": "df7feebede9861a203b480c119e38b49.png",

"license": "BSD 3-clause",

"name": "notebook",

"platform": "linux",

"subdir": "linux-64",

"summary": "Jupyter Notebook",

"timestamp": 1506020374312,

"type": "app",

"version": "5.0.0"

}

3.1.3 Sample JSON Manifest Rule for Conda Package 'index.json' File

{

"package-indicator": "index.json",

"package-type" :"conda package",

"format": "json",

"packageRules": [

{

"ruleType": "top-level",

"packageName":

{

"name": "name",

"valueType": "JSONElement"

}

,

"version":

{

"name": "version",

"valueType": "JSONElement"

},

"license":

{

"name": "license",

"valueType": "JSONElement"

}

},

{

"ruleType": "first-level",

"packageDependency":

{

"name": "depends",

"valueType": "JSONStringArrayElement"

}

]

}

4. Version Resolution Support

As an initial support for semantic version in custom-detector-framework, NuGet package manager is considered
The support for semver will be added cumulatively by adding new package managers to the framework
Please do note that the semantic version resolution is not completely generic

The following rule structure must be added to support version resolution:

"dependencyVersionResolution": {

"type":"semver",

"componentRegistryURL":https://api.nuget.org/v3-flatcontainer/#componentName/index.json "responseAttribute": "versions"

}

Here, in the above structure:

“dependencyVersionResolution”: This is the main element for the version resolution.
“type”: This attribute is used to represent the type of version resolution. For semantic version resolution, the value will be “semver”.
“componentRegistryURL”: Every package manager has its own registry URL. Also we can get the component info and its available versions from the registry. This attribute expects a generic registry URL representation to get all the available versions for a component. In the above structure https://api.nuget.org/v3-flatcontainer/#componentName/index.json is the common registry URL for nuget packages to get all available versions of a component. Also please note the #componentName substitute parameter used. This parameter should be used for other package managers also. This parameter is basically a place-holder for component name.
“responseAttribute”: This attribute expects the response attribute of componentRegistryURL which has all the available versions. For example, in the above structure “versions” attribute in the response of https://api.nuget.org/v3-flatcontainer/#componentName/index.json has all the available versions.

The below is the complete rule for NuGet having semver support.

{

"package-indicator": ".nuspec",

"package-type": "Nuspec package",

"format": "xml",

"dependencyVersionResolution": {

"type": "semver",

"componentRegistryURL": "https://api.nuget.org/v3-flatcontainer/#componentName/index.json",

"responseAttribute": "versions"

},

"packageRules": [

{

"ruleType": "top-level",

"packageName": {

"name": "package.metadata.id",

"valueType": "XMLElement"

},

"version": {

"name": "package.metadata.version",

"valueType": "XMLElement"

}

},

{

"ruleType": "first-level",

"packageName": {

"name": "package.metadata.dependencies.dependency.id",

"valueType": "XMLAttribute"

},

"version": {

"name": "package.metadata.dependencies.dependency.version",

"valueType": "XMLAttribute"

}

},

{

"ruleType": "first-level",

"packageName": {

"name": "package.metadata.dependencies.group.dependency.id",

"valueType": "XMLAttribute"

},

"version": {

"name": "package.metadata.dependencies.group.dependency.version",

"valueType": "XMLAttribute"

}

]

}

Example of Nuget nuspec XML file is as follows,

<?xml version="1.0"?>

<id>NHibernate</id>

<authors>NHibernate community, Hibernate community</authors>

<owners>NHibernate community, Hibernate community</owners>

<licenseUrl>https://raw.github.com/nhibernate/nhibernate-core/master/lgpl.txt</licenseUrl>

<projectUrl>http://nhibernate.info</projectUrl>

<iconUrl>https://raw.github.com/nhibernate/nhibernate-core/master/logo/NHibernate-NuGet.png</iconUrl>

<requireLicenseAcceptance>false</requireLicenseAcceptance>

<description>NHibernate is a mature, open source object-relational mapper for the .NET framework. It is actively developed, fully featured and used in thousands of successful projects.</description>

<summary>NHibernate is a mature, open source object-relational mapper for the .NET framework. It is actively developed, fully featured and used in thousands of successful projects.</summary>

<tags>ORM, DataBase, DAL, ObjectRelationalMapping</tags>

</dependencies>

</metadata>

</package>

5. Other

The rule in packageRules must contain one top-level rule, otherwise the rules are not processed
If there are no top-level components identified by the framework using rules, no inventories will be generated for those manifest files