FlexNet Code Insight Group Builder Script - Add-On

[#1]

Summary

A python project to gather component/license information from various package managers (e.g., npm, rubygems, rpms, etc.) and parse into groups that are importable via the import workspace XML data script.

Synopsis

A python project to gather component/license information from various package managers (e.g., npm, rubygems, rpms, etc.) and parse into groups that are importable via the import workspace XML data script for FlexNet Code Insight.

Download: Contact Technical Support
Password: 000022428

• Switch to checking for "repository" field for npm urls first (for Github component matching).
• Fix bug for .gem files.
• Fix typo in gemsource documentation.

Usage

./GroupBuilder.py [options]

Run with -h for options.

Discussion

Input

1. For Node Modules:
   • A text file with a list of files paths containing Node Modules (run with -f <file>).
   • Or a text file with a list of Node Modules (run with -p <file>).
2. For Ruby Gems:
   • A text file with a list of files paths containing .GEM files (run with -f <file> -t gems).
   • Or a text file (such as a gemfile) with a list of gems (run with -p <file> -t gems).
   • For Ruby Gems source code, use like npm with a text file of file paths (run with -f <file> -t gemsource)
3. For RPMs:
   • A text file with a list of file paths containing RPMs (run with -f <file> -t rpms).
4. For Composer (PHP) packages:
   • A text file with a list of files paths containing composer.json files (run with -f <file> -t php).
   • A composer.lock file with package JSON data. (run with -f <composer.lock> -t php).
5. For BitBake (bb) files:
   • A text file with a list of files paths containing .bb files (run with -f <file> -t bitbake).
6. For CSV file:
   • A csv (comma-separated) file with a list of groups having data in the following order: Group/Package Type, Name, Version, License, Description, URL, filepath, Component ID, and ComponentVersion ID

Requirements

1. For Node Modules:
   • An installation of npm from NodeJS (http://nodejs.org/).
   • Or local access to the package.json files.
2. For Ruby Gems:
   • Internet access.
   • The Requests python package.
3. For RPMs:
   • The rpm command (via Cygwin or your Linux package manager)
   • And local access to the rpm files.
4. For Composer (PHP) packages:
   • A composer.lock file with package information in JSON format.
   • Or local access to composer.json files.
5. For BitBake:
   • Local access to .bb files.
6. For CSV:
   • All groups must have a name, at least.

For associating groups to components, either the components.txt file (which contains component information for the core database) or a JSON with credentials to use a Palamida MySQL database. Using a MySQL connection will require the mysql.connector package (https://dev.mysql.com/downloads/connector/python/2.1.html). Please take care when using database connections. Always make a verified backup before making direct database connections. This script only queries and does not write to the Palamida database.

For all: Python 2.7.x. The lxml package is necessary to pretty-print the XML output.

Output

An XML file that's importable by the Palamida import/export script

For versions of the import/export script 3.2 and AFTER:

scriptRunner.bat <PATH_TO>importWorkspaceData.groovy --server <your core server url> --scan_server <your scan server url> --input groups.xml --workspace foo-workspace

For versions of the import/export script BEFORE 3.2:

scriptRunner.bat <PATH_TO>importWorkspaceData.groovy -input workspaceData.xml -workspace foo-workspace -check_md5_hash

If you give the script file paths, then it will try to associate the resulting groups to files and build that into the XML file. If you do not have file paths, you have the option to associate all of the resulting groups to a single file, otherwise when you import the groups they will not be attached to any files.

Caveats

Please bear in mind the following caveats:

• The information is not necessarily perfect. The developer might have been lazy or stupid and incorrectly filled out the metadata in the file (or wherever). We recommend doing a quick check against the actual licenses.
• The information is not necessarily complete. Some node modules will have ?Unknown License? given because the information was not available in the metadata. These projects likely have licenses in their source repositories (Github, etc) or other places. I recommend looking at each of these manually.
• The information does not give you P1 bundle issues. Each of these files must be checked for P1 issues as usual before being marked as reviewed (P1 search terms and P1 license matches). A major benefit of these scripts is that we have more time to be thorough looking for P1 stuff.
• This is an alpha release that is completely independent of the main Palamida product and is provided 'as is' as an add-on for convenience. It likely contains bugs and likely won't handle every special case. We welcome feedback and suggestions (fscott@palamida.com).

If you see any P6 groups with obvious know licenses, please let us know what the license text was so I can add it. License priorities can be adjusted in the Group.py file.

Additional Information

The current version is 1.9 (5-17-2016)

• Switch to checking for "repository" field for npm urls first (for Github component matching)
• Fix bug for .gem files.
• Fix typo in gemsource documentation.

Version 1.8 (5-13-2016)

• Better component matching. Attempts to find Github component first given a Github URL before doing a broader URL search. (Thanks Ed!)
• Groups will still be created when you don't have access to the npm registry.
• Clearer logging when checking npm registry.

Version 1.7 (4-6-2016)

• Now compatible with import/export scripts v3.2 (for Palamida 6.8). NOTE CHANGE IN OUTPUT ABOVE.
• Support for gem source writeups. Some day will read gemspecs directly.

Version 1.6 (3-6-2016)

• New components based on latest electronic update.
• CID and CVID for csv groupbuilding.
• Gem source code is now supported.
• Fixed typo in Ruby Gem group write-up (Thanks Ed!)
• Group.py will now try to associate a group to a component by an exact name match as a last gasp attempt. Warning that this may lead to more false postives. Feedback is welcome here.
• RPM bug fixes and improvements.

Version 1.5 (1-4-2016)

• Various bug fixes and improvements.

Version 1.4 (12-21-2015).

• The lxml is now an optional dependency. It's primarily used to pretty-print XML.

Version 1.3 (12-06-2015).

• Faster node module processing and better console info.
• Option to use connection to Palamida MySQL database.
• Selects available version ID as well as component, if available (requires DB connection)
• Now supports BitBake and CSV files.

Future Work

• More documentation
• Options for formatting output.
• Handling more special cases.

Copyright

Copyright (C) 2015-2016 Palamida Inc. All rights reserved.

This software is the confidential and proprietary information of Palamida Inc. and shall not be used, disclosed or reproduced, in whole or in part, for any purpose, without the prior written consent of Palamida Inc.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.