Code Insight MIT License Data Cleanup Project

Yes

Background

There are two licenses in Code Insight for MIT – MIT License and MIT-Style License. While most licenses declared by open source developers fall into the MIT License, the MIT-Style License is more of a template license consisting of various ways of how MIT license can be declared.

We noticed that the license mapping to majority of components are mapped incorrectly to the MIT-Style License. This is being resolved via an electronic update where the mappings are corrected and for existing projects that need mappings change a script will be provided.

Problem Details

There are two issues we are addressing as part of this MIT License data cleanup project:

Example: acorn 6.2.0 (MIT)

Here MIT is the license with the short name associated with the component acorn.

1. Short Name Change

When a particular license short name is changed and released as part of an electronic update, the short name is not automatically propagated to the inventory items with that selected license. For example, when we change the short name of license id 744 from "MIT License” to “MIT-Style” in an electronic update, the existing inventory items names with that selected license will not be updated.

2. Component to License Mapping Change

When the component to license mapping is changed, let’s say acorn is mapped with "Apache-2.0" in the electronic update, then this new mapping wouldn’t be propagated to existing inventory items. This results in inconsistency between the license mapping, existing inventory items, and future inventory items using the new license mapping.

Solution

1. Solution for Short Name Change

We need to update the names of existing inventory items with impacted selected licenses to include the new short name.

Example:

Before update – scalaz (MIT License)
After update without solution – scalaz (MIT License)
After update with solution – scalaz (MIT-Style)

2. Solution for Component to License Mapping Changes

We need to update the selected license of existing inventory items with impacted licenses per the new component to license mappings.

Example:

Component jquery is remapped from MIT License (ID: 744) to MIT (ID: 7)
Before update – jquery (MIT License)
o Selected License: MIT License (ID: 744)
o Possible Licenses: MIT License (ID: 744)
After update without solution – jquery (MIT License)
o Selected License: MIT (ID: 744)
o Possible Licenses: MIT (ID: 7)
After update with solution – jquery (MIT)
o Selected License: MIT (ID: 7)
o Possible Licenses: MIT (ID: 7)

Solution for customers taking MIT PDL Update prior to Code Insight 2021 R4 release (i.e. 2021R3, 2021R2 etc.,):

Download the MIT Cleanup script package from PLC with the name – MITCleanupPackage.zip. This zip file has two files “README.txt” and “MIT-CleanupQueries.sql”
Immediately after running the MIT Electronic update, customers should run the MIT Clean up script – “MIT-CleanupQueries.sql”, to ensure that the latest mappings are reflected in the already scanned projects. Please refer to the ‘Important Notes’ section at the bottom of this article to understand the impact, if the script is not run immediately after the electronic update is run

Solution for customers taking MIT PDL Update after Code Insight 2021 R4 release:

No Action needed for customers who are in 2021 R4. Product solution delivered as part of 2021 R4 takes care of the remappings on the already scanned projects.
In case of import scenario's respective to older projects which are exported before PDL Update, the customer needs to run the Script after importing the project. Steps and prerequisites are mentioned in the README.txt shared as part of MIT Cleanup script package.

*Tables impacted by the queries:

PAS_REPOSITORY_ITEM
PSE_INVENTORY_GROUPS

Solution for customers taking MIT PDL Update in Code Insight v6:

No action required for customers using Code Insight v6. A solution was delivered as part of an electronic update. This solution contains a groovy script that executes the required queries to handle the re-mapping of already scanned projects.
In case of import scenario's respective to older projects which are exported before PDL Update, the customer needs to run the script after importing the project.

*Tables impacted by the queries:

PAS_REQUEST_INSTANCE
PAS_POLICY
PSE_GROUPS
PSE_GROUP_LICENSES

ACTION REQUIRED:

For customers taking the electronic update with the MIT License data cleanup after installing the Code Insight 2021 R4 release:

Step 1: Take a complete old database backup.
Step 2: Apply the electronic update with the MIT License data cleanup.
No further action is needed.

For customers taking the electronic update with the MIT License data cleanup before installing the Code Insight 2021 R4 release:

Step 1: Take a complete old database backup.
Step 2: Apply the electronic update with the MIT License data cleanup.
Step 3: Immediately after the electronic update completes, and before any other operations are performed (scan, import, etc.), run the provided SQL script.

For customers taking the electronic update with the MIT License data cleanup in Code Insight v6:

Step 1: Take a complete backup of the database before applying electronic update.
Step 2: Apply the electronic update with the MIT License data cleanup.
No further action is needed.

Project Import Scenarios in Code Insight v7:

To import the old project data (exported before the MIT License data cleanup electronic update was processed), into a project after the MIT License data cleanup electronic update was run, follow the steps below to avoid inconsistencies in the project inventories:

Step 1: Import the old project export JSON file into the target project.
Step 2: Run the provided SQL script.
Step 3: Select "On data import or rescan, delete inventory with no associated files" option from Summary Screen -> Manage Project -> Edit Project -> Under General Tab.
Step 4: Upload the project codebase and schedule the scan.

Project Import Scenarios in Code Insight v6:

To import the old project data (exported before the MIT License data cleanup electronic update was processed), into a project after the MIT License data cleanup electronic update was run, follow the steps below to avoid inconsistencies in the project inventories:

Step 1: Import the old project export XML file into the target project.
Step 2: Run the SQL script present in the electronic update package:
<CodeInsight_InstallFolder>/tomcat/temp/palamida_update/scripts/sql
(In case the palamida_update folder is cleaned up in the above-mentioned location, please download the scripts from PLC.)

For MySQL, execute mit_license_remap_mysql.sql
For Oracle, execute mit_license_remap_oracle.sql
For SqlServer, execute mit_license_remap_sqlserver.sql

NOTE: Projects which are exported after the MIT License data cleanup electronic update do not require the SQL script to be run.

IMPORTANT NOTES:

Users must run the script “MIT-CleanupQueries.sql” after the PDL update is run and before initiating any scans. In case any scans are triggered before running the “MIT-CleanupQueries.sql” script on the database, the below issues arise and are explained in detail. Also, the issues do not impact any manual inventory created or any inventory created by scan and updated by users.

Short Name change per the electronic update is not reflected for existing inventory items. Meaning, instead of jquery (MIT-Style), inventory name is still retained as jquery (MIT License)
Component-License remapping would not be performed on existing inventories from license ID 744 (MIT License) to license ID 7 (MIT)

Example: acorn (MIT License) 744

If we perform a full rescan of the project, then we may end with duplicate inventory items with the two license short name variants:

acorn (MIT License) – old inventory item

acorn (MIT) – new inventory item

Version Fix Target

The electronic update having the MIT License data mapping changes is planned for January 27, 2022.