Want to learn more about Code Insight and Open Source but don't know where to begin? Check out these resources:
Software Composition Analysis Webinars
Software Composition Analysis YouTube channel
Online Documentation & Release Notes
Online Legal Certification Program
Ready to take learning to the next level? Contact our Education Specialists here.
Scanning Guidelines and Best Practices
FlexNet Code Insight v6
There are many variables that can affect the success of running a source code scan in FlexNet Code Insight v6, some examples include: hardware and software limitations, the size and type of your codebase and your scan settings. This document outlines best practices and recommendations to increase the likelihood of running a scan successfully to completion.
Workspace Maximum Recommended Limits
Breaking up your codebase into workspaces allows you to scan smaller logical groups with independent scan settings, while providing a solution for creating a single analysis across all files scanned within the project. Please keep the following limits in mind when creating workspaces.
3 Million SLOC and 150 Mb of source files
With SCF scanning enabled:
2 to 3 Million Source Lines of Code (SLOC)
150MB of source
If you have more than 150MB of source or more than 2 to 3 million SLOC, you will want to create another workspace
With SCF disabled:
No source limitations
Total Files Per Workspace:
100,000 files per workspace
If you have more than 80,000 to 100,000 files, we recommend creating another workspace. We have seen up to 150K files successfully scan but in practice try to limit workspace maximum files counts to less than 100K files.
1000 Jars per workspace
If you have more than 1000 jars, we recommend creating another workspace. We have seen up to 2500 jars scan successfully, but don’t recommend it. Other factors can include Jars bundled inside other Jars. You can also increase the number of jars per workspace by disabling namespace matching.
All limits should consider both files on disk plus files contained in archives if you have scanning files within archives enabled.
The primary features that affects the time it takes to complete a scan can be found in Workspace Settings > Detection tab (see the Configuring Workspaces tab in the Audit and Analysis Guide for details).
Figure 1: Detection Tab in Workspace Settings
The following to scan options can significantly affect scan times:
Source Code Fingerprint Scanning (SCF)
Archive Scanning options
Source Code Fingerprint Scanning (SCF)
SCF scanning allows you to detect and review snippet matches within source files. This option should be used in cases where you want to detect the modification of open source software (OSS) source code or where developers incorporate fragments of OSS source code within their code. Because this is a very rigorous and time-consuming scanning event, we recommend you turn off SCF scanning if you are not modifying OSS nor use code fragments.
If you do require SCF scanning, we recommend that you configure the Min Match setting on the Source Code Options tab. The Min Match setting sets the minimum number of snippet matches that are required to match between the codebase and the data library file before the result can be considered a match. The default minimum snippet match setting is 3—that is, 3 snippets must match between the source and library file for the file to be considered a source match and to appear in the results. Our in-house testing shows that you can typically increase the Min Match count to a value of 10 to improve scan times without adversely affecting the scan results. With a Min Match value of 10, only files with 10 snippet matches between the scanned codebase and data library file will appear in the scan results. Files with fewer matches will not.
Figure 2: Source Code Options in Workspace Settings
The default setting is to not scan files in archives. We generally recommend that you do not enable this option as it can quickly expand the scan set beyond maximums and cause scan failures. Often many archives contain test files or sample files and do not contain 3 rd -party components and consequently don’t require scanning. We recommend that you expand only the archive types that may contain 3 rd -party components prior to scanning.
If you prefer not to scan inside archives but would still like to see that contents of archive files in your codebase tree, you may do so by setting the displayContentsOfUnscannedArchives=true property in <FNCI_INSTALLATION_DIR>/config/core.properties and restarting the server. Although this will not save you the same amount of scan time as keeping the archives setting off completely, it will still favorably impact your scan time.
For workspaces that contain a lot of jar files, scanning with namespaces turned on may significantly impact scan time. For this reason, we recommend keeping Namespace Matching turned off (it is off by default) and using other techniques, such as Analyzer Group Builder to identify jar packages.
Flexera has a code estimator tool you can use to determine code metrics, including: estimated SLOC, megabytes of source, number of jar files, and total number of files. Please contact your Flexera Account Representative to obtain a copy of the estimator tool.
Logical Workspace Groupings
WinDirStat on Windows and KDirStat or QDirStat on Linux, are disk usage statistics viewers that let you view both the directory tree as well as a visual representations of your codebase. We recommend using these utilities to help you determine the best approach to breaking your code into smaller logical groups. For example, breaking out node modules, gems or jars into separate groups can be one way you can effectively divide your code to facilitate scanning by workspace and tools like WinDirStat, make identifying these groups much easier.
FlexNet Code Insight in a Disconnected (Air-Gapped) Environment
FlexNet Code Insight does not explicitly rely on inbound or outbound internet access, making it suitable for deployment in a disconnected or air-gapped environment without a major loss in functionality. Data that typically requires an internet connection is supplied to the product in alternative ways (i.e. shipped with the product or offered as a local update package).
External Data Dependencies
The FlexNet Code Insight default deployment makes use of an internet connection in one of the following ways:
Electronic Update: Code Insight downloads an update package from the Flexera Update server on a nightly basis or according to a custom schedule. The package contains information about open source projects found in the data library, their associated vulnerabilities and detection rules. In offline mode, the data is available in a local update package to be applied manually.
Security Vulnerability Data Signature: The Code Insight automation module checks for, and downloads new vulnerability data from the National Vulnerability Database (NVD) at scan time and performs a full refresh of the data weekly. In offline mode, the product accesses security vulnerability data provided by Electronic Update.
License Information: The Code Insight automation module updates license information obtained from various sites (GitHub, Maven Central, Bower, etc.) during scan time. In offline mode, the license information is obtained from Electronic Update data and from license detection capability of the Scanner.
Artifact Dependencies: The Code Insight automation module makes a call out to repositories (Maven and NPM) for artifact dependencies and version resolution at scan time. In offline mode, some of this data is obtained from Electronic Update rules or detected using the automation module. Some artifact dependency information (i.e. transitive dependencies and version resolution) may not be available in offline mode.
Remote File Data: Code Insight queries an Amazon S3 server to obtain remote file path information and remote file contents for Exact and Source Matches files during the (optional) deep analysis phase conducted by an analyst. In offline mode, remote file data is not available and dual-pane analysis of remote data is disabled.
The following table provides a detailed summary of external data dependencies, their data flow and potential impact to functionality in an air-gapped environment.
Manifest file and zip file with OSS project info and detection rules
Recommendation: Configure FNCI to read from a local update package. Obtain and apply the update regularly (at least weekly).
Impact: Manual process must be used on a regular basis (currently weekly but could be more frequent in the future) to download the local update package and apply it to the FNCI database. Otherwise no impact to product or data.
Security Vulnerability NVD Sync
Data signatures for security vulnerabilities
Recommendation: Run a local Electronic Update regularly (at least weekly).
Impact: The very latest security vulnerabilities available from NVD will be missed out, otherwise the majority of security vulnerability data will be available via Electronic Update. This will be mitigated somewhat with more frequent electronic updates planned for future FNCI releases.
Recommendation: Run a local Electronic Update regularly (at least weekly).
Impact: Minimal impact as the majority of license data is pre-indexed and shipped with the product or supplied via Electronic Update. The Scanner also has built-in license detection capability at the file level that provides license information.
Dependency and version information
Recommendation: Run a local Electronic Update regularly (at least weekly).
Impact: Transitive dependencies are not available in offline mode. Some primary dependencies are also affected. For non-Mavenized jars, artifacts may be missed if there are no existing detection rules provided by Electronic Update. For NPM, versions will not be resolved if the version is an expression.
Remote File Data
Remote file ID
Remote file path & remote file contents
Impact: Remote file path listing and remote file path contents are not available in offline mode. Dual-pane comparison of codebase and remote matched file is not available. Otherwise, no impact to Exact and Source Match detection.
Most of the automated discovery capability, including Package Analysis and Component Identification, is available out-of-the box with your Code Insight installation in the form of an independent automation module or via Electronic Update. The automation module may be upgraded any time either by migrating to the latest version of Code Insight or by replacing the existing module with an updated module in your installation directory. Electronic Update may be configured to read from a local update and can be run manually on a regular basis.
The Compliance Library (CL), which provides data required for detection of Exact Matches and Source Code Fingerprint Matches for advanced analysis by an auditor, is provided on an external SSD drive with every Code Insight installation. An internet connection is not required for detection and highlighting of fingerprints in the codebase files.
Note: Direct access to files in the Compliance library (a.k.a “Remote File Access”) is not available in offline mode. It is not possible to view, download or use dual-pane side-by-side comparison with remote files.
Code Insight functionality that requires data flow and communication between servers (i.e.
Email Notifications, User Sync & Authentication, CI/CD plugins, ALM, SCM, etc.) is not impacted in an air-gapped environment as long as the systems are configured to run on the same internal network.
If after upgrading from one version of FNCI to another version, your SSO Single Sign-On appears to be failing with an HTTP Status 500 - Internal Server Error, confirm the following:
1) Confirm that the following files have been copied over from your older FNCI installation to your new installation:
2) Confirm that the myKeystore.jks (may be named differently on your server) file is located in the location specified by the saml.keystore property in core.sso.properties.
Summary This article discusses how to configure TFS for use with HTTPS in FlexNet Code Insight Synopsis Steps to resolve the issue TFS HTTPS(sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException)
1.Get TFS server certificate(HTTPS) and file look like <"XYZ.cer">. i.Go to server machine where tfs server is installed. ii.Go to IE browser settings: Select Internet option==> Got content tab. iii.Click on certificate ==> trusted root certificate autority ==> export the certificate with <hostname>.
2.Import <"XYZ.cer"> certificate to "\jre1.8.0\lib\security"(where CodeInsight application is installed) by using below command. keytool -keystore cacerts -importcert -file <"XYZ.cer"> -alias tfs
Note: After executing above command, give the password and Accept the trusted certificate by giving yes value. 3.Download and extract TFS client(add to Path environment variable). 4.Enable the tfs option,in "scm.properties" file. 5.Start the tomcat server with new command line session.
Below are the scenario covered. 1. Verified the option of "TFS Update To" .i.e "Latest Changeset","Specific Changeset" and "Label" ==> Working as expected. 2. Verified below URL format for TFS 2015(http and https): http://<server>:<port>/<tfsroot>/<collection>/<project> http://<server>/<tfsroot>/<collection>/<project> https://<server>:<port>/<tfsroot>/<collection>/<project> https://<server>/<tfsroot>/<collection>/<project> 3. Verified below URL format for TFS 2017(http and https): http://<server>/<collection>/<project> https://<server>/<collection>/<project> 4. Verified below URL format for TFS 2012 update1(http): http://<server>/<tfsroot>/<collection>/<project>
Covered below SCM: Perforce HTTP , Git , SVN HTTP
Summary This article documents the accessibility of Secunia research through FlexNet Code Insight following the closure of the Secunia Community site in February 2019 Synopsis The Secunia Community site will become inaccessible at the end of February. A future release of Code Insight will incorporate the following changes to ensure access to Secunia data:
Deliver additional Secunia Advisory properties (currently visible on the Secunia Community site) to Code Insight through the Electronic Update service. Provide a new Get Vulnerability Details REST API to obtain the additional Secunia Advisory data. Develop a new ?vulnerability details? interface to display additional Secunia Advisory data.
Meanwhile, if you want to temporarily disable Secunia Advisories from Code Insight, refer to the instructions below for your version of the product. Code Insight 6.13.0 (or later) Follow the steps below to disable Secunia Advisories from Code Insight v6 (JIRA: SCA-8639)
Update the following properties in /FNCI_ROOT/config/core/core.properties :
disable.secunia=true (value is false by default) enable.forceupdate=true (value is false by default)
Restart Code Insight Force an electronic update
Log in as an Administrator Navigate to Administration >> Updates Manually trigger an Electronic Update
Code Insight 2019 R1 (or later) Follow the steps below to disable Secunia Advisories from Code Insight v7 (JIRA: SCA-12114)
Execute the following database SQL insert statement using a database client of your choice:
INSERT INTO PAS_GLOBAL_PROPERTIES (SERVER_ID_, KEY_, VALUE_, ENCRYPTED_) VALUES (0, 'disable.secunia', 'true', 0);
Restart Code Insight Force an electronic update
Log in as an Administrator Navigate to Administration >> Electronic Update Check the Force Full Electronic Update option Click the Schedule Update button
Summary Palamida Analyzer - Quick Assessment and Group Builder fail Due to certificate issues. This issue has been identified as a bug under reference SCA-14890. This article contains a workaround to allow the reports to be generated Symptoms A bug has been identified in the Analyzer - Quick Assessment and Analyzer - Group Builder reports due to a certification issue. The following error can be observed in the catalina.out log file under <Install Dir>/tomcat/logs/: 2019-02-06 23:46:15,741 ERROR [http-apr-8888-exec-8] [AnalyzerTask] Error running Palamida Analyzer: class javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed at com.palamida.sdk.impl.PalamidaService.get(Unknown Source) at com.palamida.sdk.impl.PalamidaService.get(Unknown Source) at com.palamida.repoanalyzer.impl.DataServiceSource.testConnection(DataServiceSource.groovy:159) at com.palamida.repoanalyzer.WorkspaceProcessor.process(WorkspaceProcessor.groovy:201) at com.palamida.repoanalyzer.ReportTask.process(ReportTask.groovy:176) at palamida_analyzer_task.run(palamida_analyzer_task.groovy:75) Resolution A bug for this issue has been reported under reference SCA-14890. The following workaround has been made available to allow the reports to be used: - Rename analyzer.properties.example to analyzer.properties - Add following property in analyzer.properties-
dataServicesUrl = https://api.palamida.io
- Restart FNCI - Now the group builder and quick assessment reports from analyzer are working fine
Summary If your environment does not allow for automated updated, this article provides instructions for performing a manual update . Synopsis The FlexNet Code Insight Electronic Update service relies on outbound access on port 22. Some FlexNet Code Insight instances are configured without outbound internet access (or otherwise somehow block outbound access on port 22). In these instances, the users will need to perform a manual update to have the most up-to-date data in the core database.
Note: These instructions apply to Palamida 6.1.5, 6.6.2, and later. If you are running an older version of FlexNet Code Insight, please contact us to schedule an upgrade. Discussion
In a browser, connect to the following URL: http://updates.palamida.com/updates/3.1/
a. Use the following credentials:
Username: updates-user Password: P@l@m!d@3
b. Download the following files to your local client machine:
Log in as an Administrator to the Palamida WebUI.
Go to Administration > Updates in the web UI.
Click on the Manual Update tab (1). Select the files that you downloaded in Step 2. For the Update Manifest File field, upload update_manifest.txt (2) and for the Update Data File field, upload update.zip (3).
Click only once on Update (4) and wait for the confirmation dialog shown below. This may take a minute or two depending on the speed of your network connection, as the file will be uploaded from your computer to the Palamida server.
Check the Scheduler tab, choosing "Core Server" in the dropdown menu, to ensure the update runs and completes successfully. Once the update is no longer running, the status can be seen in the "Task History" tab of the Scheduler.
PLEASE SEE THE ADDITIONAL INFORMATION SECTION BELOW FOR INSTRUCTIONS ON THE MANUAL UPDATE PROCESS FOR THE 2017 R1+ (VERSION 7.X) PRODUCTS Additional Information The instructions for performing a manual update on FlexNet Code Insight 2017 R1 onwards are as follows: 1. Run following insert statement to force a local update rather than downloading from update server INSERT INTO PAS_GLOBAL_PROPERTIES (SERVER_ID_, KEY_, VALUE_, ENCRYPTED_) VALUES (0, 'update.local', 'true', 0); 2. Download update_manifest.txt and update.zip 3. In a browser, connect to the following URL: http://updates3.palamida.com/updates/vnext/ Use the following credentials: Username: updates-user Password: P@l@m!d@3 4. Rename update_manifest.txt to manifest.txt 5. Move manifest.txt to $FNCI/tomcat/temp/palamida_update 6. Unzip update.zip to $FNCI/tomcat/temp/palamida_update 7. Restart tomcat 8. Got to Administration page and click the Schedule Update button to start the update immediately
Summary A python project to gather component/license information from various package managers (e.g., npm, rubygems, rpms, etc.) and parse into groups that are importable via the import workspace XML data script. Synopsis A python project to gather component/license information from various package managers (e.g., npm, rubygems, rpms, etc.) and parse into groups that are importable via the import workspace XML data script for FlexNet Code Insight. Download: Contact Technical Support Password: 000022428
Switch to checking for "repository" field for npm urls first (for Github component matching). Fix bug for .gem files. Fix typo in gemsource documentation.
Run with -h for options. Discussion Input
For Node Modules:
A text file with a list of files paths containing Node Modules (run with -f <file> ). Or a text file with a list of Node Modules (run with -p <file> ).
For Ruby Gems:
A text file with a list of files paths containing .GEM files (run with -f <file> -t gems ). Or a text file (such as a gemfile) with a list of gems (run with -p <file> -t gems ). For Ruby Gems source code, use like npm with a text file of file paths (run with -f <file> -t gemsource )
A text file with a list of file paths containing RPMs (run with -f <file> -t rpms ).
For Composer (PHP) packages:
A text file with a list of files paths containing composer.json files (run with -f <file> -t php ). A composer.lock file with package JSON data. (run with -f <composer.lock> -t php ).
For BitBake (bb) files:
A text file with a list of files paths containing .bb files (run with -f <file> -t bitbake ).
For CSV file:
A csv (comma-separated) file with a list of groups having data in the following order: Group/Package Type, Name, Version, License, Description, URL, filepath, Component ID, and ComponentVersion ID
For Node Modules:
An installation of npm from NodeJS (http://nodejs.org/). Or local access to the package.json files.
For Ruby Gems:
Internet access. The Requests python package.
The rpm command (via Cygwin or your Linux package manager) And local access to the rpm files.
For Composer (PHP) packages:
A composer.lock file with package information in JSON format. Or local access to composer.json files.
Local access to .bb files.
All groups must have a name, at least.
For associating groups to components, either the components.txt file (which contains component information for the core database) or a JSON with credentials to use a Palamida MySQL database. Using a MySQL connection will require the mysql.connector package (https://dev.mysql.com/downloads/connector/python/2.1.html). Please take care when using database connections. Always make a verified backup before making direct database connections. This script only queries and does not write to the Palamida database.
For all: Python 2.7.x. The lxml package is necessary to pretty-print the XML output.
An XML file that's importable by the Palamida import/export script
For versions of the import/export script 3.2 and AFTER:
scriptRunner.bat <PATH_TO>importWorkspaceData.groovy --server <your core server url> --scan_server <your scan server url> --input groups.xml --workspace foo-workspace
For versions of the import/export script BEFORE 3.2:
scriptRunner.bat <PATH_TO>importWorkspaceData.groovy -input workspaceData.xml -workspace foo-workspace -check_md5_hash
If you give the script file paths, then it will try to associate the resulting groups to files and build that into the XML file. If you do not have file paths, you have the option to associate all of the resulting groups to a single file, otherwise when you import the groups they will not be attached to any files.
Please bear in mind the following caveats:
The information is not necessarily perfect. The developer might have been lazy or stupid and incorrectly filled out the metadata in the file (or wherever). We recommend doing a quick check against the actual licenses. The information is not necessarily complete. Some node modules will have ?Unknown License? given because the information was not available in the metadata. These projects likely have licenses in their source repositories (Github, etc) or other places. I recommend looking at each of these manually. The information does not give you P1 bundle issues. Each of these files must be checked for P1 issues as usual before being marked as reviewed (P1 search terms and P1 license matches). A major benefit of these scripts is that we have more time to be thorough looking for P1 stuff. This is an alpha release that is completely independent of the main Palamida product and is provided 'as is' as an add-on for convenience. It likely contains bugs and likely won't handle every special case. We welcome feedback and suggestions (firstname.lastname@example.org).
If you see any P6 groups with obvious know licenses, please let us know what the license text was so I can add it. License priorities can be adjusted in the Group.py file. Additional Information The current version is 1.9 (5-17-2016)
Switch to checking for "repository" field for npm urls first (for Github component matching) Fix bug for .gem files. Fix typo in gemsource documentation.
Version 1.8 (5-13-2016)
Better component matching. Attempts to find Github component first given a Github URL before doing a broader URL search. (Thanks Ed!) Groups will still be created when you don't have access to the npm registry. Clearer logging when checking npm registry.
Version 1.7 (4-6-2016)
Now compatible with import/export scripts v3.2 (for Palamida 6.8). NOTE CHANGE IN OUTPUT ABOVE. Support for gem source writeups. Some day will read gemspecs directly.
Version 1.6 (3-6-2016)
New components based on latest electronic update. CID and CVID for csv groupbuilding. Gem source code is now supported. Fixed typo in Ruby Gem group write-up (Thanks Ed!) Group.py will now try to associate a group to a component by an exact name match as a last gasp attempt. Warning that this may lead to more false postives. Feedback is welcome here. RPM bug fixes and improvements.
Version 1.5 (1-4-2016)
Various bug fixes and improvements.
Version 1.4 (12-21-2015).
The lxml is now an optional dependency. It's primarily used to pretty-print XML.
Version 1.3 (12-06-2015).
Faster node module processing and better console info. Option to use connection to Palamida MySQL database. Selects available version ID as well as component, if available (requires DB connection) Now supports BitBake and CSV files.
More documentation Options for formatting output. Handling more special cases.
Copyright (C) 2015-2016 Palamida Inc. All rights reserved.
This software is the confidential and proprietary information of Palamida Inc. and shall not be used, disclosed or reproduced, in whole or in part, for any purpose, without the prior written consent of Palamida Inc.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Summary This article provides the requirements and instructions to upgrade FlexNet Code Insight (Palamida) to 6.10.3. Synopsis This guide is for upgrading the FlexNet Code Insight (Palamida) application. For Compliance Library upgrades, Compliance Library Upgrade Guide.
This guide is for upgrading to FlexNet Code Insight (Palamida) 6.11.3 only. For new installations, see the Install SysAdmin Guide include with the product.
Do NOT perform an upgrade on a production server unless you are an experienced FlexNet Code Insight administrator.
It is highly recommended that you contact Flexera Software Support to assist you with the upgrade.
It is your responsibility to ensure the integrity and reliability of backups. Always test and verify backups, and maintain redundant offsite copies.
Flexera Software is not responsible for data loss due to corrupt or missing backups.
You must have one of these previous versions to use this guide:
6.11.1, 6.11.2, 6.10.0, 6.8.1, 6.8.0, 6.6.2, 6.6.1, 6.1.4, 6.1.3, 6.1.2, 6.1.1, 6.1.0, 6.0.5, 6.0.2, 5.2.4, 5.2.2
You will need the following to perform the upgrade:
The plain text database password for the user and database defined in core.db.properties.
You will need to run an Electronic Update as the final step in the upgrade. The core server must have outgoing Internet access on port 22, otherwise you will need to run the Electronic Update manually.
You will need enough free disk space to perform backups. Check the size of your workspaces directory, which may be large.
The FlexNet Code Insight 6.11.3 distribution zip file. Contact your Flexera representative if you do not have a copy.
The migrationImport.groovy script, located in /scriptRunner/scripts/ of your 6.11.3 application directory. This script copies the properties and configurations from your existing application directory (OLD_DIR) to the new application directory (NEW_DIR) and notifies you of any additional steps needed
The migrate.sh / migrate.bat script, located in /scriptRunner/bin/ of your 6.11.3 application directory. This script migrates your existing database schema from the existing version of FNCI to the new version.
If you have any custom core reports, you will need to re-run the custom SQL scripts that you initially used to install them.
NOTE: The commands in this guide are written for Linux. Windows users may choose to perform the steps with a mouse. Be sure to replace the sample values below with those of your installation.
This guide makes reference to the following variables. You can set these variables on the server in order to paste the commands in this guide.
# Current installed version
# Current app directory.
OLD_DIR=" /opt/CodeInsight/6.11.1 "
# New app directory, which will be created.
NEW_DIR="/opt/ CodeInsight /6.10.3"
# Base directory for backups (a 6.11.3 subdirectory will be created)
BACK_DIR="/opt/ CodeInsight /backup"
# Core server only - MySQL Database info.
DB_NAME=" CodeInsight "
# Scan servers only - Workspaces directory.
WS_DIR="/opt/ CodeInsight /workspaces"
You can paste the above into a file on the server (for example /tmp/code_insight_env) and edit the values. Then you can run source /tmp/code_insight_env to set the variables used in this guide. After the upgrade is complete, be sure to rm /tmp/code_insight_env , if it contains the database password.
Upgrade FlexNet Code Insight
Shutdown FlexNet Code Insight. For multi-server installs, shutdown all servers.
Backup the database. This step applies to CORE only.
NOTE: These commands are for MySQL. If you are using Oracle, obtain a fresh backup from your DBA before proceeding. Make sure your DBA is available to restore the backup promptly, in case it is needed.
mkdir -p $BACK_DIR/6.11.2
mysqldump -h $DB_HOST -u "$DB_USER" --password="$DB_PASS" -r migration_db.sql $DB_NAME
Backup workspaces directory. This step applies to all SCAN servers.
NOTE: This may take a long time depending on the size of your workspaces directory.
tar cf $BACK_DIR/6.10.3/migration_ws.tar .
Backup the installation directory.
# clear the tomcat temp files
rm -r tomcat/temp/*
tar czf $BACK_DIR/6.10.3/migration_app.tgz .
Extract the 6.11.3 distribution zip file (CodeInsight-6.11.3.zip) and move it to the new directory.
unzip -q CodeInsight-6.11.3.zip -d /tmp
mv /tmp/CodeInsight-6.11.3.zip $NEW_DIR
Place the migration import script in the new directory, and run the script.
./scriptRunner.sh -n ../scripts/migrationImport.groovy $OLD_DIR
Check the TODO log for any additional steps needed. Complete any necessary steps before continuing.
Run the database schema migration. This step applies to CORE only.
If database errors are seen, please re-run the database schema migration after resolving the error.
Run the new reports.sql to install new reports. Use the appropriate file according to your database vendor (MySQL in this example). This step applies to CORE only.
Note: The reports.sql file will overwrite any modifications to the report tables in the database. If you have custom reports, you will need to re-run the custom SQL to install them after you have run the new reports.sql file. Make sure you have your custom SQL scripts before you run this.
mysql -h $DB_HOST -u "$DB_USER" --password="$DB_PASS" -D $DB_NAME \
-e "source $NEW_DIR/dbScripts/mysql/reports.sql"
NOTE: FlexNet Code Insight 6.11.3 has features which requires a new Data Services Enabled key. You can continue to start and use the application with your existing key, but there will be errors seen with the features that require this key.
Start the new FlexNet Code Insight application. For multi-server installs, do this after you have completed the previous steps on all servers.
./startup.sh && tail -f ../logs/catalina.out
Check the log for any errors, and resolve them before continuing.
Log into the WebUI and run the Electronic Update. This step applies to CORE only.
NOTE: DO NOT SKIP THIS STEP.
In most cases, the electronic update will be scheduled automatically. Check the Scheduler tab in the WebUI. If the update is not running, you can trigger it through Administration > Updates, and clicking Check for Electronic Update.
If your installation does not have outgoing Internet access on port 22, you will need to run the update manually.
If you face certificate errors on startup of the scan server or if you are unable to see your scan server from the application UI, then you will need to import the certificate being served by Tomcat on the scan server into the JDK of the core server.
Log into the WebUI and go to Help > About to verify the version.
Create a test project and workspace.
Ensure that the Detector client launches for the workspace.
Close Detector and schedule a scan.
Revert to Previous Version
Ensure the FlexNet Code Insight server is stopped. For multi-server installs, ensure all servers are stopped.
Restore the database. This step applies to CORE only.
NOTE: These commands are for MySQL. If you are using Oracle, have your DBA restore the backup.
mysql -h "$DB_HOST" -u "$DB_USER" --password="$DB_PASS" -D "$DB_NAME" < db_migration.sql
Restore workspaces backup. This step applies to all SCAN servers.
NOTE: If you did not open, create, or scan any workspaces while the new version was running, you can skip this step.
tar xf $BACK_DIR/6.11.2/ws_migration.tar
Start the previous installation. For multi-server installs, do this after you have completed the previous steps on all servers.
./startup.sh && tail -f ../logs/catalina.out
Summary This script will validate the MD5 checksums of the Compliance Library files. Synopsis This script will validate the MD5 checksums of the Compliance Library files. Discussion Compliance Library Upgrade Guide for instructions on installing a new Compliance Library.
Download validateSigMD5s.sh for CL 2.34 and later
Download validateSigMD5s-old.sh for CL 2.33 and earlier
Summary This article discusses why a product can't be found in Product and License Center even though customer has just renewed the maintenance and can log in and download other products from PLC. Symptoms After renewing the maintenance for a product, a customer finds that he can't see the product in Product and License Center, even though he can log in and download other products from PLC. Cause The maintenance renewal order has been processed under a different account of this big company, but the contact person doesn't have access to that account. Resolution Submit a ticket to Orders to request the PLC access for the customer contact and update the entitlement contact accordingly.
Summary This article provides steps to investigate an issue where the OS freezes during scanning Symptoms The following problems occurred at customer 's environment. OS freezes during scan execution. Resolution Customer confirmed the issue no longer appears. No further access to the database to test resolution.
Summary This article documents an issue experienced following an upgrade to 2018 R3. Synopsis Following an upgrade the web application now fails to start with the following error: java.lang.IllegalStateException: Unable to complete the scan for annotations for web application [/codeinsight] due to a StackOverflowError. Possible root causes include a too low setting for -Xss and illegal cyclic inheritance dependencies. The class hierarchy being processed was [org.bouncycastle.asn1.ASN1Boolean->org.bouncycastle.asn1.DERBoolean->org.bouncycastle.asn1.ASN1Boolean] Discussion Reinstalling the system resolved this issue
Summary This articles discusses how to configure claimed copyrights. Synopsis Claimed copyright settings can be used to have Palamida designate a selection of copyrights to be ignored when the software scans for third-party indicators. Only "unparseable" copyrights can be ignored. An unparseable copyright is one that is not strong enough to match a pattern but still appears to be copyright statement. Files containing copyrights put into the "claimed bucket" will not be flagged for third-party indicators if the customer's claimed copyright is the only indicator.This behavior ensures that real copyright matches are still detected. Ignored copyrights are not accessible in the scan results. (For more information on how to control the sensitivity of copyright detection, please refer to the section, "Tuning Copyright Detection" in the Enterprise Installation and System Administration Guide.) Discussion Edit the file $palamida/config/scanEngine/claimedCopyrights.txt
# This file may be used to force detected copyrights into the -claimed- bucket.
# The purpose of this is to move all copyrights that belong to the user into a
# separate bucket (-claimed-) so that they do not cause the scanned file to be
# flagged as having third-party indicators if the only evidence is the customer's
# The case-sensitive strings or regular expression patterns below will cause a
# detected copyright to be moved into the -claimed- bucket.
Defining copyright patterns for which to take ownership and move to the -claimed- bucket:
Do not use commas (,) while defining copyright, date or owner patterns. Since commas are used as delimiters, they will end up tokenizing your patterns into multiple values. For example: Palamida, Inc. should be defined as Palamida Inc. (Note that there is no comma in the owner pattern.) Do not use any of the following reserved characters: \n \r | # ! % : ; , / * " space. If you use these reserved characters, they will be replaced by a whitespace (' ') character. If Java special characters are part of the defined copyright pattern, ensure that double slashes are used to escape both the properties file parsing as well as the Java interpretation. An example of this is Copyright 2008 would be defined as Copyright \[C\] 2008.
Summary This article discusses how to use the new code search feature in Detector 6.8. Synopsis For the new code search feature, scanned code is indexed so that a search may be performed at anytime. This eliminates the need for a text/grepping tool outside of the product. This feature can be combined with Palamida Groups, tags and custom filters. The tree is filtered to files containing results such that these files can be Marked as Reviewed, Added to Group or tagged. Discussion If you click the search button, the following options are offered:
Search for File Name
You can now search for a file name across the code base using this new feature menu and seeing the results as a new highlighting feature in the Source matches:
Search for Selected Text
You can also search for any selected string or text in files across the scanned codebase. You can find this in a new menu when you select and right-click on text in the Partial Matches Pane.
Summary You are unable to launch Tomcat or Detector and receive an error in $palamida/tomcat/logs/catalina.out . Symptoms You are unable to launch Tomcat or Detector and receive the following or similar errors in $palamida/tomcat/logs/catalina.out:
WARNING: Due to a permissions issue accessing the log files, the status of the database has been changed to read only.
java.io.FileNotFoundException: /home/palamida/workspaces/ePortal/workspace/log/log3.dat (Permission denied) Cause This and similar permissions errors can be caused by launching Tomcat with root permissions. The result of this is that any data created during that Tomcat session (such as scanning a workspace) will be written with root ownership; subsequent attempts to access data created during these sessions will thus be denied for having insufficient privileges.
NOTE: Tomcat should never be run with root user privileges Resolution There are two main approaches to this issue:
Re-launch Tomcat with your desired user. Create a new Workspace. Re-scan the materials from the inaccessible workspace in your new workspace.
Determine the name of your Palamida database.
You can find the name of your database on the db.url line of $palamida/config/core/core.db.properties. For example:
Within your workspaces/ directory, run the ls -al command to list the workspace directories along with their owner username and group.
For any workspaces with root ownership, run the following command, using the username you intend to use to launch the Palamida server, for example:
chown -R palamida:palamida /opt/palamida/workspaces
Summary This article provides a resolution for the message: You are not authorized to access workspaces for the current project. Symptoms When launching Detector, you see this message:
NOTE: If you are on Palamida 6.1.0 or earlier, you must have JRE 7u40 or earlier on the client machine to launch Detector.
Resolution Log out of the WebUI, log in again, and relaunch Detector from a new JNLP file.
Clear temporary files from Java control panel:
Clear all Palamida code-signing certificates from the Java Control Panel. Clear all browser cache and restart your browser.
Synopsis Almost always, the first step in error resolution is log analysis. This document shows how to fully shut down the Palamida Tomcat servers, kill any hanging Java processes, and generate a new set of logs without historical data. This helps the Support team isolate an error as it occurs. Discussion
Please STOP all Palamida by running $palamida/tomcat/bin/shutdown.sh.
After shutdown of the Tomcat servers, ensure that there are no hanging Java processes.
You can ensure there are no Java processes by the following commands:
This might return a process called "Bootstrap" with an associated process ID.
ii. ps aucx | grep java
This might return a process called "java" with an associated process ID.
iii. Kill the Java process if necessary by the kill -9 command:
Next, delete all the logs in $palamida/logs and $palamida/tomcat/logs
Start the Palamida Tomcat servers ? this will generate a set of new logs to help isolate the error
Attempt the action that caused the error before.
If you experience the error again, please zip and email the new logs with the error.
Please include all logs in $palamida/logs and $palamida/tomcat/logs Please also include the following configuration files:
$palamida/config/core/core.properties $palamida/config/core/core.db.properties $palamida/config/scanEngine/scan.properties $palamida/config/scanEngine/scanEngine.properties $palamida/tomcat/bin/catalina.sh (For Windows, use catalina.bat instead) NOTE: Please rename this file, as e-mail clients such as Outlook will block it to be quarantined.) $palamida/tomcat/conf/server.xml
Additional Information If you are experiencing Detector related issues, we may suggest for you to send us the logs. Please refer to Enable Java Console Logging for enabling Java Console Logs.
Summary On opening Detector you receive the error: Unable to launch the application. Symptoms On opening Detector you receive the error:
And the following error:
JAR resources in JNLP file are not signed by the same certifica
Cause There are cached expired certificates or JAR files from an earlier version. Resolution Delete all Palamida code-signing certificates from the Java Control Panel.? Clear your Java cache from the Java Control Panel:
Clear all browser cache and restart your browser.