This website uses cookies. By clicking Accept, you consent to the use of cookies. Click Here to learn more about how we use cookies.
Accept
Reject
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
mezhaka
Level 2
‎Mar 14, 2023 03:14 PM

ERROR: Server has already hit the limit of MAX_CONNECTIONS with respect to checkout connections

I am trying to analyse an issue with a licensing server that we use as a customers of a company Foobar that provides us with a product that uses Flexera. So Foobar provides their vendor daemon to us. AFAIU we are not a typical users of their software--we run a few order of magnitude more intensive loads than their usual customers. After some time their product (it's a webserver) gives me the following error:

 

File "build\bdist.win-amd64\egg\Foobar\workspace.py", line 50, in __init__
  File "build\bdist.win-amd64\egg\Foobar\workspace.py", line 1225, in _create
  File "build\bdist.win-amd64\egg\Foobar\webservice.py", line 94, in put
  File "build\bdist.win-amd64\egg\Foobar\webservice.py", line 358, in process_response
Foobar.error.FoobarError: : b'{"errors":["Looks like we encountered an error... TransformationException occurred, with message: Error when creating Workspace: \\nThis was caused by a DAOProcessingException, with message: Error creating Derby DAO: \\nThis was caused by a IllegalStateException, with message: Error creating Workspace. No valid Foobar_Portfolio license available\\nMaximum number of connections reached for this vendor daemon.\\nMaximum connections limit for this vendor daemon has been reached\\nFeature:       Foobar_Portfolio\\nLicense path:  @10.156.15.237:@localhost:*.lic:\\nFlexNet Licensing error:-237,10007"]}'

 

When this error pops up, I go to a license server machine to check its vitals. First check the licenses in use:

 

$ lmutil lmstat -a
lmutil - Copyright (c) 1989-2019 Flexera. All Rights Reserved.
Flexible License Manager status on Thu 3/9/2023 14:14
License server status: 27000@Foobar-license-research
    License file(s) on Foobar-license-research: /home/btschroer/license_server/lic/2023-02-09/Netlic100_Foobar-license-research_E1A246C64D33.lic:
Foobar-license-research: license server UP (MASTER) v11.16.6
Vendor daemon status (on Foobar-license-research):
    Foobar: UP v11.16.6
Feature usage info:
Users of Foobar_Portfolio:  (Total of 100 licenses issued;  Total of 0 licenses in use)
Users of Foobar_Optimize:  (Total of 100 licenses issued;  Total of 0 licenses in use)
Users of Foobar_API:  (Total of 100 licenses issued;  Total of 0 licenses in use)
Users of Foobar_API_Web_Service:  (Total of 100 licenses issued;  Total of 0 licenses in use)
Users of Foobar_API_Python:  (Total of 100 licenses issued;  Total of 0 licenses in use)
Users of Alpha_Factor:  (Total of 100 licenses issued;  Total of 0 licenses in use)
Users of Foobar_Robust:  (Total of 100 licenses issued;  Total of 0 licenses in use)
Users of Foobar_Portfolio_Analytics:  (Total of 3 licenses issued;  Total of 0 licenses in use)
Users of PA_GUI:  (Total of 3 licenses issued;  Total of 0 licenses in use)
Users of RA_GUI:  (Total of 3 licenses issued;  Total of 0 licenses in use)
Users of PA_TASK:  (Total of 3 licenses issued;  Total of 0 licenses in use)
Users of RA_TASK:  (Total of 3 licenses issued;  Total of 0 licenses in use)
Users of NB_GUI:  (Total of 3 licenses issued;  Total of 0 licenses in use)

 


None of the licenses are borrowed. 

I check the amount of connections:

 

$ sudo netstat -tupan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      511/sshd
tcp        0      0 127.0.0.1:42008         127.0.0.1:27000         ESTABLISHED 24291/Foobar
tcp        0      0 127.0.0.1:20202         127.0.0.1:33100         TIME_WAIT   -
tcp        0      0 127.0.0.1:33756         127.0.0.1:51841         ESTABLISHED 24291/Foobar
tcp        0      0 10.156.15.237:22        99.67.184.39:33815      TIME_WAIT   -
tcp        0    368 10.156.15.237:22        35.235.241.177:35229    ESTABLISHED 6565/sshd: ant [pri
tcp6       0      0 :::47025                :::*                    LISTEN      24291/Foobar
tcp6       0      0 :::27000                :::*                    LISTEN      24289/lmgrd
tcp6       0      0 127.0.0.1:27000         127.0.0.1:42008         ESTABLISHED 24289/lmgrd
tcp6       0      0 10.156.15.237:47025     10.156.0.27:55494       ESTABLISHED 24291/Foobar
tcp6       0      0 127.0.0.1:51841         127.0.0.1:33756         ESTABLISHED 24291/Foobar

 


Nothing suspicious here as well. Not a lot of connections. There is one external connection open to Foobar vendor daemon. I also check the amount of open file descriptors for the license server processes:

 

$ ps -aux | grep license
ant       6688  2.7 52.0 1072736 1062636 pts/1 S+   13:30   1:44 vim license_server.log
ant      10068  0.0  0.0   6072   884 pts/2    R+   14:33   0:00 grep license
root     24289  0.0  0.1  13856  2928 ?        Sl   Mar05   2:26 /home/btschroer/license_server/Enduser/lmgrd -c /home/btschroer/license_server/lic/2023-02-09/Netlic100_Foobar-license-
research_E1A246C64D33.lic -l /home/btschroer/license_server.log
root     24291  0.5  0.3 225476  6292 ?        Ssl  Mar05  37:56 Foobar -T Foobar-license-research 11.16 7 -c :/home/btschroer/license_server/lic/2023-02-09/Netlic100_Foobar-license-re
search_E1A246C64D33.lic: -lmgrd_port 6978 -srv c7honthNaG015ke2PIqDv3xmyKdB2iAwk2OJSJgfid8yxiZJfby4R096OjhKRab --lmgrd_start 640421d2 -vdrestart 0
$ sudo ls /proc/24289/fd/ | wc -l
8
$ sudo ls /proc/24291/fd/ | wc -l

 


As soon as we hit this problem the only thing that helps is to restart the license server. We have a debug log of lmgrd enabled which shows the license drain. Today it looked like this:

 

9:00:15 (Foobar) WARNING: Server has hit 80.00% of the max connections limit, please try disconnecting some clients or some non-client connections from the server
11:00:10 (Foobar) WARNING: Server has hit 90.00% of the max connections limit, please try disconnecting some clients or some non-client connections from the server
11:45:12 (Foobar) WARNING: Server has hit 95.00% of the max connections limit, please try disconnecting some clients or some non-client connections from the server
12:45:48 (Foobar) WARNING: Server has hit 100.00% of the max connections limit, please try disconnecting some clients or some non-client connections from the server

 

Note the timestamps--in a matter of about three hours we have lost the capacity to connect to the license server completely. I wrote a little script to parse the IN and OUT events displayed in the log file and it tells me that there was a total of 3999808 borrow-return events from 901 hosts during the lifetime of this instance (we started it around 05:00 UTC on 2023-03-12 and it worked till around 12:45 UTC of 2023-03-14 -- that is about two days.

1. Could someone suggest a direction of further investigation? It seems all the vitals are fine, but we still get the errors.
2. Where can I find documentation and SDK for implementing vendor daemon--I hope I might get some insights on whether there is something Foobar could have done wrong, when implementing their vendor daemon.



AFAIU from the user manual the vendor is responsible for building the vendor daemon. Could this be an error in their implementation? Where can I get some documentation on how to build my own vendor daemon--maybe it could shed some light if Foobar is not handling something in the vendor daemon proerly.

0 Kudos
(3) Replies
jyadav
Flexera Alumni
‎Mar 14, 2023 10:49 PM

Hi @mezhaka  ,

We maintain a vendor_connections count on vendor daemon side. After successful connection, when client sends first FNP proprietary message to vendor daemon, vendor daemon checks the type of the Client and checks the vendor_connection count .If count has still not reached the threshold for that type of Client then it increments the connection by 1 and send the appropriate message back to Client. In case threshold value is already reached, then vendor daemon drops the connection and send appropriate error message back to the Client.
netstat -a -o -n , o/p is considered but the above checks are made too . I would suggest you to raise a support case with us so that we can investigate this further . 

0 Kudos
mezhaka
Level 2
In response to jyadav
‎Mar 15, 2023 08:53 AM

@jyadav 
Thank you for a swift response,

How do I raise a support case? Is this something the provider of the software that uses FlexNet Publisher should do--in my case the Foobar company or can is it something I can do directly?

0 Kudos
jyadav
Flexera Alumni
In response to mezhaka
‎Mar 15, 2023 08:56 AM

Hi @mezhaka  ,

Yes , that's right . Foobar has to raise a case with us .

0 Kudos