- Revenera Community
- :
- FlexNet Publisher
- :
- FlexNet Publisher Forum
- :
- Re: ERROR: Server has already hit the limit of MAX_CONNECTIONS with respect to checkout connections
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Subscribe
- Mute
- Permalink
- Report Inappropriate Content
ERROR: Server has already hit the limit of MAX_CONNECTIONS with respect to checkout connections
I am trying to analyse an issue with a licensing server that we use as a customers of a company Foobar that provides us with a product that uses Flexera. So Foobar provides their vendor daemon to us. AFAIU we are not a typical users of their software--we run a few order of magnitude more intensive loads than their usual customers. After some time their product (it's a webserver) gives me the following error:
File "build\bdist.win-amd64\egg\Foobar\workspace.py", line 50, in __init__
File "build\bdist.win-amd64\egg\Foobar\workspace.py", line 1225, in _create
File "build\bdist.win-amd64\egg\Foobar\webservice.py", line 94, in put
File "build\bdist.win-amd64\egg\Foobar\webservice.py", line 358, in process_response
Foobar.error.FoobarError: : b'{"errors":["Looks like we encountered an error... TransformationException occurred, with message: Error when creating Workspace: \\nThis was caused by a DAOProcessingException, with message: Error creating Derby DAO: \\nThis was caused by a IllegalStateException, with message: Error creating Workspace. No valid Foobar_Portfolio license available\\nMaximum number of connections reached for this vendor daemon.\\nMaximum connections limit for this vendor daemon has been reached\\nFeature: Foobar_Portfolio\\nLicense path: @10.156.15.237:@localhost:*.lic:\\nFlexNet Licensing error:-237,10007"]}'
When this error pops up, I go to a license server machine to check its vitals. First check the licenses in use:
$ lmutil lmstat -a
lmutil - Copyright (c) 1989-2019 Flexera. All Rights Reserved.
Flexible License Manager status on Thu 3/9/2023 14:14
License server status: 27000@Foobar-license-research
License file(s) on Foobar-license-research: /home/btschroer/license_server/lic/2023-02-09/Netlic100_Foobar-license-research_E1A246C64D33.lic:
Foobar-license-research: license server UP (MASTER) v11.16.6
Vendor daemon status (on Foobar-license-research):
Foobar: UP v11.16.6
Feature usage info:
Users of Foobar_Portfolio: (Total of 100 licenses issued; Total of 0 licenses in use)
Users of Foobar_Optimize: (Total of 100 licenses issued; Total of 0 licenses in use)
Users of Foobar_API: (Total of 100 licenses issued; Total of 0 licenses in use)
Users of Foobar_API_Web_Service: (Total of 100 licenses issued; Total of 0 licenses in use)
Users of Foobar_API_Python: (Total of 100 licenses issued; Total of 0 licenses in use)
Users of Alpha_Factor: (Total of 100 licenses issued; Total of 0 licenses in use)
Users of Foobar_Robust: (Total of 100 licenses issued; Total of 0 licenses in use)
Users of Foobar_Portfolio_Analytics: (Total of 3 licenses issued; Total of 0 licenses in use)
Users of PA_GUI: (Total of 3 licenses issued; Total of 0 licenses in use)
Users of RA_GUI: (Total of 3 licenses issued; Total of 0 licenses in use)
Users of PA_TASK: (Total of 3 licenses issued; Total of 0 licenses in use)
Users of RA_TASK: (Total of 3 licenses issued; Total of 0 licenses in use)
Users of NB_GUI: (Total of 3 licenses issued; Total of 0 licenses in use)
None of the licenses are borrowed.
I check the amount of connections:
$ sudo netstat -tupan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 511/sshd
tcp 0 0 127.0.0.1:42008 127.0.0.1:27000 ESTABLISHED 24291/Foobar
tcp 0 0 127.0.0.1:20202 127.0.0.1:33100 TIME_WAIT -
tcp 0 0 127.0.0.1:33756 127.0.0.1:51841 ESTABLISHED 24291/Foobar
tcp 0 0 10.156.15.237:22 99.67.184.39:33815 TIME_WAIT -
tcp 0 368 10.156.15.237:22 35.235.241.177:35229 ESTABLISHED 6565/sshd: ant [pri
tcp6 0 0 :::47025 :::* LISTEN 24291/Foobar
tcp6 0 0 :::27000 :::* LISTEN 24289/lmgrd
tcp6 0 0 127.0.0.1:27000 127.0.0.1:42008 ESTABLISHED 24289/lmgrd
tcp6 0 0 10.156.15.237:47025 10.156.0.27:55494 ESTABLISHED 24291/Foobar
tcp6 0 0 127.0.0.1:51841 127.0.0.1:33756 ESTABLISHED 24291/Foobar
Nothing suspicious here as well. Not a lot of connections. There is one external connection open to Foobar vendor daemon. I also check the amount of open file descriptors for the license server processes:
$ ps -aux | grep license
ant 6688 2.7 52.0 1072736 1062636 pts/1 S+ 13:30 1:44 vim license_server.log
ant 10068 0.0 0.0 6072 884 pts/2 R+ 14:33 0:00 grep license
root 24289 0.0 0.1 13856 2928 ? Sl Mar05 2:26 /home/btschroer/license_server/Enduser/lmgrd -c /home/btschroer/license_server/lic/2023-02-09/Netlic100_Foobar-license-
research_E1A246C64D33.lic -l /home/btschroer/license_server.log
root 24291 0.5 0.3 225476 6292 ? Ssl Mar05 37:56 Foobar -T Foobar-license-research 11.16 7 -c :/home/btschroer/license_server/lic/2023-02-09/Netlic100_Foobar-license-re
search_E1A246C64D33.lic: -lmgrd_port 6978 -srv c7honthNaG015ke2PIqDv3xmyKdB2iAwk2OJSJgfid8yxiZJfby4R096OjhKRab --lmgrd_start 640421d2 -vdrestart 0
$ sudo ls /proc/24289/fd/ | wc -l
8
$ sudo ls /proc/24291/fd/ | wc -l
As soon as we hit this problem the only thing that helps is to restart the license server. We have a debug log of lmgrd enabled which shows the license drain. Today it looked like this:
9:00:15 (Foobar) WARNING: Server has hit 80.00% of the max connections limit, please try disconnecting some clients or some non-client connections from the server
11:00:10 (Foobar) WARNING: Server has hit 90.00% of the max connections limit, please try disconnecting some clients or some non-client connections from the server
11:45:12 (Foobar) WARNING: Server has hit 95.00% of the max connections limit, please try disconnecting some clients or some non-client connections from the server
12:45:48 (Foobar) WARNING: Server has hit 100.00% of the max connections limit, please try disconnecting some clients or some non-client connections from the server
Note the timestamps--in a matter of about three hours we have lost the capacity to connect to the license server completely. I wrote a little script to parse the IN and OUT events displayed in the log file and it tells me that there was a total of 3999808 borrow-return events from 901 hosts during the lifetime of this instance (we started it around 05:00 UTC on 2023-03-12 and it worked till around 12:45 UTC of 2023-03-14 -- that is about two days.
1. Could someone suggest a direction of further investigation? It seems all the vitals are fine, but we still get the errors.
2. Where can I find documentation and SDK for implementing vendor daemon--I hope I might get some insights on whether there is something Foobar could have done wrong, when implementing their vendor daemon.
AFAIU from the user manual the vendor is responsible for building the vendor daemon. Could this be an error in their implementation? Where can I get some documentation on how to build my own vendor daemon--maybe it could shed some light if Foobar is not handling something in the vendor daemon proerly.
- Mark as New
- Subscribe
- Mute
- Permalink
- Report Inappropriate Content
Hi @mezhaka ,
We maintain a vendor_connections count on vendor daemon side. After successful connection, when client sends first FNP proprietary message to vendor daemon, vendor daemon checks the type of the Client and checks the vendor_connection count .If count has still not reached the threshold for that type of Client then it increments the connection by 1 and send the appropriate message back to Client. In case threshold value is already reached, then vendor daemon drops the connection and send appropriate error message back to the Client.
netstat -a -o -n , o/p is considered but the above checks are made too . I would suggest you to raise a support case with us so that we can investigate this further .
- Mark as New
- Subscribe
- Mute
- Permalink
- Report Inappropriate Content
@jyadav
Thank you for a swift response,
How do I raise a support case? Is this something the provider of the software that uses FlexNet Publisher should do--in my case the Foobar company or can is it something I can do directly?
- Mark as New
- Subscribe
- Mute
- Permalink
- Report Inappropriate Content
Hi @mezhaka ,
Yes , that's right . Foobar has to raise a case with us .
![](/skins/images/A5EADE085F77DE82FB3B14D6B307AE6A/responsive_peak/images/icon_anonymous_message.png)