Running into request failures from ServiceNow intermittently when responses take longer than 30 seconds

[a_cutler2]

Using App Broker with Intune we are seeing a problem where requests from ServiceNow fail with the error "Exception caught: No response for ECC message request with sysid=xxxxxxxxx after waiting for 30 seconds in ECC Queue". Based on the requests log it does seem like the successful requests are processed much faster than the failed ones. Please see the example screenshots attached. Has anyone run into an issue like this? Any ideas how to resolve it? 

[jdempsey]

I ran into this back in 2020.  I've only seen this in non-production instances of ServiceNow, where they generally run slower (or at least intermittently slower).  Here's the solution I found, but ServiceNow doesn't recommend using this in a production environment.

I added a new system property (introduced in Madrid) to the system properties table (sys_properties.list).  ServiceNow added a new property called glide.http.outbound.max_timeout, which defaults to 30 seconds and that's the highest it can be set.  But they also added another property called glide.http.outbound.max_timeout.enabled which defaults to true but can be manually set to false.  If this value is set to false, then it will ignore the max_timeout value and wait longer.  So I just added this second property to my instance under the Flexera App Portal application scope, and now it will wait longer than 30 seconds.  Please note that this is probably a bad thing for a production environment, but since DEV instances tend to take a long time, it may be necessary when using a DEV instance.

​Available System Properties

Add a System Property

Add glide.http.outbound.max_timeout = 60 (integer) - adjust higher as needed
Add glide.http.outbound.max_timeout.enabled = false (true | false)

Start by putting these in Flexera Software App Portal application scope.  If that doesn’t work, move them to Global scope.  Check the “Private” option on each so it doesn’t apply to exported update sets.

[CharlesW]

Setting the timeout should help, but it is not ideal. I might suggest upgrading to 2023 R2.. The root cause of the issue is likely that the REST API in App Broker needs to "warm up" after a period of inactivity. this is similar to an Application Pool shutting down to inactivity, but is isolated to the REST API. In R2 they have the service pinging the REST API  every couple of minutes in an effort to keep it perpetually warm.