gyorisziszi
Level 2

SCCM data upload failure

Dear Flexera Community.

I have a problem with my SCCM imports. I have one which is running completely succesful and I have several ones that can't upload the gathered data. The data gathering in all cases are good, I think I have a problem with the upload.

The interesting part of the situation that if the gathered data file (*.zip) is smaller than 3 MB (don't have to cut it into pieces) the beacon can upload without any errors, but if greater then 3MB the upload is failed. 

Here is the LOG from the beacon:

LOG1: This appears when the beacon is retrying upload the segments:

 (error: Upload segment failed (An error occurred while sending the request.))

LOG2: After several tryings the beacon gives me this ERROR:

[Upload.UploadProcess ] [ERROR] Package upload failed
Flexera.SaaS.Transport.Core.ComplianceApiException: Upload segment failed (An error occurred while sending the request.) ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: The request was canceled. ---> System.InvalidOperationException: The stream was already consumed. It cannot be read again.
at System.Net.Http.StreamContent.PrepareContent()
at System.Net.Http.StreamContent.SerializeToStreamAsync(Stream stream, TransportContext context)
at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context)
at System.Net.Http.HttpContent.CopyTo(Stream stream)
at System.Net.HttpWebRequest.EndWriteHeaders_Part2()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Flexera.SaaS.Transport.Upload.UploadClient.UploadSegment(FileSegment segment)
at Flexera.SaaS.Transport.Upload.UploadProcess.TryUpload(UploadClient client, UploadedPackageFile package, Action notifySuccessfulConnection)
at Flexera.SaaS.Transport.Upload.UploadProcess.Retry[T](String actionName, Func`2 action, Int32 times)
at Flexera.SaaS.Transport.Upload.UploadProcess.ProcessPackage(UploadedPackageFile package)
at Flexera.SaaS.Transport.Upload.UploadProcess.ProcessPackageList(Int32 exitCode, List`1 packages)

There wasn't any network, infrastucture etc. change.

I tried to change the "NetworkTime" registry and deleted all the previous file from the IntermediateData folder.

Thanks for you time!

0 Kudos
3 Replies
ChrisG
Community Manager Community Manager
Community Manager

I've seen various reports of errors with similar logging to what you're seeing, but no clear root cause or resolution identified. Other organizations who have been affected by this issue have had some success at configuring the upload process to run at times when they thought the network was less busy, or by turning off some kind of specialized logging of network activity they had configured on the destination server. None of that is very specific to help get to the bottom of what is going on though.

Some thoughts of things to consider are:

  • Have you looked in IIS logs and FlexNet logs on the parent (destination) server to see what activity they show happening on the parent at the same time as the failure is recorded in the child beacon?
  • Is there any kind of firewall, load balancer, reverse proxy, or other similar network device in place between the child beacon and the parent server it is uploaded to? Something like that could be interfering in a way that breaks the upload communications.

One way to investigate further may be to run some PowerShell statements like the following. (Change the details in the first 2 lines as appropriate for your setup.)

$parentBeaconURL = "http://your-parent-server.com/"
$beaconInstallPath = "${env:ProgramFiles(x86)}\Flexera Software\Inventory Beacon"

Write-Output "Dummy file" >test.txt
Compress-Archive test.txt test.zip

$wc = New-Object System.Net.WebClient

Add-Type -Path "$beaconInstallPath\DotNet\bin\Flexera.Beacon.Common.dll"
$bc = New-Object Flexera.Beacon.Common.BeaconConfig

$wc.Credentials = New-Object System.Net.NetworkCredential($bc::BeaconServicesUserName, $bc::BeaconServicesPassword)

$wc.UploadFile("$parentBeaconURL/inventory-beacons/api/upload/test.zip", "POST", "test.zip")

 If this fails, it may give some output which gives further insight into what is going on. If it succeeds, try repeating the final "$wc.UploadFile" command to see if it consistently succeeds or whether the results intermittently vary. You could try it with different size .zip files too.

(Did my reply solve the question? Click "ACCEPT AS SOLUTION" to help others find answers faster. Liked something? Click "KUDO". Anything expressed here is my own view and not necessarily that of my employer, Flexera.)
0 Kudos
gyorisziszi
Level 2

Dear ChrisG,

Thanks for your suggestions. I tried your script and the a file (with the same name) appeared in the processing server but with 0 KB size. -> I found some segments of our SCCM import zip files but with 0 KB and I found some with 3072KB size (this is the segment limit). If you tried to open it tells it can not open the files.

Answers to your questions:

  • Yes I checked the IIS and Flexnet logs and unfortunately I didn't find "suspicious" log record.
  • We have firewall but it worked before with that configurations. As I know we don't have loadbalancer, reverse proxy etc.

One of the SSCM imports gives me this log record:

2022-01-26 09:54:51,301 [Upload.UploadProcess ] [INFO ] Found 2 new package(s) to upload
2022-01-26 09:54:51,473 [Upload.UploadProcess ] [INFO ] Processing package 'I[S=SCCM_Import_Name]_20220126085135.zip' in C:\ProgramData\Flexera Software\Beacon\IntermediateData
2022-01-26 09:54:51,504 [Upload.UploadProcess ] [INFO ] Uploading to https://beacon_url.com/inventory-beacons/api/upload/
2022-01-26 09:54:51,614 [tionProtocolIdentifier] [INFO ] The protocol used for communication is Tls12
2022-01-26 09:54:51,614 [Upload.UploadProcess ] [INFO ] Uploading using TLS version Tls12
2022-01-26 09:54:51,614 [otNetVersionIdentifier] [INFO ] .Net framework version identified is .NET Framework 4.6.2
2022-01-26 09:54:51,614 [Upload.UploadProcess ] [INFO ] .Net framework present: .NET Framework 4.6.2
2022-01-26 09:54:51,614 [Upload.UploadProcess ] [INFO ] Starting upload of 804,2KiB package in 1 segments...
2022-01-26 09:54:51,614 [Upload.UploadProcess ] [INFO ] - uploading segment 1: 804,2KiB
2022-01-26 09:54:52,254 [Upload.UploadProcess ] [INFO ] File queued successfully on server
2022-01-26 09:54:52,254 [Upload.UploadProcess ] [INFO ] Upload successful, move to uploaded folder

This upload worked like as expected but the other 3 are not.

I recognised a strange "behaviour" from the ComplienceUpload.exe:

2022-01-26 10:06:31,361 [Upload.UploadProcess ] [INFO ] Processing package 'I[S=SCCM_Import_Name]_20220126090627.zip' in C:\ProgramData\Flexera Software\Beacon\IntermediateData
2022-01-26 10:06:31,393 [Upload.UploadProcess ] [INFO ] Uploading to https://beacon_url.com/inventory-beacons/api/upload/
2022-01-26 10:06:31,486 [tionProtocolIdentifier] [INFO ] The protocol used for communication is Tls12
2022-01-26 10:06:31,486 [Upload.UploadProcess ] [INFO ] Uploading using TLS version Tls12
2022-01-26 10:06:31,502 [otNetVersionIdentifier] [INFO ] .Net framework version identified is .NET Framework 4.6.2
2022-01-26 10:06:31,502 [Upload.UploadProcess ] [INFO ] .Net framework present: .NET Framework 4.6.2
2022-01-26 10:06:31,502 [Upload.UploadProcess ] [INFO ] Starting upload of 27,3MiB package in 10 segments...
2022-01-26 10:06:31,502 [Upload.UploadProcess ] [INFO ] - uploading segment 1: 3,0MiB
2022-01-26 10:06:32,064 [Upload.UploadProcess ] [INFO ] - uploading segment 2: 3,0MiB
2022-01-26 10:06:32,080 [Upload.UploadProcess ] [WARN ] Upload package failed, retrying in 16 seconds (error: Upload segment failed (An error occurred while sending the request.))
2022-01-26 10:06:48,173 [Upload.UploadProcess ] [INFO ] Resuming upload of 27,3MiB package - 24,3MiB to upload in 9 segments...
2022-01-26 10:06:48,173 [Upload.UploadProcess ] [INFO ] - uploading segment 1: 3,0MiB

In the bold lines you can see It can upload one segment of the 10 but when the second one is coming it gives the error.

 

Do you have any other idea? What to check, what log to analyse or it could be some network related issue(s)? 

 

It's just really "strange" that some of them are working and some of them are don't.

 

Thanks for your time!

0 Kudos
ChrisG
Community Manager Community Manager
Community Manager

Seeing the file on the parent server as 0KB in size has indicated that something has gone wrong - you should see the contents of the file you uploaded appearing there.

The intermittent nature of the failures here feels like it is something between the ComplianceUpload.exe process on the beacon and the IIS process on the parent server that is interrupting the communication for some reason.

I don't have many other concrete ideas of how to drill in to it further sorry. It might be interesting to compare the IIS log lines that appear on the parent server when a successful segment upload occurs and when a failed segment upload occurs. If the IIS log shows a success (HTTP 200) status when the upload log on the child beacon shows a failure, that would help to point to something between the servers interrupting the communication.

It would be interesting to be able to try the upload process without the firewall in place to see if that makes a difference. Although I expect that would not be easy/feasible to arrange!

(Did my reply solve the question? Click "ACCEPT AS SOLUTION" to help others find answers faster. Liked something? Click "KUDO". Anything expressed here is my own view and not necessarily that of my employer, Flexera.)
0 Kudos