cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

AWS inventory of accounts taking FOREVER.

Hello all,

I wanted to see if what i'm experience is expected.  We have an EC2 instance running the beacon and is assigned an IAM role to grant the policies it needs to read all the instance data from the various accounts.  Our Cloud OPS team has granted this across the board so it appears to be working as designed.   There are currently about 70+ accounts that our beacon has access to, providing info on almost 4000 instances.  

Best practice says that this job should be running every 30 minutes so that you capture "terminated" instances before they disappear and that association is lost.  The problem is that this job takes almost 1213 hours to complete.  

Here's the log snippet that shows how long each piece is taking on the reader.

---------------------

2022-01-05 10:00:54,054 [INFO ] Reading 'Computer' data from 'AWS Inventory' (ver 1.0) data source
2022-01-05 10:00:54,054 [INFO ] Reading data for 'Computer' successfully processed in 0 seconds.
2022-01-05 10:00:54,054 [INFO ] Reading 'VirtualMachine' data from 'AWS Inventory' (ver 1.0) data source
2022-01-05 10:00:54,069 [INFO ] CreateTempTableForEc2Instances (Execute on FNMP)
2022-01-05 10:00:54,069 [INFO ] ExecuteOnTarget steps only execute when connected to the target.
2022-01-05 10:00:54,069 [INFO ] Completed in 0 seconds.
2022-01-05 10:00:54,069 [INFO ] Get AWS EC2 dedicated hosts information (Transfer data from source 'AWS Inventory' to FNMP)
2022-01-05 15:34:52,353 [INFO ] Wrote 0 records to C:\ProgramData\Flexera Software\Beacon\IntermediateData\AWS Inventory\VirtualMachine--110--Get AWS EC2 dedicated hosts information.int.xml
2022-01-05 15:34:52,353 [INFO ] Successfully processed in 5 hours, 33 minutes, 57 seconds.
2022-01-05 15:34:52,353 [INFO ] Get AWS EC2 instance information (Transfer data from source 'AWS Inventory' to FNMP)
2022-01-05 18:11:40,311 [INFO ] Wrote 3540 records to C:\ProgramData\Flexera Software\Beacon\IntermediateData\AWS Inventory\VirtualMachine--120--Get AWS EC2 instance information.int.xml
2022-01-05 18:11:40,311 [INFO ] Successfully processed in 2 hours, 36 minutes, 47 seconds.
2022-01-05 18:11:40,311 [INFO ] Get AWS EC2 reserved instance information (Transfer data from source 'AWS Inventory' to FNMP)
2022-01-05 20:51:43,311 [INFO ] Wrote 757 records to C:\ProgramData\Flexera Software\Beacon\IntermediateData\AWS Inventory\VirtualMachine--130--Get AWS EC2 reserved instance information.int.xml
2022-01-05 20:51:43,311 [INFO ] Successfully processed in 2 hours, 40 minutes, 2 seconds.
2022-01-05 20:51:43,311 [INFO ] Get AWS EC2 account information (Transfer data from source 'AWS Inventory' to FNMP)
2022-01-05 20:52:30,109 [INFO ] Wrote 2 records to C:\ProgramData\Flexera Software\Beacon\IntermediateData\AWS Inventory\VirtualMachine--140--Get AWS EC2 account information.int.xml
2022-01-05 20:52:30,109 [INFO ] Successfully processed in 46 seconds.
2022-01-05 20:52:30,109 [INFO ] Populate the AWS EC2 dedicated hosts data on the target database (Execute on FNMP)
2022-01-05 20:52:30,109 [INFO ] ExecuteOnTarget steps only execute when connected to the target.
2022-01-05 20:52:30,109 [INFO ] Completed in 0 seconds.
2022-01-05 20:52:30,109 [INFO ] Populate the AWS EC2 instance data on the target database (Execute on FNMP)
2022-01-05 20:52:30,109 [INFO ] ExecuteOnTarget steps only execute when connected to the target.
2022-01-05 20:52:30,109 [INFO ] Completed in 0 seconds.
2022-01-05 20:52:30,109 [INFO ] Populate the AWS EC2 reserved instance data on the target database (Execute on FNMP)
2022-01-05 20:52:30,109 [INFO ] ExecuteOnTarget steps only execute when connected to the target.
2022-01-05 20:52:30,109 [INFO ] Completed in 0 seconds.
2022-01-05 20:52:30,109 [INFO ] Set terminated instances on the staging tables (Execute on FNMP)
2022-01-05 20:52:30,109 [INFO ] ExecuteOnTarget steps only execute when connected to the target.
2022-01-05 20:52:30,109 [INFO ] Completed in 0 seconds.
2022-01-05 20:52:30,109 [INFO ] CleanUpTargetTempTable (Execute on FNMP)
2022-01-05 20:52:30,109 [INFO ] ExecuteOnTarget steps only execute when connected to the target.
2022-01-05 20:52:30,109 [INFO ] Completed in 0 seconds.
2022-01-05 20:52:30,109 [INFO ] Reading data for 'VirtualMachine' successfully processed in 10 hours, 51 minutes, 33 seconds.

----------------------------

If it only runs once a day, i'm fine with that....i just want to make sure there isn't another issue, and that this is expected with these numbers.

We have another environment equal to this, and another that is double to triple this size....so i'm concerned about the larger environment when that time comes.

Thanks.

(1) Solution
ChrisG
By Community Manager Community Manager
Community Manager

The AWS reading process can take a long time when running against many AWS accounts. See the following article for a reference: Known Issue: Import process reading data from AWS may run for a long time when there are many AWS accounts on the one connection (IOJ-2160646).

Also see the following forum thread for some discussion about working with many AWS accounts: AWS Connector - scale to 500+ accounts.

(Did my reply solve the question? Click "ACCEPT AS SOLUTION" to help others find answers faster. Liked something? Click "KUDO". Anything expressed here is my own view and not necessarily that of my employer, Flexera.)

View solution in original post

(3) Replies

........and the long part, the information on Dedicated hosts, isn't even returning any records according to the log.

Okay...so is anyone using AWS inventory and it works within 30 minutes to go through multiple accounts?   or is just no one using this? 

ChrisG
By Community Manager Community Manager
Community Manager

The AWS reading process can take a long time when running against many AWS accounts. See the following article for a reference: Known Issue: Import process reading data from AWS may run for a long time when there are many AWS accounts on the one connection (IOJ-2160646).

Also see the following forum thread for some discussion about working with many AWS accounts: AWS Connector - scale to 500+ accounts.

(Did my reply solve the question? Click "ACCEPT AS SOLUTION" to help others find answers faster. Liked something? Click "KUDO". Anything expressed here is my own view and not necessarily that of my employer, Flexera.)