AWS Connector - scale to 500+ accounts
Anyone else that have deployed the AWS Connector for more than 100+ accounts?
We currently use "AWS beacon installed on EC2 instance using IAM Roles"
Our cloud team tells me that this setup will only scale up to about 75 accounts and we have 500+. The reason is that we need to list all accounts that the connector shall look into, and in AWS this role or policy has limitation in number of characters, which will only cover 75 entries.
I was thinking if we need instead use one of the other methods of connecting or if we need to have multiple beacons running these jobs or configure several policies or roles.
@ccrawford any thoughts how to scale too 500+ accounts?
Re: AWS Connector - scale to 500+ accounts
Hey @mag00_75 ,
Thanks for reaching out.
It should be possible to create multiple policies, each to the character limits, encompassing several accounts and then attaching all of these policies to the role which is used by the server running the beacon.
For example, here is a role used by a server of ours with the beacon installed. This beacon uses EC2 metadata (no access keys needed), same as you. As you can see, there are two policies applied, each with their own list of ARNs:
And when the adapter runs it will attempt to assume both roles in each policy:
One issue you might encounter is with the 'Maximum Session Duration'. Because the adapter discovers all associated roles and then assumes them, thus marking the start of our session, the delay between fully populating and assuming all the roles before the inventory can complete may lead to the session becoming invalidated. In this case you can try increasing the maximum session duration.
However if this doesn't work you can request an enhancement to the adapter, so that the roles are assumed just before they are inventoried.
Alternatively, you can use one of the other methods but bare in mind these are less reliable and less performant than using EC2 metadata as you currently are.
I think if after performing the above, of splitting out the policies and seeing if the inventory completes within a reasonable time before session expiry... if you are struggling to implement it then the best course of action would probably be the above enhancement.
― Jean-Luc Picard