It sounds like with that many agents attempting to connect to a single beacon your IIS server is likely indicating an overload condition when too many servers are trying to upload their inventory file at the same instant. You should be able to identify this condition by reviewing the IIS logs for "503" errors. By default IIS allows 1000 concurrent connections. If you find there are "503" errors in the logs, you can try to increase the "Queue Length" on your application pool (it's in "advanced settings").
I'm working through some tuning of this myself. We have 30K+ agents reporting into 2 beacons. My goal is to minimize the "503" errors in the logs and ultimately to receive all of the agent files every day without pushing the web server beyond its allocated capacity. Both beacons are virtual but I'd rather not spin up additional beacons if there could be untapped capacity already available.