%% date:: [[2023-02-22]] parent:: [[Load Testing]], [[Performance Testing]] %% # [[Tips for running large load tests]] Created for [[Tricentis]] [[Flood (company)]] as things to tell customers who say they're running large tests, especially [[Hotstar]]. - [[Communicate planned load tests to stakeholders.]] - "Let us know in advance when you're planning to run tests involving over 1000 nodes. This will enable us to make sure that you get the support you need." - [[Remove unnecessary files and test data from your test archives.]] - "Minimize the size of your script and data files. This will be transferred to every node, so very large libraries will affect how long your test takes to start up. Remove all unnecessary libraries." - [[Consider starting tests on load generators as they become available instead of waiting for them all.]] - "The default behaviour on Flood is that we will only start a test after all nodes on all grids are up. However, sometimes when you start many nodes, AWS has trouble fulfilling your request. To prevent this, we are able to enable a switch on our end that will allow your test to start on individual nodes while waiting for others to start. If you're okay with this, we can turn this on for you. We've found that this tends to be better for larger tests." - [[Consider using On Demand pricing rather than Spot Pricing]] - "Consider using normal On Demand instances rather than spot instances. While spot instances can save money on provisioning costs, it also means that if someone bids higher on spot instances than you did, those grids can be taken from you and allocated to someone else. This obviously isn't ideal in a load test, so if you do have issues starting enough grids, I'd suggest increasing your spot bid or just using normal On Demand EC2 instances." - [[Start all cloud instances before beginning a load test.]] - "Start all the grids through the Grids screen and wait for all the nodes to go green. It's normal for this to take 20+ minutes. If there are certain regions that don't come up during that time, it's likely that you've chosen a region with smaller capacity. Stop those grids and start an equal number of grids in another region until you have the right number of grids that you need for your test." - "After all grids are started and green, start your test, selecting all those grids. Allow some time for the flood to start. Even though the grids are started, your files will need to be transferred over." - [[Do not manually stop or start cloud instances when starting them programmatically.]] - "Do not manually stop or start instances through the AWS console. This will lead to Flood not picking up the new instances, which means you'll have fewer nodes running your test than you expected. Along with this, do not cancel spot instance requests. The Auto Scale Group will automatically start up new instances to replace instances with issues."