%% date:: [[2022-11-02]] %% # [[Load test execution environments]] Which environment should you [[Executing load tests|run]] your [[Load Testing|load tests]] in? [[Production-like environment]]s are ideal, but testing in pre-production environments and in production environments both add value. ## The case for testing in pre-production If you have the luxury of multiple environments, it's best practice to run tests as early as you can, which means running them in pre-production environments: staging, test, system integration testing, user acceptance testing, and production replica environments. Finding performance defects early can save a lot of time and effort (and reputation) later on. The main advantage of testing in test environments is that you can often afford to be more aggressive with your tests. You can also run tests more frequently, needing only to sort out environment contention internally, rather than being beholden to real user behaviour. However, testing in pre-production environments also makes your load profile more critical to get right. It's common to see production incidents occur that were never identified in pre-production testing, usually as a result of the test profile being not quite as realistic as it could have been. ## The case for testing in production At minimum, every team has one environment: production. Often that's the *only* environment that you might have. Consider large companies like Netflix and Amazon, whose networks are too big to feasibly replicate in their entirety. How do they get their testing done? There's still a lot of testing that can be done in production without disrupting regular operations. Consider the following options: - Use lower levels of load when running load tests during peak hours. - Schedule load tests for off-peak hours, when the least amount of users would potentially be affected. - Choose load test types that are less risky: for example, consider saving stress or break point tests for pre-production. - Publicly announce a short window during which your system will be down for maintenance and testing. - Use techniques like synthetic monitoring, which makes requests in a more staggered way than typical load testing. - Use real user monitoring tools to get snapshots of user performance at load. - Ensure that your observability stack is working at peak efficiency. In general, load testing in production is less about generating high amounts of traffic (since production traffic exists alongside test traffic) and more about measuring and quantifying user experience.