%% date:: [[2022-09-28]] %% # [[Hybrid load testing]] Hybrid load testing is a mixture of [[Protocol-based load testing|protocol-based]] and [[Browser-based testing|browser-based]] [[Load Testing|load testing]]. Using hybrid load testing is the ideal way to do load testing; however, it can sometimes take a lot of time, effort, or money to do it. <iframe width="560" height="315" src="https://www.youtube.com/embed/SFag8ggkXKo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> *In search of the best pokémon: Browser automation and load testing in one script with k6 (Automation Guild 2022).* <iframe width="560" height="315" src="https://www.youtube.com/embed/KwlU6unOD9E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> *[Hybrid load testing: the best of testing using protocols and real browsers (PerfGuild 2020).](https://nicolevanderhoeven.com/blog/20200602-hybrid-load-testing/)* ## Advantages of hybrid load testing - You get a holistic view of [[Performance|Application Performance]]: both [[Frontend performance testing|front-end]] and [[Backend performance testing|back-end]]. - Hybrid load testing simulates high load while also tracking the real user experience of the application rendering on the browser. ## Common issues with hybrid load testing - Doing hybrid load testing often includes two different tools, which leads to duplicating effort required to maintain two scripts. - Introducing browser-based load testing increases costs because browser-based tests by their very nature consume more resources. - Results collation between two different tools can be a challenge, if it can be done at all. ## Approaches to hybrid load testing ### Use one tool to generate the load and another to run concurrently Use one tool (the protocol-based one, for efficiency) to simulate the majority of the load, and then run a handful of users with the other (the browser-based one) to get a sample of browser users as well. This approach reduces costs while still providing both types of metrics. The easiest and most lightweight version of this would be to use profiling tools like [[Lighthouse]] to get front-end metrics while a protocol-based tool is running. However, profiling is not really the same as testing: it involves a single static page rather than a series of actions, like you could use a browser-based script to emulate. A typical tool combination for this approach is [[JMeter]] and [[Selenium WebDriver]]. In particular, JMeter has a Webdriver plugin that drives Selenium scripts. However, this does mean that both JMeter and Selenium need to be running to run a single browser-based script, which isn't at all resource efficient. A better solution would be to use something like [[flood.io]] to orchestrate the execution of two different scripts and combine the results. Flood includes support for JMeter, [[Gatling]], [[Selenium WebDriver]], and [[Flood Element]] (a load testing wrapper around [[Playwright]]). In this situation, the combination of JMeter/Gatling and Flood Element would be what I'd recommend (mainly because Element is a more modern and efficient tool than Selenium). In this video, [[Marie Cruz]] and I talk about using two different tools for front-end and back-end performance: Cypress and k6. <iframe width="560" height="315" src="https://www.youtube.com/embed/yPmbZl2Qtus" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> ### Use protocol-based tool only, and then use a manual profiling tool Use one protocol-based testing tool to simulate virtual users, and while it's running, use a [[Performance profiling]] tool at the same time. A performance profiling tool is different from a testing tool in that it doesn't support complex scenarios. Instead, it accesses a page and analyzes it for front-end metrics. While these types of tools are not load testing tools, they can provide front-end performance metrics while your protocol-based load tests are executing. For many teams, this is the easiest way to improve the reach of performance testing without having to commit to another tool. ![[Performance profiling#Tools for performance profiling]] ### Use a browser-based tool for both Browser-based tools simulate real user interaction with browsers while also generating the underlying load on the application server, so a browser-based tool could actually be used on its own. The disadvantage to this is the cost: it is significantly more expensive to simulate ALL the users with a browser-based tool than it is to run two different tools, as in the prior approach. Another concern with this approach is that many browser-level tools don't track server-side metrics like response time, and instead operate by timing how long elements take to render. Of the browser-based tools on the market, I would recommend Playwright or Puppeteer (or Flood Element). ### Use a tool that can do both The ideal approach is to find a tool that can natively script both protocol-based and browser-based tests. The advantages of this approach are: - Simplicity: Using one tool and one language reduces effort in learning and script creation - Ease of maintenance: You only need to maintain a single test suite for both types of tests - Cost: Only one tool license is required (if premium offerings are required) The main disadvantage is that finding that tool can be difficult. Below are the only tools that I've found: #### [[k6 (tool)|k6]] k6 is primarily a protocol-based tool, but there is an extension to it called [[system/cards/Real browser testing with xk6-browser]] that allows for the scripting of browser-driven tests. This is a very promising approach, as it also means that all the metrics are unified under a single tool and format. However, xk6-browser is very much still in a public beta phase. #### [[Karate]] I haven't personally used this tool yet, so I can't really say much about it. However, it does seem to support both protocol-based (API) and browser-based ("functional testing") scripting.