%%
date:: [[2021-05-26]], [[2022-08-12]]
%%
# [[Browser-based vs protocol-based testing]]
Below are some considerations when deciding between [[Browser-based testing|browser-based]] and [[Protocol-based load testing|protocol-based]] testing.
In this presentation at PerfGuild 2020, I illustrate the differences between browser-based and protocol-based testing and how to combine both using [[JMeter]] and [[Flood Element]].
<iframe width="560" height="315" src="https://www.youtube.com/embed/KwlU6unOD9E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
And here's a video from Automation Guild 2022, where I talk about hwo to do browser automation and load testing in one script with [[k6 (tool)|k6]] with [[k6 browser]]:
<iframe width="560" height="315" src="https://www.youtube.com/embed/SFag8ggkXKo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
## Realism
Browser-based testing is [[Making load testing scripts more realistic|more realistic]] than [[Protocol-based load testing]] because it tests [[Frontend performance testing|front-end performance]] as well as [[Backend performance testing|back-end performance]]: it tests what a users sees, how objects are rendered on a page, etc., in addition to testing how quickly a server returns a response.
## Reusability
Browser-based tools also have the advantage of being reusable for both functional tests and nonfunctional tests, reducing the amount of tests that need to be created and maintained. Protocol-based testing cannot be used to test a UI, so it almost always requires the creation and maintenance of a separate test suite of cases.
## Ease
Browser-based testing can be easier to get started with than protocol-based testing because browser-based testing scripts are written in terms of interactions. For example, here's a snippet from a script in [[Flood Element]] (a browser-based tool):
```js
//Click Next.
let nextButton = await browser.findElement('.btn')
await nextButton.click()
```
And here's the same thing in [[k6 (tool)|k6]], a protocol-based tool:
```js
let i = 0;
let data = {
authenticity_token: authTokenVal,
"challenger[step_id]": stepIDVal,
"challenger[step_number]": "4",
commit: "Next",
};
challengerOrders.forEach(function(item){
data[item] = challengerValues[i];
i++
});
let res = http.post(domain + "/start", data);
```
Granted, this example was chosen to illustrate situations where browser-based tools can make scripting easier: apps with dynamic values that are passed along with requests.
## It's resource-hungry
Browser-based testing requires an instance (or several) of a browser to be carried out, which uses up more resources (CPU or memory) to do. This means that it's great for running a few users, but for load testing, where hundreds or thousands of users are common, browser-based testing requires more machines than protocol-based testing does. This also leads to the next point...
## It's more expensive
More machines required = more cost. As an example, an [[AWS]] m5.xlarge machine can run about 5 [[Selenium WebDriver]] (browser-based) users comfortably, OR about 1000 [[JMeter]] (protocol-based) users. Therefore, simulating 1000 users would require 200 machines using Selenium, and only 1 using JMeter. This 200x difference also applies to the cost of provisioning the machine(s) used as load generators.
## Its use case is end-to-end only
For load testing, browser-based testing is only usable for end-to-end (measuring the performance from the point of view of the user all the way through the application servers that are involved). By contrast, protocol-based testing can be used to test only specific components, so it's more usable for unit testing, or for testing an application that isn't completely integrated or functional end-to-end yet.