%% date:: [[2022-10-21]] Parent:: [[Software Testing]] %% # [[Test criteria]] A test criterion is a way to judge whether your test has passed or failed. Test criteria can be applied to different levels of testing and development, and can, as a whole, determine the boundaries of "acceptable" for your application. Before you begin [[Software Testing|testing]], it's important to get a good understanding of _why_ you're doing load testing. Now is a good time to talk to your team and other stakeholders about what you're looking to get out of this round of testing. Are you reacting to a production incident that was performance-related? Are you trying to determine whether this application version degraded performance when compared to the version in production? Your reason for load testing may significantly affect the types of tests you run, so getting clarity on the purpose of load testing before you begin is essential. A fundamental idea in software testing in general is having criteria that help you decide whether a test has passed or failed. Different levels of criteria are applied to testing. High-level criteria directly address the purpose of testing, and low-level criteria have to do with specific functionalities in application code. Below, we'll discuss the following test criteria and the differences between them: - Requirements - SLAs - SLOs - Thresholds - Checks - Assertions ## [[Requirements]] Requirements are high-level testing criteria. Requirements related to application performance are often called non-functional requirements or NFRs. The term "non-functional" is outdated and misleading, because it implies that how well or poorly an application performs is not related to the function of an application. A better term is "operational". Operational requirements are statements about load testing, performance, and many other aspects of the application. They describe the focus of the testing cycle and the desired state of the application. ## [[Service Level Agreement (SLA)]] SLAs are test criteria that exist as a guarantee to users of how an application will perform. They set expectations for the quality, reliability, or stability that give potential or current customers a degree of assurance and confidence in the application. ## [[Service Level Objective (SLO)]] SLOs are another level of criteria within a project. SLO is a term borrowed from the related discipline of [[Site Reliability Engineering]], and deals primarily with observable metrics that describe the state of an application. For example, SLOs may involve: - CPU utilization - memory utilization - response time - throughput - Network I/O - latency #### [[SLAs vs SLOs vs SLIs]] ### [[Performance Thresholds]] In k6, thresholds are used to codify SLOs; that is, developers and testers using k6 are encouraged to define SLOs for each test as part of the [test script options](https://k6.io/docs/using-k6/thresholds/). In this context, thresholds are a way to implement testing criteria rather than testing criteria themselves. Thresholds could also be used to implement SLAs. ## [[Checks]] Checks are test criteria that are more closely related to the execution of the test script rather than the application. An example of a check is that a response returned should include "Login" in its body. This check verifies whether the request that a script sent returned an expected response. When a check fails, the test tool reports it, but otherwise, the test execution proceeds as normal. ## [[Assertions]] Assertions are similar to checks in that they also verify some aspect of the response message, but assertions go a step further and fail a request if they are not met. In this sense, assertions are like failure criteria, and could also be used to stop test execution entirely. ## Criteria Examples Here are some examples of what test criteria could look like at each of the levels we've discussed: _Operational requirement:_ The application should process requests with a 95th percentile response time of 2 seconds even while under the expected peak load of 1,000 requests per second. *SLA:* [The Company] ensures that 90% of all withdrawal requests from [Client] will be processed within 2 seconds. _SLO:_ The 95th percentile response time for [Application component] is 2 seconds or less. *Threshold:* The 95th percentile response time is less than or equal to 2 seconds. _Check:_ Verify whether the response time for this request is less than or equal to 2 seconds. *Assertion:* If the check failure rate is 10% or higher, fail the test.