Emergent load testing - TestCon Europe 2023

note: I'm Nicole van der Hoeven. I'm a Senior Developer Advocate at Grafana Labs. I've been in performance testing for 12 years. Today I'm talking to you about emergent load testing. I'll be talking about what emergence is, and I'll be raising questions (and some potential answers) about how to apply it to our code, our environments, and our tests. I say "potential" because while I do have code for certain parts, you should consider this presentation conceptual-- it's more about an approach to testing than a fleshed-out framework. I'm also going to be talking about a lot of tools in this presentation, but all of them are free and open source. ---  <style> .with-border{ border: 2px solid red; } </style> <grid drag="50 50" drop="left"> ## Emergence the evolution of the whole beyond its *parts* in unexpected ways. </grid> <grid drag="40 100" drop="right"> ![[/assets/emergence-spill.png|500]] </grid> note: This is my definition of emergence. Emergence is the evolution of the whole beyond its parts in unexpected ways. So it talks about changes: new, complex behaviours of the system that arise despite not having been displayed by any of its parts. - These changes are called an *evolution* because they came about iteratively. - *The whole* refers to an entire system, something larger that is formed. - *The parts* refer to individuals, or components, within the system. - Yet the changes are *unexpected*: not just because the changes are surprising, but because there was nobody there to expect them in the first place. How can that be? Let's look at an example of emergence in nature. ---  ![[/assets/alex-wild-ant-bridge.jpg|800]] note: Ants are fascinating creatures. Individually, they have simplistic brains and are driven by instinct. It's actually a bit misleading to say they have an ant "queen", because they don't have leaders the way we do. The ant queen is really no different from any other drone: she, or they, have a job, and they're just focused on doing it. Nobody's really coordinating ants or planning out the strategy for the entire colony. Yet ants as a colony manage to display some impressive behaviour. In this picture, they're forming a bridge with their own bodies so that the colony can get to a source of food not otherwise reachable. The amount of coordination is tremendous. It's almost as if the colony has a mind of its own that's somehow larger and more complex than the sum of the individual ants' brains that are part of it. This is an example of emergence. When I read about this, it fired up my imagination about how this can be applied to software that we build. It's a bit difficult to pin RIGHT what causes emergence, though, because by its very nature it's something spontaneous. But there are still some factors that can *facilitate* emergence. ---  ## Factors that facilitate emergence + A system of **[[Organized Complexity|organized complexity]]** + Diversity + Bidirectional feedback loops + Agency note: - (RIGHT) Emergence requires a system of organized complexity. - If something were entirely random, devoid of any internal logic, any larger change that occurs would be purely coincidental. That's not what we're looking for. - It also can't be too simple: if it were a simple problem, then no emergence needs to occur; simple problems have simple solutions. - (RIGHT) Emergence requires diversity: - Diversity in sample size ensures that whatever we're measuring, or whatever population we're analysing, there are enough of them that any observation we make about them can be applied to the population as a whole. - Diversity in approach means that the individual ants can't be clones of each other, all doing exactly the same thing. - (RIGHT) Emergence requires feedback, and it has to go both ways: an ant is constantly getting feedback about how well its efforts have fared, but it also passes on that information to other ants. With the feedback, ants form a picture of the state of the colony as a whole. - (RIGHT) Emergence requires agency: every ant should be able to act on its own to *some* degree. ---  note: So, time to apply this to software. What if software could be *evolved*, not *written*? What would that look like? What would emergent code, or emergent systems, look like? But you might already be thinking, "isn't that just artificial intelligence"? ---  ## Is emergent software AI? ![[/assets/Midjourney made this thumbnail.png]] note: The answer is no. Artificial intelligence is not the same as emergent software. - Good AI is *explainable*: the outputs of the system, and how it got from one step to another, must have a logical explanation. But think about natural selection, which is another form emergence in nature. For example, the Galapagos batfish suddenly evolved a feature that looks completely out of place on a fish: red lips. Why? No one really knows. - Also, there's a wide range of things we now call "AI". Sometimes just a particularly clever or complex bit of code is called "AI", but that code isn't *emergent* unless NEW behaviour comes from it unplanned. AI *can* be emergent, but not always. So if emergent software isn't AI, then what could it look like? First, let's talk about what emergent code looks like. Then, we'll talk about what emergent tests look like. ---  <grid drag="50 50" drop="left"> ![[/assets/icon-command-line.png]] </grid> <grid drag="50 50" drop="right"> ### Imperative code + Sequential + Instructional + Precise </grid> note: Imperative code is the kind of code that we commonly see today. It's (RIGHT) sequential and (RIGHT) instructional: it consists of exactly the things that we want to have executed, in order. It's also (RIGHT) precise: there's no room for ambiguity. This kind of code is like the Borg, from Star Trek: there may be many moving parts to it, but every part is a drone, controlled by a central mind, the queen. This is not emergent. (RIGHT) ---  <grid drag="50 50" drop="left"> ![[/assets/icon-target.png]] </grid> <grid drag="50 50" drop="right"> ### Declarative code + Goal-oriented + Defined output + Limited instructions </grid> note: The next kind of code that already exists is declarative. We more and more often use this to describe our environments. - (RIGHT) it is *goal-oriented*: Declarative code describes the end result, but now how to get there. - (RIGHT) It has a defined output: there's a state that you set at the end that is the intended output. - (RIGHT) Yet it has limited instructions: you're not spoonfeeding it what to do at every step; instead you give it some room to get to the goal on its own. This type of code is kind of like the puzzles in [[Zelda - Breath of the Wild]]. There's one end result, but there are different ways you could get there. This is closer to emergence, but it's still not quite there. Remember that emergence is about new and unexpected behaviours arising from the whole that the parts didn't have. That's not really happening here. So what kind of code could facilitate that? (RIGHT) ---  <grid drag="50 50" drop="left"> ![[/assets/icon-command-line.png]] </grid> <grid drag="50 50" drop="right"> ### Generative code + Given starting point + Determines a range of affordances + Open-ended </grid> note: Generative code is given a sufficiently large sample of input and autonomously delivers output that is surprising. It is capable of a wide range of output, and is often used in algorithms for Artificial Intelligence. This still isn't entirely new. For example, the [[Actor model]] in computer science describes an actor that can make its own decisions and formulate its own responses, but can only communicate with other actors. - (RIGHT) Generative code provides input up front. Here's where you are, and here's what the situation is. - (RIGHT) Then it determines a range of afforances. Affordances are possible ways of interacting with an object, and not just the *intended ways*. For example, a chair is intended to be sat on, but being able to stand on it or straddle it are affordances as well. - (RIGHT) And generative code is open-ended. There *is* no intended end result, just a range of possibilities. If imperative code is like the borg and declarative code is like Zelda: Breath of the Wild, then generative code is kind of like a [[Tabletop Roleplaying Games|TTRPG]] like D&D. I play in several campaigns a week. D&D starts out like a video game: there might be a boss in front of you, but an almost infinite number of situations you could end up in. The monster could end up dead, you could end up dead, you could make friends with it, you could distract it, you could sneak past it and steal the treasure without it knowing, you could charm it... Generative code, then, defines parameters of [[Play]] rather than defining the end goal or how to do it. Generative code is the kind of code that facilitates emergence. (RIGHT) ---  <grid drag="50 50" drop="left"> Imperative ![[/assets/icon-command-line.png|200]] + what, how </grid> <grid drag="50 50" drop="center"> Declarative ![[/assets/icon-target.png|200]] + what </grid> <grid drag="50 50" drop="right"> Generative ![[/assets/icon-ant-red.png|200]] + where </grid> note: (RIGHT) So imperative code explains what the goal is and how to get it done. (RIGHT) Declarative code sets the *what* but not the *how*. (RIGHT) Generative code doesn't set either of those-- instead, it sets the *where*. (RIGHT) ---  note: To some degree, our code and our systems already display some elements of all of these. But I'm not a developer OR an ops engineer. I consider myself to be first and foremost a tester. So when I look at something like this, I think about the testing. And when looking at emergent systems and generative code, my question is: why hasn't our testing caught up? ---  note: Our systems are increasingly emergent. Our code is increasingly generative, in that we're handing over more and more of the logic for how to respond to events to machines. But our tests are still predominantly imperative. We're still treating tests like they need to be absolutely sequential, instructional, and precise. It's a bit like trying to test art like this by measuring the ratio of red to black paint. Yes, that information can be useful, too. But those kinds of tests fail to capture a lot of the soul of a system like this. How could we encourage our tests-- in particular, load tests, to be emergent? (RIGHT) ---  ## An emergent load test... + does not orient around an expectation + is more than just a load test + facilitates two-way information flow + has enough agency to behave in unexpected ways note: (RIGHT) *does not orient around an expectation.* An emergent load test isn't written in such a way that it seeks to verify a hypothesis. It tests a broader area of play rather than heading for the goal posts. (RIGHT) *is more than just a load test.* An emergent load test goes beyond its programming. It may include a load test, but it may also include an automated browser functional test, or some chaos experiments. An emergent load test is capable of more activities than what we might typically ascribe to a load test. (RIGHT) *facilitates two-way information flow.* Typically, we think of a load test as something that generates data and gives us output in the form of test results. An emergent load test is one that is capable of gathering information about the system under test. But more than that... (RIGHT) *has enough agency to behave in unexpected ways.* An emergent load test can also *act* on that information. It can adapt its behaviour to what's happening in the system under test. It can change what it's going to do next based on how the system has behaved to previous interventions. Well, you can imagine that this is a pretty tall order. This sounds nice in theory, but there are a few things holding us back. ---  ## What keeps us from doing emergent load testing? + Requirements + Limited affordances of load testing tools + Complexity + Time note: (RIGHT) *Requirements*. It would be nice to think that we could have free reign to improve overall application quality, whatever that might mean. In reality, most of us get given requirements, and have to code to those specifications. This restricts not just what we test, but also *how* we test, and the test types we choose to run. I'm going to show you a way to reframe requirements to make tests more emergent. (RIGHT) *Limited affordances of load testing tools.* Most tools are unitaskers. They've been developed for one thing, might even do it well, and that's that. Emergent load testing requires a lot more flexibility, however. We need tools that can do multiple things well-- otherwise we'd be maintaining several scripts across several tools, and at some point it's just going to be infeasible. I'll show you a specific tool, k6, that I've found useful for this purpose. (RIGHT) *Complexity.* The more we add to our testing stack, the harder it gets to figure out what's actually going on in the black boxes that we test. For us to do emergent load testing, we'd need to find a way to see exactly what's going on at all times. (RIGHT) *Time.* I'm not going to lie-- emergent load testing provides many benefits and improvements in the quality of the applications we test, but it's not a quick fix. It's something that you need to invest time and effort into, and the benefits are reaped over the long term. ---  ### Affordances, not requirements - API endpoints - frontend UI - infrastructure - components and services - output and telemetry note: In design, an affordance is a way that we can interact with an object. For example, a chair *affords* sitting-- its primary purpose. However, a chair technically affords standing as well, or as my nephew has found, becoming a pretend car when turned on its side. So affordances aren't just requirements or intended purposes for things. When I think about the ways in which a load test interacts with a system, these are some of the affordances I think it can play with. We can look at these factors of the overall performance of a system as a range of affordances: the *where* upon which we can do generative (and hopefully emergent) load testing. Performance isn't just about speed. In the next few slides, I'm going to delve a bit more into the technical aspects of this idea as a proof of concept of how exactly emergent load testing might work. To do that, I am going to talk about a tool that Grafana, the company I work for, does build-- but if you can find another tool that does all of the same things, then the concepts will apply to that tool too. And, in fact, what I'm about to talk about can only be done with the free, open source version. ---  <grid drag="70 80" drop="left"> ![[/assets/k6-red.png]] </grid> <grid drag="50 50" drop="right"> ### Testing multitool - open source - written in Go - scripted in JavaScript - extensible - plays well with others </grid> note: (23 MINS) That tool is called k6. (k6.io) k6 is a free and open source tool that's written in Go, with all the performance benefits that confers, but the scripting is done in JavaScript to take advantage of some of the convenience of an interpreted language that is near ubiquitous for testers, developers, and ops engineers alike. It's got an open source extension ecosystem which is essential to this discussion because I'm going to be talking about doing some funky stuff with it. And it plays well with others: k6 is built to be composable and modular, and because we were acquired by Grafana in 2021, it has the advantage of working well with the entire Grafana observability stack. ---  #### API load test ```javascript import { check } from 'k6' import http from 'k6/http' export default function() { const res = http.get('https://otel-demo.field-eng.grafana.net/') check(res, { 'status is 200': res => res.status === 200 }) } ``` `k6 run test.js` note: This is an example of a typical, protocol-level test in k6, written in JavaScript. It makes a single HTTP request and then does a check to see whether the HTTP response code that was returned for that request was a 200. This is a simple script that tests one specific thing, but it's not the breadth of what k6 can do. In the next few slides, I'm going to show you how it can be used to test some of the other affordances we talked about. Then, I'm going to show you how I think it could be used as an emergent load testing tool. To run this test, all you need to do is execute the command at the bottom: `k6 run test.js` ---  #### k6 browser ```javascript import { check } from 'k6'; import { browser } from 'k6/experimental/browser'; export const options = { scenarios: { ui: { executor: 'shared-iterations', options: { browser: { type: 'chromium', }, }, }, }, thresholds: { checks: ["rate==1.0"] } } export default async function () { const page = browser.newPage(); try { await page.goto('https://test.k6.io/my_messages.php'); page.locator('input[name="login"]').type('admin'); page.locator('input[name="password"]').type('123'); const submitButton = page.locator('input[type="submit"]'); await Promise.all([page.waitForNavigation(), submitButton.click()]); check(page, { 'header': p => p.locator('h2').textContent() == 'Welcome, admin!', }); } finally { page.close(); } } ``` `K6_BROWSER_ENABLED=true k6 run script.js` note: Browser testing is another affordance of an application. It can tell us things that API tests, which run on the protocol level, can't. Browser tests in k6 can give us metrics like the first meaningful paint, or how long it takes for the browser to render important elements of a webpage such as those we see above the fold. ---  #### xk6-kubernetes ```javascript import { Kubernetes } from 'k6/x/kubernetes'; const manifest = ` apiVersion: batch/v1 kind: Job metadata: name: busybox namespace: testns spec: template: spec: containers: - name: busybox image: busybox command: ["sleep", "300"] restartPolicy: Never ` export default function () { const kubernetes = new Kubernetes(); kubernetes.apply(manifest) const jobs = kubernetes.list("Job", "testns"); console.log(`${jobs.length} Jobs found:`); pods.map(function(job) { console.log(` ${job.metadata.name}`) }); } ``` note: The xk6-kubernetes extension lets you interact with a Kubernetes cluster from within a k6 script. This gives a test a lot of affordances to play with. Imagine a test script that can spin up and modify the environment it's testing. ---  #### xk6-disruptor ```javascript import { PodDisruptor } from 'k6/x/disruptor'; import http from 'k6/http'; export default function(data) { http.get(`http://${data.SVC_IP}/delay/0.1`); } export function disrupt(data) { if (__ENV.SKIP_FAULTS == "1") { return } const selector = { namespace: namespace, select: { labels: { app: "httpbin" } } } const podDisruptor = new PodDisruptor(selector) // delay traffic from one random replica of the deployment const fault = { averageDelay: "50ms", errorCode: 500, errorRate: 0.1 } podDisruptor.injectHTTPFaults(fault, "30s") } ``` `xk6-disruptor run --env SKIP_FAULTS=1 --env SVC_IP=$SVC_IP disrupt-pod.js ` note: In a similar vein, the xk6-disruptor is an extension that interacts with Kubernetes clusters in particular to inject HTTP faults into pods (selected randomly or according to criteria). It can do things like delay traffic or change the error code or body that's returned. So we've talked about a lot of the affordances of an application, and how we could interact with them. But how do we put it all together? (RIGHT) ---  #### Scenarios ```javascript import { protocol } from './protocol.js'; import { browser } from './browser.js'; import { disruptor } from './chaos.js'; export const options = { scenarios: { checkout_http: { executor: 'ramping-vus', exec: 'http', startVUs: 0, stages: [ { duration: '1m', target: 10 }, { duration: '3m', target: 10 }, { duration: '1m', target: 0}, ] }, checkout_browser: { executor: 'per-vu-iterations', exec: 'xk6browser', vus: 10, iterations: 1, startTime: '1m', }, disruptor: { executor: 'shared-iterations', exec: 'disruptor', vus: 1, iterations: 1, } } } export function http () { protocol(); } export function xk6browser () { browser(); } export function disruptor () { disruptor(); } ``` note: Now, one way to string all of these together is to put them in a scenario. This is what that would like within a k6 script. Each scenario can test a different type of affordance, and you can assign different executors that change the test parameters - how many virtual users it runs with, its duration, the number of iterations, etc. You might be thinking, though-- it's hard enough to keep track of ONE load test, and all the errors happening in it. Now we want to add different types of tests? How will we track all that? (RIGHT) ---  <grid drag="50 50" drop="left"> ![[/assets/red-magnifying-glass.png]] </grid> <grid drag="50 50" drop="right"> ### Observing complexity - Logs - Metrics - Traces </grid> ---  #### Logs: xk6-loki ```javascript import sleep from 'k6'; import loki from 'k6/x/loki'; /** * URL used for push and query requests * Path is automatically appended by the client * @constant {string} */ const BASE_URL = `http://localhost:3100`; /** * Client timeout for read and write in milliseconds * @constant {number} */ const timeout = 5000; /** * Ratio between Protobuf and JSON encoded payloads when pushing logs to Loki * @constant {number} */ const ratio = 0.5; /** * Cardinality for labels * @constant {object} */ const cardinality = { "app": 5, "namespace": 5 }; /** * Execution options */ export const options = { vus: 10, iterations: 10, }; /** * Create configuration object */ const conf = new loki.Config(BASE_URL, timeout, ratio, cardinality); /** * Create Loki client */ const client = new loki.Client(conf); export default () => { // Push a batch of 2 streams with a payload size between 500KB and 1MB let res = client.pushParameterized(2, 512 * 1024, 1024 * 1024); // A successful push request returns HTTP status 204 check(res, { 'successful write': (res) => res.status == 204 }); sleep(1); } ``` https://github.com/grafana/xk6-loki https://grafana.com/docs/loki/latest/send-data/k6/ note: k6 already sends ---  #### Metrics: xk6-dashboard ![[/assets/xk6-dashboard.png|600]] https://github.com/grafana/xk6-dashboard note: If you want ready-made visualizations, you can use the xk6-dashboard extension for k6, which gives you graphs like these out of the box and also lets you generate an HTML report of the whole run. But maybe you want more customizability, and you want to send your metrics elsewhere. ---  #### Metrics: k6 Prometheus Remote Write `K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9090/api/v1/write \ k6 run -o experimental-prometheus-rw script.js` note: k6 can also send test results to a Prometheus Remote Write endpoint, just by adding an output flag. That means both the protocol-level and browser-level metrics could be going to the same place. If you're not familiar with Prometheus, it is a tool for capturing metrics and alerting based on them. It's also a time-series DB, which means you can send load testing metrics, like response time, directly to an instance of Prometheus. From there, you can use something like Grafana to visualize that data and customize graphs and dashboards to your heart's content. ---  #### Traces: k6 tracing ```javascript import { check } from 'k6' import http from 'k6/http' import tracing from "k6/experimental/tracing"; tracing.instrumentHTTP({ propagator: "w3c", }); export default function() { const res = http.get('https://otel-demo.field-eng.grafana.net/', { headers: { "X-Example-Header": "instrumented/get", }, }); check(res, { 'status is 200': res => res.status === 200 }) } ``` https://grafana.com/docs/grafana-cloud/k6/analyze-results/integration-with-grafana-cloud-traces/ note: (15 MINS) To take things further, if you've already instrumented your system using the OpenTelemetry Collector or Grafana Agent, the k6 tracing module lets you instrument HTTP requests, so that each one emits traces as it's executed. ---  ![[/assets/gck6-distributed-tracing.png]] note: Visualized in Grafana, it looks kind of like this. ---  #### k6 redis ```javascript import { check } from "k6"; import http from "k6/http"; import redis from "k6/experimental/redis"; import { textSummary } from "https://jslib.k6.io/k6-summary/0.0.1/index.js"; export const options = { scenarios: { usingRedisData: { executor: "shared-iterations", vus: 10, iterations: 200, exec: "measureUsingRedisData", }, }, }; // Instantiate a new redis client const redisClient = new redis.Client({ addrs: __ENV.REDIS_ADDRS.split(",") || new Array("localhost:6379"), // in the form of "host:port", separated by commas password: __ENV.REDIS_PASSWORD || "", }); // Prepare an array of crocodile ids for later use // in the context of the measureUsingRedisData function. const crocodileIDs = new Array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9); export function setup() { redisClient.sadd("crocodile_ids", ...crocodileIDs); } export function measureUsingRedisData() { // Pick a random crocodile id from the dedicated redis set, // we have filled in setup(). redisClient .srandmember("crocodile_ids") .then((randomID) => { const url = `https://test-api.k6.io/public/crocodiles/${randomID}`; const res = http.get(url); check(res, { "status is 200": (r) => r.status === 200, "content-type is application/json": (r) => r.headers["Content-Type"] === "application/json", }); return url; }) .then((url) => redisClient.hincrby("k6_crocodile_fetched", url, 1)); } export function tearRIGHT() { redisClient.del("crocodile_ids"); } export function handleSummary(data) { redisClient .hgetall("k6_crocodile_fetched") .then((fetched) => Object.assign(data, { k6_crocodile_fetched: fetched })) .then((data) => redisClient.set(`k6_report_${Date.now()}`, JSON.stringify(data)) ) .then(() => redisClient.del("k6_crocodile_fetched")); return { stdout: textSummary(data, { indent: " ", enableColors: true }), }; } ``` note: k6 Redis is an experimental module that lets you use Redis as a data store during the test. Among other things, this could provide the limited and networked communication that is one of the factors that facilitate emergence. ---  ### Affordances - API endpoints: [[HTTP]] module - frontend UI: [[k6 browser]] - infrastructure: [[xk6-kubernetes]] - components and services: [[xk6-disruptor]] - output and telemetry: - Logs: [[Grafana Loki|Loki]] - Metrics: [[xk6-dashboard]] - Metrics: [[k6 Prometheus Remote Write]] - Traces: [[k6 tracing]] - (Storage): [[k6 redis]] note: 30 MIN To recap, here are the things that can be done with a single k6 test. Each one represents an affordance, a way to interact with the system under test. We now have the components we need to play with the affordances of the system. But how do we get from here to emergence? Well, the truth is that here's where we depart from the practical and go into theory. I made a few attempts at mocking up some code for this, but I think in the end, this is just going to be one of those things that's too application specific to provide guidance for. But here's the idea. ---  <grid drag="50 30" drop="top"> #### Agency </grid> <grid drag="50 50" drop="left"> ##### Feedback - response time - response code - uptime - latency - processing time - number of pods - etc... </grid> <grid drag="50 50" drop="right"> ##### Affordances - Increase browser-level users - Increase test throughput - Scale test environment up - Disrupt component - Terminate pods - etc... </grid> note: To get from where we are to a truly emergent load test, we need agency. Agency in this case would mean the test's ability to constantly send and receive feedback about the system, about other virtual users, and about itself, feedback in the form of the things on the left. These things could be stored in something like Redis, or maybe a message queue. Agency then involves acting on that feedback in some way, using the affordances on the right column. But the key here is not to write code that says, if you get a high response time, then decrease the number of browser-level users or anything like that. That would be imperative code, and that would be removing the agency of the test. Instead, what we can do is let it freely determine the relationship between the metrics it gets and the affordances it chooses to interact with as a result. But what if the tests don't all choose to react in the same way? Well, that's actually a good thing. In fact, I think we should steer into it and improve that diversity by encouraging them to react in different ways. ---  ## What emerges? note: (35 mins) What emerges? What is the point of testing? Why do we even test? We might be tempted to say that the reason we're testing is to establish whether or not an application fails, but that is futile. Everything fails. ---  **Thresholds of operation**  note: Instead, the point of an emergent load test should be to establish thresholds of operation: the range of tolerances that we can expect our application to exhibit in response to a variety of stimuli. Emergent load testing excels at pushing the boundaries of what an application can withstand by pushing the boundaries of our testing itself. ---  <grid drag="30 30" drop="top"> ### Disadvantages of emergence </grid> <grid drag="40 40" drop="left"> + It takes time + It's unpredictable + It isn't repeatable + It's unreliable </grid> <grid drag="40 40" drop="right"> ![[/assets/icon-unlike-red.png]] </grid> note: ---  <grid drag="80 30" drop="top"> ### When does emergence work best? </grid> <grid drag="50 50" drop="left"> ![[/assets/emergence-bricks.png]] </grid> <grid drag="50 50" drop="right"> + When a system is exceedingly complex + When a system displays internal logic + When there is sufficient time + When exploration is the goal </grid> note: + When a system is exceedingly complex + When a system displays internal logic + When there is sufficient time + When exploration is the goal ---  ## Summary + Emergence is the evolution of the whole beyond the capacities of its parts. + Imperative vs. declarative vs. generative + Tests haven't kept up with other forms of code. + To encourage emergence in load testing: + identify the affordances of the application + choose a tool that can play with all of them + explore ways to have it react and adapt to information it receives + Emergence is what you learn about the tolerances of your application. ---  ## References *These slides are available at nicolevanderhoeven.com.* Johnson, S. (2002). *Emergence: the connected lives of ants, brains, cities, and software*. Scribner. Waldrop, M. M. (2019). *Complexity: the emerging science at the edge of order and chaos.* Open Road Media. Wild, A. (2014). *The remarkable self-organization of ants* [Photograph]. https://www.quantamagazine.org/ants-build-complex-structures-with-a-few-simple-rules-20140409/ note: