- Last Updated: [[2020-12-11]] - [[JMeter]] #[[Migrating from JMeter to k6]] - ## You need a plugin for everything - Everything is a plugin, and most JMeter users would argue that there is a standard set of plugins that you should download before you even begin to use it. A new JMeter user might not necessarily know where to find these and may have a less-than-satisfying experience out of the box. - To be able to use JMeter effectively for load testing, you need to use plugins. Out-of-the-box functionality is a little limiting. - ## It can be overly complex - JMeter's approach is to support everything, but the downside is that including "everything" in a load testing tool means it can become bloated. - ## JMeter doesn't natively support thresholds for the whole test - {{embed: ((Qgq7IPvhp))}} - ## Scripting is done within the UI - ### It takes longer to get started. - It takes time to learn any new UI, whereas javascript is near-ubiquitous and it's a good language to learn when you're load testing web sites anyway. - Compare JMeter script with k6 script. - ### Local environment - You need everything installed locally (Java, JMeter, plugins, script) to create a script. - Who hasn't struggled with environment variables when using JMeter, especially on Windows? - With [[k6 (tool)]], all you need is one installer (on [[Microsoft Windows]]) or a single command on your terminal, and scripts are just text files. - ## More difficult to collaborate - Everyone who wants to work on a script needs to have JMeter installed, but you might run into issues when you're working on the same file. - JMeter scripts are in XML format. It's very long-winded and not a great way to script - you pretty much have to use the UI. - Compare this to code-based load testing tools, where the script is a plain text file that can be versioned in [[Git]]. - "> Easy to use with Version Control Systems No Yes" - If you and your team don't have the same plugins installed, JMeter may not even let you open the script in the GUI. - ## JMeter is relatively resource-hungry - JMeter requires more memory to run, especially if you add listeners - "JMeter" - "" - "k6 took up 256 MB of memory" - "" - "Rafaela's findings are further affirmed by our own benchmarks." - JMeter is built on Java, which is a programming language that was first released in 1995. One of the basic premises of Java is that it's self-regulating, in that it manages resources (memory) automatically. It does this by placing Java apps, including JMeter, inside a JVM ([Java Virtual Machine](https://en.wikipedia.org/wiki/Java_virtual_machine)). The disadvantage to this approach is that it adds another layer of complexity between the load generator and JMeter. - JMeter is built on Java, so you have to pay attention to [[Heap memory]] allocated - it's common to run into [[OOM]] errors otherwise. - "> Number of Concurrent Users: Thousands, under restrictions -- [[Rafaela Azevedo]]" - Even JMeter official documentation suggests that you increase the Java Heap size beyond 1GB before you even run your first load test. - "> Increase the Java Heap size. By default JMeter runs with a heap of 1 GB, this might not be enough for your test and depends on your test plan and number of threads you want to run " - Source: https://jmeter.apache.org/usermanual/get-started.html - When you're running a load test, the best practice is to run it from the command line anyway-- because otherwise, it interferes with your results. Code-based load testing tools are performant from the beginning. - JMeter documentation: - "> Don't run load test using GUI mode ! " - Source: https://jmeter.apache.org/usermanual/get-started.html - On startup of JMeter -  - - ### Even among Java-based load testing tools, JMeter lags behind. - As a load tester, when given the choice between open source load testing tools, I'd choose JMeter - unless the project required simulating tens of thousands of users, in which case I would choose Gatling. [Gatling](https://gatling.io/) is another load testing tool that is built on Java, yet it still manages to outperform JMeter in memory usage - with [some benchmarks](https://abstracta.us/blog/performance-testing/gatling-vs-jmeter-findings/) pegging the difference at 30% less memory than JMeter-- a figure that corresponds with [our own internal testing benchmarks](https://k6.io/blog/comparing-best-open-source-load-testing-tools#memory-usage). -  - Source: [[Ragnar Lönn]], in our [Load Testing Tool Comparison](https://k6.io/blog/comparing-best-open-source-load-testing-tools) - This is due in part to the fact that Gatling uses a message-based architecture, whereas JMeter uses the 1 Thread: 1 Virtual User Paradigm. - Arguably one of JMeter's biggest strengths is its GUI mode, but the app itself will warn you that you shouldn't be running load tests using the GUI anyway: -  - "Even JMeter official documentation suggests that you increase the Java Heap size beyond 1GB before you even run your first load test." - ""> Increase the Java Heap size. By default JMeter runs with a heap of 1 GB, this might not be enough for your test and depends on your test plan and number of threads you want to run "" - "Source: https://jmeter.apache.org/usermanual/get-started.html" - "JMeter requires more memory to run, especially if you add listeners" - ## JMeter often lags behind Java releases - Java 9 released September 21, 2017. - Source: https://en.wikipedia.org/wiki/Java_version_history#:~:text=Java%20SE%209%5Bedit%5D&text=The%20first%20stable%20release%20of,was%20on%20September%2021%2C%202017. - JMeter 4 was the first version to support it. It was released on February 10th, 2018 - five months afterwards. - https://en.wikipedia.org/wiki/Apache_JMeter - > JMeter 5.0 officially requires Java 8 or 9, although later versions may be unofficially compatible. From our internal testing, JMeter will run on Java 11, albeit with some expected warnings such as: All illegal access operations will be denied in a future release Nashorn engine is planned to be removed from a future JDK release Our guess is that the next major release of JMeter (6.0) will fully support Java 11, which has been marked as the next LTS (Long-Term Support) version after the widely used version 8. For now, we will support Java 8 for consistency across the other tools we support. - [[Antonio Jimenez]] - > You may encounter the following error: `java.security.cert.CertificateException: Certificates does not conform to algorithm constraints ` if you run a HTTPS request on a web site with a SSL certificate (itself or one of SSL certificates in its chain of trust) with a signature algorithm using MD2 (like md2WithRSAEncryption) or with a SSL certificate with a size lower than 1024 bits. This error is related to increased security in Java 8+. To allow you to perform your HTTPS request, you can downgrade the security of your Java installation by editing the Java jdk.certpath.disabledAlgorithms property. Remove the MD2 value or the constraint on size, depending on your case. This property is in this file: `JAVA_HOME/jre/lib/security/java.security ` See [Bug 56357](https://bz.apache.org/bugzilla/show_bug.cgi?id=56357) for details. - https://jmeter.apache.org/changes_history.html - # Quotes - "> __Jmeter is a great and powerful tool, but depending on what you really need (something more lighter) then Jmeter might become an over complex, slow, hard to maintain tool.__ - [[Rafaela Azevedo]]"