Please post your Web Driver questions in official Web Driver forum

Tuesday, November 24, 2015

JMeter Load Testing Presentaton

Tuesday, November 3, 2015

JMeter and Load Testing Best Practices

As the saying goes, there are no best practices or probably in context. Following are some guidelines which I followed when working on load testing with JMeter. Some of these guidelines/practices are not limited to only JMeter and can be applied to load testing in general, So let’s begin -

  1. Number of Test Runs - The worse thing you can do with load test is to conduct test only once. Given that your test environment would depends on many factors, it is wise to conduct load test more than once to verify consistency of results. If test results have more than 5~10% discrepancy then it is sign of inconsistent system. Figure out the cause of problem first and fix it.

  1. State of system - If you are conducting load test consecutively then beware that each consecutive run would leave system in a state which would have memory, db etc resources utilized. Probably your goal is to run each test on a clean state of system.

  1. Identify client limitations - Many a times you would encounter that client becomes a bottleneck when there are high number of threads on one client, moreover quicker the application response the more work JMeter has to do to process results. From various sources it is usually recommended that you should not use more than 300 threads on one machine.
While load testing one API I encountered that more than 70 threads on one jmeter client resulted in very high 95 and 99 percentiles response times but when I distributed load from multiple agents each having 70 threads then response times were within acceptable limits.
Hence once you start seeing higher response times, unexpected throughput etc then you should also consider if you are hitting the limit on client side. And one way to figure it out is to divide big large load on one test agent to many smaller loads on multiple test agents and gauge how results change. Once you identify the max threads you can have from one test machine then you know how many more machines you need to generate required amount of load

  1. Assertions - Like manual testing, it is important to find out if a web page or API response is right under load test. A plain 200 response code does not guarantee that page or API response is how it is supposed to be. Such check can be achieved by adding assertions to sample response. Example JMeter assertion - Response Assertion, Duration Assertion etc.
The result of assertion can be seen in Assertion Result Listener.
Even if you don’t add Assertion Result Listener, a failed assertion would always be reported in test results.
Response Assertion and the Duration Assertion are more or less safe to use, whereas the Compare Assertion and other XML-based ones like XPath Assertion take up the most CPU and memory.

  1. Wait Period (aka sleep time) - If you have worked UI test automation tools then you know how much static waits are hated. But when it comes to load test then wait period is recommended to be used between transactions. This is the time to give pause between subsequent sample requests. Wait period is required to emulate real user behavior since real user does not hit one request after another, after another etc but pauses for “some” duration before continuing with next request. You can use constant timer with JMeter to achieve this. Like other JMeter elements, timer can be added at the test plan level (which would then be applicable to all http requests) or specific samplers to add different constant timer for each sampler. There is another timer available in JMeter, known as Uniform Random Timer. This can be used to generate random time pause.

  1. Retrieving embedded resources - A web page is made of many components, there are css files, images, js files etc. You can instruct JMeter to download the resources during load test. HTTP Sampler > “Retrieve All Embedded Resources” - Check this checkbox to make JMeter download javascript, css and images just as real browser would do, also set
Use thread/connection pool to simulate the browser parallel fetching (use between 2-4 threads). In addition, for every one of these threads simulating a user, JMeter creates separate thread pools of given pool size with thread names like pool-n-thread-m. The main page is downloaded by the user’s thread "Thread Group 1-k" while the embedded resources are downloaded by its associated thread pool with thread names like pool-n-thread-m. when setting the concurrent pool size, keep in mind the number of users being simulated, because a separate thread pool is created for each of these simulated users. If there are many users, too many threads may get created and start affecting the response times adversely due to bandwidth contention at the JMeter side. If many users are to be simulated, it’s recommended to distribute JMeter testing to multiple machines.

Screenshot from 2015-10-26 16:31:09.png

When retrieving embedded resources then make sure you exclude external domain from download using “URLs must match” field. You don’t want to load test external URL you don’t have control on
ex - Add the following RegEx to the edit box named Embedded URLs must match to exclude external domains :
^((?!<domain #1>|<domain #2>|<domain #3><domain #4>|<domain #5>).)*$
Set  “Retrieve All Embedded Resources” in “HTTP Request Default” if resources are to be retrieved for all the http samplers
To download all resources from web site - use following URL pattern - .*test\.appdoamin\.net.*
  1. Listeners - Listeners are the way to monitor test results in JMeter either during test execution or once test execution is over. Listeners receive Sample Results and do some processing with it, this takes resources (memory, CPU). Hence during Load Testing do not use any listener.  Run your test in non GUI mode and save the test result for future analysis. GUI mode of JMeter with Listeners should only be used for debugging and making sure your test script works before going for full blown test.

  1. Out of memory Exception - If you encounter Out of memory Exception during test execution then you need to make more memory available to JMeter during test execution. There is a line in jmeter.bat (on windows) or (on linux) script which launches JMeter with following default value -
JVM_ARGS="-Xms512m -Xmx512m"

You can try increasing maximum heap until you'll stop receiving Out of memory errors. You could set it to somewhere like 70% of your hardware RAM.

The flag Xmx specifies the maximum memory allocation pool for a Java Virtual Machine (JVM), while Xms specifies the initial memory allocation pool.
This means that your JVM will be started with Xms amount of memory and will be able to use a maximum of Xmx amount of memory. For example, starting a JVM like below will start it with 256MB of memory, and will allow the process to use up to 2048MB of memory:
java -Xmx2048m -Xms256m
The memory flag can also be specified in multiple sizes, such as kilobytes, megabytes, and so on.

  1. Viewing JMeter results summary in non GUI mode - When running test in non GUI mode, you would want to see a snapshot of test run, like number of threads, response time etc information. To do this just uncomment the following within the JMeter properties file (remove the ‘#’):

    These parameters are already enabled on latest version of JMeter :-)
    The summary report looks as -

Screenshot from 2015-11-03 16:26:49.png

  1. Distributed (Remote) Testing -
Once you reach the limits of one machine, you can switch to distributed or remote
testing. But JMeter defaults are not fine for efficient remote testing, so in, add:
This will:
  • remove some data from the SampleResults as the response body, as you don’t need heavy response body when running heavy load test
  • will send Sample Results as Batches and not for every sample reducing CPU, IO and network roundtrips

  1. Use CSV as output for Save Service - XML is verbose, it takes cpu and memory resources for writing and analysis, CSV is great as utilizes least resources. Hence save your test output in csv file. When running test from command line, you can add following parameter to make jtl file to be in csv format -

    Furthermore, for massive load tests there are many result data you don't need.

So, in, add:;

You can also print variable, parameters used during test in csv file. For example if you use variable / parameters - counter, accessToken in your test plan then you can print them in csv file using following from command line -


  1. Using regx expression extractor - Use Regular Expression Extractor for extracting data BUT never ever check Body (unescaped), choose among:
Response Code
Response Message

Use efficient Regular expressions and extract as less data as possible

  1. Threadgroup name for distributed testing - For distributed testing use thread group name as -  
${__machineName()}_My Threadgroup name
This would identify thread group name exclusively for a machine

  1. Use cache manager to simulate browser cache
  2. Use cookie manager to simulate browser cookie
  3. By default, JMeter does not save threads count in JTL files. If you plan to work with JMeter JTL files, you should enable it by uncommenting in JMETER-INSTALL-DIR/bin/ the line and set it to true:
  1. To simulate browser, add user agent string in Header Manager. You can copy it from your browser -
It does not matter where you place header manager, headers in request will be same -

  1. Sharing variables between threads and thread groups -
Variables are local to a thread; a variable set in one thread cannot be read in
another. This is by design. For variables that can be determined before a test starts,
see Parameterising Tests (above). If the value is not known until the test starts, there
are various options:
  • Store the variable as a property - properties are global to the JMeter instance
  • Write variables to a file and re-read them.
  • Write your own Java classes

  • When testing web application login as one user manually and navigate to screen under test, you may be in for application bugs

  1. Application logs-
During manual testing, analysing application logs helps to uncover application defect which would otherwise be missed. This is equally true with load test. You may find errors on connection timeout, application operation failures ec which would add more value to load test report. So don’t forget to scan application logs when carrying out load test :-)

Monday, October 26, 2015

JMeter maven project

Maven JMeter project is a simple load test project which demonstrates jmeter and maven integration. Please don't execute it with more than 5 threads as it uses publicly available application
You can set up a CI build based on this project and parameterize build as -
Parameterized CI build
And now you can provide these values from building the job -
Execute CI Build
To execute test locally execute following command - -DUSERS=$USERS -DDURATION=$DURATION 

Replace $USERS with number of users you want to run test with. Specify required values for other parameters and execute the tests
Once test execution is over then you can browse through test results under -
/target/jmeter/results/index.html directory
Fork me on GitHub