Igor Baloš

Running JMeter tests with Jenkins

Performance tests are a very important part of Quality Assurrance process. As Postmark and Beanstalk are growing, the needs for more frequent and more complex performance tests are growing too.

Performance tests help us to identify the maximum operating capacity of our web applications, as well as determining any bottlenecks and elements which could cause performance degradation. By running performance tests we are making sure that you don’t experience any slowdown while using our web applications.

We are using JMeter for performance testing, mostly for testing Postmark. JMeter is working great for us, it’s highly configurable and allows us to stress and load test easily.

The only problem with JMeter (as with most of other tools for performance tests) is that its not a tool that everyone can use. The scenario I use for performance testing is the following:

• login to our test machine
• fire up JMeter
• load our performance tests
• setup the parameters
• run the tests
• watch the results

Performance testing is a process which requires a lot of time, frequent repetition, in order to get all the important data. What I wanted to do is to simplify the process as much as possible and make performance tests available to everyone on the team.

My main goal was to:

• make the most used tests easily accesible by everyone on the team
• allow simple configuring of the tests
• remove the need to login to testing machine for running the tests - allow remote execution

Running JMeter headless

First step in achieving my goals was to run JMeter tests headless - to run the tests without the need for UI. JMeter allows you to run your tests simply from the command line.

In our case what I wanted to do is to have a test which will be sending email. When running the test, the following options should be configurable:

• set environment we will use (staging/production)
• set credentials
• set email content
• set the load (number of users sending the emails, number of runs, duration)

In order to do this, you can use properties to fetch data from user input to JMeter. You can pass arguments from command line when running JMeter and they will be picked up in your tests.

For example, in my tests I can have a property called ENV, which can have the value: “staging” ro “production”. In JMeter, I can read the property with following call: ${__P(ENV)} and check its value. Depending on the value JMeter will load the settings for appropriately environment and run the test.

My JMeter call would look something like this:

sh jmeter.sh -n -p user.properties -t /smtp_test_param.jmx -l /reports/smtp_test_result.jtl -JUSERNAME=username -JPASSWORD=password -JUSERS=5 -JHOWMUCHTIMESTORUN=4 -JDURATION=20 -JENV=staging -JFILE=content.txt

This way I can run the JMeter tests right from console with no need to open JMeter UI. Now all I had to do is to allow me and my team to run these commands remotely.

Running JMeter Tests with Jenkins

I was already using Jenkins for running our automated tests, so Jenkins seemed to be a natural choice to run the JMeter performance tests. For every test build, you can simply setup parameters, even default values, so all I had to do is to create a Jenkins Test Project in which I will:

• set the parameters and default values
• open up shell to run JMeter
• run JMeter with properties which would be the parameters which I have setup in Jenkins

By creating parametrized test project in Jenkins now everyone can run the performance tests. Not only that, it reduced the time I need to spend on performance testing, plus it allowed me to do even manual testing faster, in cases when I need to send larger number of test emails.

Viewing JMeter Test reports with Jenkins

After setting up parametrized test projects in Jenkins, there was one piece missing - viewing performance tests reports. In order to do this we need to install Performance Plugin for Jenkins.

This plugin includes the feature of setting the final build status as good, unstable or failed, based on the reported error percentage. This would be very important, since we would not be able to determine easily whether test passed or failed, and why it failed.

After installing the plugin all we need to do is update test project by adding post build action “Publish Performance test results report” and set JMeter report files to “/reports/smtp_test_result.jtl” which I have setup before in JMeter call.

Now we can run the JMeter tests easily and review the reports straight from Jenkins. This allows us also to create a history of all performance tests.

Let me know what do you think about running performance tests with Jenkins and JMeter.

How to build high quality features

Building a high quality product takes time and effort. But as hard as is to build it, it’s even harder to maintain that quality throughout the years, by continually improving product with high quality features, features that people can use with ease.

A good example of high quality product is Beanstalk with beautiful web interface, powerful and ease to use set of features.

Powerful and easy is easier said then done. It takes a huge effort to maintain a perfect balance between great design, number of features and simplicity at the same time.

But.. that is what you, the customers, should see when you sign in to your Beanstalk account: simple and easy to use application, which has powerful feature set, but yet, its not complicated to use. You rely on Beanstalk everyday, and as long as you are happy we are happy too.

Behind Beanstalk, there is a huge amount of work, and I would like to give you a quick tour on how we work on features to make sure you love the result. I would like to quide you through the process of how we built new Beanstalk deployments.

Quality features take Time

In order to build, actually rebuild and improve something like Beanstalk deployments Wildbit team needs time. It’s a process which involves many steps. Some of them are:

• discussing design improvement
• designing the feature
• handling the design files to developers
• developing the new feature
• handling the feature to tester on first review
• tester, designer and developer working closely together on polishing the feature until it’s ready
• approving the feature by tester for release
• developer and tester testing migration from old feature to new feature
• performance testing the feature on staging
• releasing the feature, and make sure to document and announce at correctly
• watching performance of the feature on production
• support the new feature by helping customers and fixing new bugs found on the spot

That sounds like a LOT, right? In order to have a high quality feature and product people can rely on, you need to go through all the steps mentioned above, and maybe even more.

Figuring out UI workflow and designing it, is an Art

It all starts with discussing how to improve the design of the feature. Discussion is lead by Eugene , Chris and Natalie , and all team members can participate. This discussion is one of the key elements for new feature to succeed. If wrong path is taken for design improvement, new feature will be a FAIL.

Thanks to Eugene, discussion gets transformed to new design files, and after approval, development of the new feature begins.

Development is more than making features designed beforehand work

As a software tester, I always admired exceptional developers like the ones working in Wildbit. Nothing is more satisfying for software tester, than too work with a developer who is great at his work, willing to collaborate with you on pursuing the goal: releasing a feature with smallest number of bugs.

Development begins, soon after, we can see in our staging environment the new feature starting to work. But, that is just a beginning of the long road to get the feature, deployments in this case, working smooth and correctly.

New features are implemented one by one, and soon as they are tested and approved by developers, they are assigned to software tester for first round of checkup. First round of test begins on developers side.

Testing software is an Art too

When big features like deployments are assigned on review to a software tester, software tester is prepared for testing section by section and back and forth communication with developers, due variety of sections needing to be checked.

Some might think that testing features which are improvements of existing features is easier to test and release. It’s actually oposite, testing feature which is improved is harder, you need to:

• check whether new feature works
• check whether old features work correctly
• check whether all old features are preserved
• check whether features are as easy to understand for users as before improvements
• do a migration checkup
• check whether performance is the same as before feature was released

For new features, you would only need to do step 1.

When reviewing features, I would first check whether features work at all, and then I would go into further details per feature, always focusing on one section, but balancing testing in order to test sections equally.

As testing goes into more details, we would all suggest adjustments in discussion, to improve the look and feel of the application. As we go more and more into the details, the number of small tickets in ticketing system will increase, and number of features finished and approved will grow.

After approving the whole feature set, deployments are almost ready for release.

Migration is important part of upgrading a feature

After finishing work on feature improvement, key part of releasing the feature is to do a migration test. You must be sure that the feature you have improved will work correctly for old and new users, that everyone can use it continuously without any problems. When deployments were ready for release, Dima did a migration checkup and I helped him a bit on finalizing the release.

Documentation plays important role for every feature

When feature like new deployments is released, you have to make sure customers know about it, and not take it by surprise. Alex and Natalie make sure every feature at Wildbit is properly documented and announced to customers.

Don’t just release it, support it

When all the work is done, and you are ready for glass of champaign after hard work, you have to make sure to support the feature that you have released. After the release, there will be bugs, or suggestions for changes customers will make.

We always make sure to take suggestions into account and fix bugs on the spot. That is the only way to make features like deployment long lasting.

New deployments feature is one of the biggest improvements up to date in Beanstalk in backend and front end too. It opens a window to variety of new features in future.

For adding this feature, more than 200 tickets were opened in issue tracking system, and 15% of Beanstalk interface was redesigned to support it. From day 1 after release, deployments worked for Beanstalk users.

Jenkins, Nodes and Windows

One of the cool features of Jenkins or Hudson, depending on distribution you choose, are nodes. Nodes allow you to prioritize jobs, separate their execution and more.

For example, for Postmark, we have a separate node, which runs highest priority tests (like email sending tests), and we have another node which runs the general tests.

Running multiple Nodes as Windows Services

When you are adding more than one node as windows service, keep in mind that you need to use different names for each service, otherwise you will get an exception, which will not give descriptive message at the moment. Exception will be thrown when you try to install the service.

That happened to me, while creating the second node, which had a same service display name as node created before. Even though service name was different, display name needs to be different too.

You can change display name easily by running the following command, in command line:

sc config service_name DisplayName=

for me service name was:

jenkinsslave-c__Igor_jenkinds_node-postmark-priority