DRTnOOber: An Introduction

On August 17 2012 08:50 DRTnOOber wrote:

Ok here's a bit more detail about my job...

So the kind of applications I test are; public websites (for example government websites, big corporate websites, ones where many thousands of people use them), and internal web applications (for example I worked on a project which upgraded the system which manages all benefits and social welfare in New Zealand). Basically, whenever an application is going to be used by a lot of people or have a lot of activity there is a big risk that when you go live, it's going to crash and burn. This might be because the hardware of the servers isn't sufficient, or the software contains bugs (maybe a memory leak, or a really slow SQL query), and it can also be because of settings on the servers such as Java garbage collection, or connection pools, there are a LOT of variables.

So the basic process is; attach monitoring to the system, apply load against the system and record how it performers, analyse the results and sometimes do some troubleshooting where we look into the system logs and really try and pinpoint the cause of a performance problem.

For monitoring we mostly monitor the hardware usage on all the web, application, database, and any other servers involved. We have our own tool called System PI which is written in Perl and basically connects via ssh to all the servers (we also have a Windows monitor but most servers use Unix/Linux of some flavour) and runs various commands such as vmstat and top to assess the system. The second part of monitoring is getting access to the system logs to look for errors during the test.

In terms of applying load to the system, that's usually done using a tool such as LoadRunner, SilkPerformer, NeoLoad, JMeter (open source), IBM Rational Performance Tester etc. What these tools do at a very basic level is capture the HTTP traffic between a client (usually a web browser) and server. Using the tool you are then able to modify these HTTP requests and send them back to the server to replicate the behaviour of a real user. Because you're only sending HTTP requests rather than actually rendering the page in a browser it uses a lot less resources on your computer so one normal desktop PC can be used to create maybe 500 or more "virtual users".

Now, a bit part of the job is accurately producing load which reflects how real users will interact with the system. There's no "one" answer... but you often want to find the main business transactions that will occur. For a basic public website that had products I had just two transactions "Browser" and "Purchaser" (for example) but I've worked on complex systems where I've needed many more. You don't need to hit every transaction or part of the system, but you need to hit the parts which will be used the most or are most performance critical. For system upgrades you can also get this information from production web server logs. Ultimately you build a "workload model" which shows which transactions you will do and how many transactions to apply over an hour - and that is essentially what you build in your performance tool by recording those transactions, and modifying the records to be dynamic enough to run multiple times.

This phase, of building up a set of tests is called "scripting", and when that's done you actually get to run the tests. There's lots of tests for different reasons; a "peak load" test puts predicted peak load on a system so you can measure the end user response time. It's useful for know what the users can expect when the system goes live. Then there's headroom tests which incrementally increase the load until the breaking point is found. That's good for understanding the capacity of the system. Another type of test is a "soak test" which runs for a long period of time to check the stability of the system (often a bug here is a memory leak).

As for the analysis, we gather all the user response times from the test tool, the hardware usage from the servers, and look at the server logs. We analyse using a graphical visualisation tool called Tableau but you can use Excel to do most of the same stuff (although it's more time consuming). And finally we report our findings to the client.

All of the load testing tools require you to know programming or scripting languages. You can do the basic stuff with the GUI interface, but if you want to do anything fancy at all you need to code. That's a part I love about the job. We also get to write utilities to do data translation or other tasks to help us in our job; and Perl is the language our company uses because it's ideal for doing powerful filters and translations with a very small amount of code.

I kind of was lucky finding my way into this career. If you wanted to do it, which I actually really recommend (it pays better than a programmer) then you could always take the ISTQB Foundation course and get your certificate. The course is a bit arse (I actually teach it...) and it focuses entirely on functional testing, but with it + a computer science degree you have all the qualifications you could possibly get to get into the performance testing field. It's usually a 3-5 day course and a 90 minute exam (from memory) but it does cost a few thousand dollars to do. Realistically I'd just keep your ear our for performance testing/engineer jobs because not many people know about the field. Even knowing those terms will help you find the jobs on a job finding website.

Hope this has been somewhat educational... it's not very well explained but I have my daughter demanding attention and it's been very distracting