AgileLoad Video Center AgileLoad Presentation Agileload Setup Monitoring Reporting Scripting Testing methodology Agileload Getting started - an application performance testing Overview A video guide to professional performance engineering / testing. Covers questions such as why do we need to do performance testing, what is performance testing and what type of performance tests there are, what's needed to run a performance test. Performance Testing Overview in less than 20 minutes! Why do performance testing? Maybe your boss or your customers are asking these questions. They’re going to be broken down into things like: How application is going to deal with unexpected loads which are traffic spikes What about our fall over or load balancing. Is that working correctly?Do we know this? How reliable is our application? What will happen if we leave it running over the weekend? Will someone need to come in and reboot it? It might be that you’ve already deployed an application and performance is poor. So you need to know why that is. Is it down to design or you might just actually want to find where the bottlenecks are in the application. Looking at what the business does, you need to know that your application can comply with the business performance requirement. For example, the business knows it needs to sale so many insurance polices or books or whatever it might be in one day; and most of that business will be done in key critical hours. Can the application deal the busiest hours? It might be that you already have an existing application and you need to understand how this new version of the application will compare with the old version. And finally, it might be just the case that you want to know what happens if your business grows. Say for example, next year we have a phenomenal year and we double our transactions. How is the application going to do with that? What is the maximum amount of users we can handle ? How many sales per second, maximum concurrent users that the application can deal with? Well, performance testing can answer these questions. If you don’t know the answers to these questions, then eventually your customers will find them out and probably the hard way. So we don’t really want our customers to end up looking like this guy. So let’s swiftly move on because performance testing really all about risks. And you got a choice, do it or don’t do it. So here we’ve got a grid that will illustrate what are the outcomes of those cases. So if we have a high performance application in this section here and a poor performance application here. But the thing is, we do not know how the application performs. So what if we take the action of doing nothing and the application is a high performance application? Well, hey we’re in the money. We’ve spent no money and we’ve got away Scott free. It was a bit of a gamble but hey it paid off. Well, the thing is, you still don’t know anything about the performance of your application. You know it works in that scenario but will happen – what are the upper limits of the application? What’s going to happen over the weekend? What’s going to happen over the month if you leave the application running? So we know nothing about the performance of the application and its upper limits and so on. What if we do evaluate the performance of the application and it is high performance? Well, we’ve burnt money right. I mean, we could have got away without that. True. But there is always a return on our action because from that cost of effort we now understand the limits of the application. We understand the performance of the application. So what if the application doesn’t perform very well? Well, can your business afford to take that risk? What would that mean to your business if the application does not perform? It’s something you need to evaluate. And if it doesn’t perform and we evaluate it, well, it’s happy day because we’ve avoided the catastrophe. We know early on that we’re not going to put this application live on the date that we’ve proposed; or if we are, then we know we need to step up efforts and take immediate action very quickly to make sure that we have a high performance application at the end of the day. As Conclusion, evaluation or performance is always going to reap benefits. By doing nothing you have risks. The benefits of doing nothing are very small with the amount of time and effort and costs that you save. On the downside, it could mean absolute catastrophe. So it really is a question of risks and how much do you want to take. What is performance testing? Well, it’s also known as load testing, stress testing, performance testing, capacity testing, volume and my least favorite of all, non-functional testing. Effectively, it is to appraise any user experience in realistic scenarios on your target application. It will allow you to assess machine and infrastructure capability. Performance testing, load, stress, capacity are all based on business requirements. It’s a professional approach to evaluating the performance of an application. It requires the simulation of real load scenarios running against your target applications or websites. So that means concurrent users. Now, concurrent users are not the be all and end all of a performance test. It’s only one part. It’s an important part but really an equally important part is having a mix of relative transactions. For example, if you just have 10,000 users hitting your homepage, you’re not testing the web application. All you’re testing is the web server. In the case of a content management server, then yeah, then it’s going to hit the backend database and that might be all you need to do. It might be a very simple sales website or something like these. But for testing sophisticated web applications, then you really do need a mix of relative transactions running at correct pacings. So that’s the speed, which each of the users is running at, and accurately and correctly generated load. The load must be generated conforming to a HTTP standards. If you’re following all of those for points, then you end up with realistic load, which will then give you a fingerprint of the response time that the application will deliver. In fact it will give you the response time that you can expect and also the transaction rate that we know the application can deal with at its peak. This maximum support; number of concurrent users if we’re doing a stress test. Now this does require the use of tools and there are a lot of tools out there. There’s a whole commercial and technical spectrum. There’s all the way through the most expensive commercial tools down to what people might think of as free software or open source there already. There is still a cost, which has to be evaluated to getting these tools setup, which leads us on to a technical spectrum. You go all the way down to purely scripted based approach, no GUI at all, to something that is really over simplified and gives you no control over visibility over the load that you’re generating. There’s also software as a service, which is certainly suitable for public web applications and really only simple transactions. So if you’ve got a complex web app, quite often these vendors are not able to work with you. The thing is, it has to be realistic. Really I would say, do it properly or just don’t do it at all because if you do it incorrectly and use the incorrect methods and tools, you can end up with a performance false negative or a performance false positive. A false negative for example would be if you just take any old tool and just record some scenarios without really understanding what you’re doing, fire it at maximum rate using all the CPU power that you have; and the tool says, hey looks I brought the web app. The performance is terrible. The response times are awful. The thing is that test probably wasn’t realistic and more than likely went way over and beyond the performance requirements from the business. So don’t waste time on trying to fix an application that’s actually not broken. On the other hand, a tool or service might come back and say, well, this application’s performing fine. In fact, we had a situation with the bank where they had a very well understood set of business performance requirements and it was actually a call center application. Where users would come in at 9:00 and work till 5:00 and work all day and there were 1,000 of these users. So they replicated those scenarios and used a tool to do that. The tool came back with very positive results and the application seemed to perform. And the application went live in fact only 250 of the users could actually get it to work. This was down to the fact that the application couldn’t deal with more than 250 users at rear load and the fact that the tool had not generated their load correctly and accurately. This is back in the day of HDV 1.0 and the number of connected threads per user was not accurate. In fact the load that was generated was well below the equivalent footprint that real users would have generated. What is the different between stress test, load test, soak test, spike test ? Well, if we take a look at normal expected utilization and maximum design capacity, then our tests are going to fall somewhere in between that. a stress test is where we escalate the amount of load over time until we find the limits of the system. a load test is where we understand the business requirements very well and can ramp up the load, leaving it running at a plateau until we’re happy that the performance is not continuing to degrade and then ramp down on the load. And what we’re looking for in that plateau is an equilibrium. We’re looking for CPU levels to stay the same. Response times to stay the same. Memory levels to stay the same. a soak test would be something that we’d run over a weekend or over 24-hour period and we would be looking generally for things like memory leaks a spike test where we go over and above the maximum design capacity, just to see how the application can deal with a big spiking load. Generally, what you’ll find will be a ripple affect but what we’re looking for is for the application not to fall over. These days with fast broadband networks people don’t wait much more than around five seconds to get a response for their page. In fact, this is what we would call the knee in the curve. That when you get a linear degradation, until you get a to point where response time is really not good enough. In fact, this quite a steep knee. Basis of performance engineering. Performance engineering is the application of performance testing into an application life cycle. We start with the analysis phase, where we understand the business requirements and start to form the footprint of the load that we need to generate. At this point you might want start running proof of concepts on the tools that you think might be suitable for the job. We then move on to the planning phase, where we detail our test schedule. So what types of users do we want to simulate? Some users making sales, some users doing searches, some users doing updates. Then we define how many of those types of users will be running in the test. For example, 90 percent might be browsing a site and maybe 5 percent will be buying it. Then another five percent might be doing some other set of obituary transactions on the site. Also, part of the planning phase is where we put together our critical paths through the application so the user journeys. And we’re going to use that to capture our scenarios in the tools. So what ultimately the tools is generating a script right. So we do that with a record and playback. We use those script to put together our scenarios and really importantly, we take the date requirements. If you’re going to do a load test with 100 users, in many cases it is very important that they are unique users. So you need to have some user accounts on the system. Incidentally, you can also use a tool to generate this test data using the application to face itself. Also, consider with data, if you’re going to update 10,000 policies you’re going to need quite a few thousand insurance policy IDs in order to make sure that you’re not getting some kind of deadlock with two users trying to update one insurance. I’m taking the scenario of an insurance website here that two users not trying to update the same policy at one time. Then we move on to the execution phase. This can be anything from two hours to 24 hours, depending on the type of test that we’re running. Depending on the two, you might be able to start to get some results as the test is running, while it’s in runtime. Based on that, you’ll be able to know whether to call the test off if it’s running particularly badly, the application or continue to the end. How AgileLoad can be use in the application testing cycle ? Using something a performance tool like AgileLoad can gives you recommendations on what to do, which are very useful for system tuning. This becomes a cyclical process. Test. Tune. Test. Tune. Test. Until we’re at a point where the performance engineering cycle is over and the results can be documented properly in a report and the recommendations, if there are any further recommendations can be made. A number of users are going to be going in through the network infrastructure into the application, so a number of web servers, application servers and database. We use AgileLoad Center to control AgileLoad injectors. Now, these injectors can run on the same machine as AgileLoad Center or if you’ve got particularly large performance test requirement, you might need to distribute that one to several machines or you might want to distribute the load around the world. For example or around your network in order to get an insight on response times from different network locations. We generate load on the application. We hit the web server, which in turn makes service request to the application server, which in turn makes database request to the database and then it sends messages back up stack. We end it with an end-to-end performance test while this is running AgileLoad can also monitor the performance of each of our infrastructure notes. Things like CPU or database calls or memory utilization or network utilization and so on. It pulls this altogether and puts it into AgileLoad Center. What do we get as an output? We spend a lot of time and effort creating our input, which is designing the tests, capturing the user journeys and scripts. Plugging the data in, setting up their platforms. So that’s talent and effort; really the output is the reports. Once you walk away from a project that is really how you would be judge because that is sum total of the effort with the performance testing cycles. So the report has got to be very good. In the case of AgileLoad reporting is automatically generated but what we do suggest is that whoever’s doing the test also adds to that point an executive summary for key findings from the test. Now, before the reports is generated really the results can give you a very early insight into the performance of the application. In fact, even when the test is running and that might be the number of virtual users attained, the response times, any errors that we’re getting back. Performance of different parts of the architectural, like the web server or application server and so on. It’s a great way to get a first view of performance and start to make a plan as to whether you’re going to continue the test or stop the test and then go to do some tuning. Once the test is stopped, you can start to make some recommendations based on the output of your monitors. AgileLoad is pretty good for this using anomaly detection. You may or may not have also some performance engineers available with expertise on hand. But anomaly detection itself can actually star to give you ideas about the first steps for making remedial action i.e., fixing the performance bugs. Why would you choose AgileLoad ? Well preparation time is much of a reduce. You’ll see in the videos that you can teach AgileLoad how to deal with complex transactions. So the more you use AgileLoad, the more efficient it becomes. This can save hours and hours, if not days when it comes to performance testing. It has a very broad monitoring capability. Anomaly detection uses expertise built-in to AgileLoad to analyze results and help make recommendations to you. Also, it has a very customizable reporting capability. So within minutes you can have reports for project manger level, for database, administrator level and for test manager level, using the automated report output. The fact that it’s customizable is you can control the level of detail and there’s specific items that you want in that report. The finally two points, well, it’s free for setup, preparations, and small test. Large tests with AgileLoad are the lowest costs on the market and we don’t charge any more for Cloud testing. For example, if you have Amazon 82 account, we could just plug into that and then use those Amazon data centers around the world, west coast, east coast, Asia, South America, Europe, and I believe South Africa now as well to generate load from around the world. How does AgileLoad compare to other tools? Well, we compared it to a very well established commercial toll and found that we are 30 to 45 percents savings, making a lot of those savings in the model of user transactions. a capture, a generation of scripts and simulation . AgileLoad has got this ability to compare capture traffic with generated traffic and we’ll highlight areas that might need to be addressed in the script that’s produced allowing you to very quickly produce correct and accurate script. Load simulation is also simple, our very lightweight injector can be deployed simply and easily using any Cloud infrastructure or machines that you have in your data center. Analysis and diagnostics using built-in expertise in AgileLoad will help you quickly find remedial action for your performance problems expatiating tuning modification correcting. As I mentioned in the previous slide, reporting is very fast. The quality reports is very high. So that’s it. Thanks for listening. Head over to agileload.com to see more videos and more technical videos or to download AgileLoad itself or take a look on the forums or at our blog. So I’ll hopefully see you there.