1 How to Monitor Performance Contents 1.1. Introduction ..................................................................................................................... 1.1.1. Purpose of this How To ........................................................................................ 1.1.2. Target Audience ................................................................................................... 1.2. Performance - some theory ............................................................................................. 1.3. Performance - basic rules ................................................................................................ 1.4. Recognizing some common performance problems ........................................................... 1.5. Monitoring, and optimizing, the performance ..................................................................... 1.5.1. Tools and mechanisms ......................................................................................... 1.5.2. Interpreting the request.log .................................................................................... 1.5.3. Caching ................................................................................................................ 1.5.4. Analyzing Search .................................................................................................. 1 1 1 1 3 3 4 4 5 7 9 1.1 Introduction CQ5 encompasses several applications, and interacts with several more. Performance (or the lack of it) is one of the first things that your users notice, so as with any application with a user interface, performance is of key importance. To optimize the performance of your CQ5 WCM installation various attributes of the behavior should be monitored. 1.1.1 Purpose of this How To To give a short introduction on what to monitor, and how, when optimizing the performance of your CQ5 implementation. Note This document deals primarily with the mechanics of monitoring performance. Issues such as setting the target metrics of the performance you want to achieve are covered elsewhere. See the CQ5 WCM Project Manager Guide for more information. 1.1.2 Target Audience • Power Users • System Administrators • Project Managers 1.2 Performance - some theory The problems that cause performance issues are often difficult to track down, even when their effects are easy to see. A basic starting point is a good knowledge of your system when it is operating as normal. If you don't know how your environment "looks" and "behaves" when it is performing properly, it can be difficult to locate the problem when performance deteriorates. This means that you should spend some time investigating your system when it is running smoothly and ensure that collecting performance information is an ongoing task. This will provide you with a basis for comparison should the performance suffer. Page 1 of 9 CQ 5.1 WCM Copyright 1993-2008 Day Management AG How to Monitor Performance The following diagram illustrates the path that a request for CQ5 content can take - and therefore the number of different elements which can impact the performance. Figure 1.1. CQ5 request - the web-chain Performance is also a balance between Volume and Capacity: Volume the amount of output that is processed and delivered by the system. Capacity the system’s ability to deliver the volume. This can be illustrated in various locations throughout the web-chain. Page 2 of 9 CQ 5.1 WCM Copyright 1993-2008 Day Management AG How to Monitor Performance Figure 1.2. Capacity vs. Volume There are several functional areas which are often responsible for impacting the performance: • Caching • Application (your project) code • Search functionality 1.3 Performance - basic rules Certain rules should be kept in mind when optimizing performance: 1. Performance tuning must be part of every project. 2. Do not optimize early in the development cycle. 3. Performance is only as good as the weakest link. 4. Always think about capacity vs. volume. 5. Optimize important things first. 6. Never optimize without realistic goals. Note Bear in mind that the mechanism you use to measure performance will often affect exactly what you are trying to measure. You should always try to account for these discrepancies, and eliminate as much of their effect as possible; in particular browser plug-ins should be de-activated wherever possible. 1.4 Recognizing some common performance problems The following lists common performance issues which occur, together with proposals on how to spot and counteract them. Page 3 of 9 CQ 5.1 WCM Copyright 1993-2008 Day Management AG How to Monitor Performance Table 1.1. Recognizing common performance problems Area Symptom(s) To increase capacity... To reduce volume... Client High client CPU usage. Install a client CPU with higher performance. Simplify (HTML) layout. Low server CPU usage. Upgrade to a faster browser. Improve client-side cache. Some clients fast, some slow. Server Network CPU usage low on both servers and clients. Remove any network bottlenecks. Improve/optimize the configuration of the client cache. Browsing locally on the server is (comparatively) fast. Increase network bandwidth. Reduce the "weight" of your web pages (e.g. less images, optimized HTML). Cluster your web-servers. Reduce the hits per page (visit). Web-server CPU usage on the webserver is high. Use a hardware loadbalancer. Application Server CPU usage is high. Cluster your CQ5 instances. Search for, and eliminate, CPU and memory hogs (use code review, timing output, etc). High memory consumption. Improve caching on all levels. Low response times. Optimize templates and components (e.g. structure, logic). Repository Cache 1.5 Monitoring, and optimizing, the performance Performance issues may stem from a number of causes that have nothing to do with your website, including temporary slowdowns in connection speed, CPU load, and many more. It may also impact either all your visitors, or only a subset of them. All this information needs to be obtained, sorted and analyzed before you can optimize the performance. If you experience a performance issue: • try to replicate it: with one (or preferably more) standard web-browsers, on a different client that you know has good general performance and/or on the server itself (if possible) • check whether anything (related to the system) has changed within an appropriate time-space, and if any of these changes could have impacted the performance • collect as much information as possible to compare with your knowledge of the system under normal circumstances 1.5.1 Tools and mechanisms The following gives a short overview of some of the tools available for monitoring performance. Page 4 of 9 CQ 5.1 WCM Copyright 1993-2008 Day Management AG How to Monitor Performance Note Some of these will be dependent on your operating system. Table 1.2. Tools and mechanisms for monitoring performance Tool Used to analyze... Usage / More information... request.log Response times and concurrency. Interpreting the request.log. truss/strace Page Loads Unix command. Include the misc.truss log level to INFO. CQ5 status monitor Monitor CQ5 requests; including request-type, long-running requests, pending requests. Thread dumps Observe JVM threads. Dependent on the operating system, e.g. kill -QUIT Identify contentions, <pid> on Unix/Linux whereas Ctrl-Break on Windows. locks and long-runners. System calls Identify timing issues. Calls to System.currentTimeMillis() or com.day.util.Timing are used to generate timestamps from your code, or via HTML-comments. Note: These should be implemented so that they can be activated / deactivated as required; when a system is running smoothly the overhead of collecting statistics will not be needed. Apache Bench Identify memory leaks, selectively analyze response time. For full details: http://httpd.apache.org/docs/2.0/programs/ ab.html; basic usage is: ab -k -n <requests> -c <concurrency> <url> Search Analysis Execute search queries Analyzing Search. offline, identify response time of query, test and confirm result set. JMeter Load and functional tests. http://jakarta.apache.org/jmeter/ JProfiler In-depth CPU and memory profiling. http://www.ej-technologies.com/ truss/strace, lsof In depth kernel call and Unix/Linux commands. process analysis (Unix). 1.5.2 Interpreting the request.log This file registers basic information about every request made to CQ5. From this valuable conclusions can be extracted. 1.5.2.1 Monitoring traffic on your website The request log registers each request made, together with the response made: 09:43:41 [66] -> GET /author/y.html HTTP/1.1 09:43:41 [66] <- 200 text/html 797ms By totaling all the GET entries within a specific periods (e.g. over various 24 hour periods) you can make statements about the average traffic on your website. 1.5.2.2 Monitoring response times with the CQ5 request.log Page 5 of 9 CQ 5.1 WCM Copyright 1993-2008 Day Management AG How to Monitor Performance A good starting point for performance analysis is the request log. You can find the request log at data\author\logs and data\publish\logs for the author and publish instance respectively. The log looks as follows (the lines are shortened for simplicity): 09:43:41 09:43:41 09:43:47 09:43:48 09:43:49 09:43:49 [66] [66] [67] [67] [68] [68] -> <-> <-> <- GET 200 GET 200 GET 404 /author/y.html HTTP/1.1 text/html 797ms /author/z.gif HTTP/1.1 image/gif 141ms /author/x.html HTTP/1.1 text/html 0ms This log has one line per request or response: • The date at which each request or response was made. • The number of the request, in square brackets. This number matches for the request and the response. • An arrow indicating whether this is a request (arrow pointing to the right) or a response (arrow to the left). • For requests, the line contains: • the method (typically, GET, HEAD or POST) • the requested page • the protocol • For responses, the line contains: • the status code (200 means “success”, 404 means “page not found”) • the MIME type • the response time Using small scripts, you can extract the required information from the log file and assemble the statistics you want. From these, you can see which pages or types of pages are slow, and if the overall performance is satisfactory. 1.5.2.3 Monitoring search response times with the CQ5 request.log Search requests are also registered in the log file: 22.05.2008 12:27:40 [338] -> GET /author/playground/en/tools/search.html? query=dilbert&size=5&dispenc=utf-8 HTTP/1.1 22.05.2008 12:27:42 [338] <- 200 text/html 1562ms So, as above, you can use scripts to extract the relevant information and build up statistics. 1.5.2.4 Monitoring the numbers and impacts of concurrent users Again the request.log can be used to monitor concurrency and the system's reaction to it. Page 6 of 9 CQ 5.1 WCM Copyright 1993-2008 Day Management AG How to Monitor Performance Tests must be made to determine how many concurrent users the system can handle before a negative impact is seen. Again scripts can be used to extract results from the log file: • monitor how many requests are made within a specific time span e.g. one minute • test the effects of a specific number of users all making the same requests at (as close as possible) the same time; e.g. 30 users clicking Save at the same time 22.05.2008 12:27:22 [333] -> GET /author/libs/Personalize/content/statics.close.gif HTTP/1.1 22.05.2008 12:27:22 [334] -> GET /author/libs/Personalize/content/statics.detach.gif HTTP/1.1 22.05.2008 12:27:22 [335] -> GET /author/libs/CFC/content/imgs/ logo.rZMNURccynWcTpCxyuBNiTCoiBMmw000.default.gif HTTP/1.1 22.05.2008 12:27:22 [335] <- 304 text/html 0ms 22.05.2008 12:27:22 [334] <- 200 image/gif 31ms 22.05.2008 12:27:22 [333] <- 200 image/gif 31ms 22.05.2008 12:27:22 [336] -> GET /author/libs/CFC/content/imgs/ logo.rZMNURccynWcTZRXunQbbQtvuuCMbRRBuWXz0000.default.gif HTTP/1.1 22.05.2008 12:27:22 [337] -> GET /author/titlebar_bg.gif HTTP/1.1 22.05.2008 12:27:22 [336] <- 304 text/html 0ms 22.05.2008 12:27:22 [337] <- 304 text/html 0ms 1.5.2.5 Timing Statistics for Page Loading To see timing statistics for page loading you can use Ctrl-Shift-U - with ? debugClientLibs=true set in the URL. 1.5.3 Caching The following diagram shows the different cache locations that can be used for the various content types. Figure 1.3. What can be cached where? The following can act as a rough guide for target values: Page 7 of 9 CQ 5.1 WCM Copyright 1993-2008 Day Management AG How to Monitor Performance Figure 1.4. Cache vs. Uncached - maximum hits / second Although there are many algorithms to ensure that data is retrieved from the source system when appropriate, circumstances can arise where the data residing in a cache is out of date. Retrieving every page individually is the only guaranteed method of ensuring your content is up-to-date, but it is very costly in terms of response, and can indeed cause knock-on effects. This is particularly relevant when using personalized pages, where at least some content of a page is dependent on the user, and the account they used to login. Figure 1.5. Cache speed vs. Data Integrity Page 8 of 9 CQ 5.1 WCM Copyright 1993-2008 Day Management AG How to Monitor Performance 1.5.3.1 Optimizing your content for cache performance Make sure you use realistic cache settings for the browser cache. If you have disabled the browser cache for development, this may increase traffic and decrease responsiveness. 1.5.4 Analyzing Search First steps to analyzing the search function can be made with Monitoring search response times with the CQ5 request.log. However, once you have determined the response time, you may need to analyze why the request is taking the time it does, and what can be done to improve the response. Page 9 of 9 CQ 5.1 WCM Copyright 1993-2008 Day Management AG