Document 176380
How to setup Verbosegc (Grabage Collection) and run Oracle Thread dumps for WebLogic performance Monitor:
When an Oracle/BEA ticket is needed to troubleshoot WebLogic performance issues the Garbage Collection and Oracle
Thread dumps are required by Oracle/BEA. The following steps will outline how to setup the verbosegc collection and create Oracle thread dumps to send to Oracle/BEA for analysis.
The following topics will be covered in this document
1. Add the –verbosegc to the Java options in the Managed Server startup
2. Script to extract all the GC entries in the log and create Excel graph displaying the WebLogic memory
3. Run the Oracle Thread dumps if required
Telcordia Technologies, Inc. Confidential — Restricted Access This document and the confidential information it contains shall be distributed, routed or made available
solely to authorized persons having a need to know within Telcordia, except with written
permission of Telcordia.
Page 1
1. Add the –verbosegc parameter.
Locate the file setXngEnv.sh - typically found in the bin directory (i.e.—/app/bea921/user_projects/domains/granite/bin)
Copy the file before modification - cp setXngEnv.sh setXngEnv.sh .orig
If running in multiple environments add a section for the managed server you wish to add the verbose parameter.
File: setXngEnv.sh
#!/bin/sh
echo Setting XngParameters for ${SERVER_NAME}
case "${SERVER_NAME}" in "admin") echo NO CHANGE;;
"gateway") echo NO CHANGE;;
*) echo CHANGE
CLASSPATH=$LONG_DOMAIN_HOME/Xng_config_${SERVER_NAME}$CLASSPATH;;
esac
# If Specific Server environment changes are necessary, they can be made much like the following
#
if [ "$SERVER_NAME" = "admin" ] ; then
#MEM_ARGS=""
MEM_ARGS="-Xms256m -Xmx256m -XX:CompileThreshold=8000 -XX:PermSize=48m -XX:MaxPermSize=128m"
echo MEM_ARGS
fi
if [ "$SERVER_NAME" = "xng650_00" ] ; then
MEM_ARGS="-Xms512m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=256m"
JAVA_OPTIONS="-XX:CompileCommand=exclude,com/granite/asi/dto/AbstractDataObject,hashCode $JAVA_OPTIONS"
fi
Added lines
Verbose
Parameter
if [ "$SERVER_NAME" = "xng650_02" ] ; then
MEM_ARGS="-Xms512m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=256m"
JAVA_OPTIONS="-verbosegc -XX:CompileCommand=exclude,com/granite/asi/dto/AbstractDataObject,hashCode
$JAVA_OPTIONS"
fi
#MEM_ARGS="-Xms1024m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=256m"
#export MEM_ARGS
Telcordia Technologies, Inc. Confidential — Restricted Access This document and the confidential information it contains shall be distributed, routed or made available
solely to authorized persons having a need to know within Telcordia, except with written
permission of Telcordia.
Page 2
Stop and Start the Managed Server that has the verbose parameter.
View the WLSM.LOG file—usually found in the directory— /app/bea921/user_projects/domains/granite/xng650_02
<Feb 1, 2010 11:55:49 AM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 9.2 MP1 Sun Jan 7 00:56:31 EST 2007 883308 >
<Feb 1, 2010 11:55:53 AM EST> <Info> <WebLogicServer> <BEA-000215> <Loaded License : /app/bea921/license.bea>
<Feb 1, 2010 11:55:53 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Feb 1, 2010 11:55:53 AM EST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Feb 1, 2010 11:55:54 AM EST> <Notice> <Log Management> <BEA-170019> <The server log file /app/bea921/user_projects/domains/granite/servers/
xng650_02/logs/xng650_02.log is opened. All server s
ide log events will be written to this file.>
<Feb 1, 2010 11:55:54 AM EST> <Error> <Socket> <BEA-000438> <Unable to load performance pack. Using Java I/O instead. Please ensure that a native
performance library is in: '/opt/java1.5/jre/l
ib/IA64N:/opt/java1.5/jre/lib/IA64N/server:/opt/java1.5/jre/../lib/IA64N:/app/bea921/patch_weblogic921/profiles/default/native:/lib::/app/bea921/weblogic92/
server/native/hpux11/IPF32:/app/bea9
21/weblogic92/server/native/hpux11/IPF32/oci920_8:/usr/lib'
[GC 131072K->15821K(502464K), 0.0617879 secs]
<Feb 1, 2010 11:55:58 AM EST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Feb 1, 2010 11:56:02 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Feb 1, 2010 11:56:02 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
[GC 146893K->27467K(502464K), 0.1018594 secs]
<Feb 1, 2010 11:56:13 AM EST> <Warning> <EJB> <BEA-010100> <This server is not part of a cluster. Hence "InMemory replication feature" for StatefulSession EJB "WorkflowEJB" will have no effect
on this server.>
<Feb 1, 2010 11:56:15 AM EST> <Warning> <HTTP> <BEA-101369> <weblogic.servlet.internal.WebAppServletContext@f72499 - appName: 'template-tool6.5.0.02', name: 'template-tool-6.5.war', context-p
Telcordia Technologies, Inc. Confidential — Restricted Access This document and the confidential information it contains shall be distributed, routed or made available
solely to authorized persons having a need to know within Telcordia, except with written
permission of Telcordia.
Page 3
2. Script to extract all GC entries in file that can be opened in Excel
Check if file gcstats.csv already exist remove or rename to another file name if found
Create the column names as the 1st line of the file:
echo "Memory Before, Memory After, Total JVM" > gcstats.csv
(creates the gcstats.csv file)
Run the following GREP command to extract the GC entries into gcstats.csv file
grep "K)" wlsm.log | grep -v Full | cut -c 5-27 | sed 's/K->/,/g' | sed 's/K(/,/g' | sed 's/K//g' >> gcstats.csv
FTP the gcstats.csv file using ASCII mode back to your PC.
Open gcstats.csv file in Excel, should look like image below:
Telcordia Technologies, Inc. Confidential — Restricted Access This document and the confidential information it contains shall be distributed, routed or made available
solely to authorized persons having a need to know within Telcordia, except with written
permission of Telcordia.
Page 4
The memory items can be graphed to display the GC release of memory.
Highlight the A, B and C columns
Click the Insert tab, then choose Line chart, choose 2-D Line
Telcordia Technologies, Inc. Confidential — Restricted Access This document and the confidential information it contains shall be distributed, routed or made available
solely to authorized persons having a need to know within Telcordia, except with written
permission of Telcordia.
Page 5
Chart should display as follows
Telcordia Technologies, Inc. Confidential — Restricted Access This document and the confidential information it contains shall be distributed, routed or made available
solely to authorized persons having a need to know within Telcordia, except with written
permission of Telcordia.
Page 6
3.
Collection of Thread Dumps for Oracle/BEA support ticket.
To collect thread dumps the Unix Process ID must be determined— run the script below to determine
ps –ef | grep bea921 (bea921 is the WebLogic User ID)
bea921: ps -ef |grep bea921
bea921 18530 18529 0 Feb 1 pts/tc 0:00 -ksh
bea921 28035 28033 0 Jan 15 ?
0:00 /bin/sh /app/bea921/user_projects/domains/granite/bin/startWebLogic.sh nodebug nopointbase noiterativedev
notestconsole
bea921 26569 26552 0 Jan 15 ?
153:40 /opt/java1.5/bin/IA64N/java -server -Xms256m -Xmx256m -XX:CompileThreshold=8000 -XX:PermSize=48m XX:MaxPermSize=128m -da -Dpla
bea921 18980 18978 0 Feb 1 pts/tc 0:00 /bin/sh /app/bea921/user_projects/domains/granite/bin/startWebLogic.sh nodebug nopointbase noiterativedev
notestconsole
bea921 24704 18530 0 10:02:18 pts/tc 0:00 grep bea921
bea921 28051 28035 0 Jan 15 ?
121:04 /opt/java1.5/bin/IA64N/java -server -Xms512m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m
-XX:MaxPermSize= 256m -da -Dpl
bea921 18978 1 0 Feb 1 pts/tc 0:00 /bin/sh ./bin/startManagedWebLogic.sh xng650_02 qabhp1:7003
bea921 18996 18980 0 Feb 1 pts/tc 13:47 /opt/java1.5/bin/IA64N/java -server -Xms512m -Xmx512m -XX:CompileThreshold=8000 XX:PermSize=128m -XX:MaxPermSize=256m -verbose
bea921 24703 18530 1 10:02:18 pts/tc 0:00 ps -ef
Locate the process that shows the verbose option, in this example the process ID is 18996
Run the script
Kill –3 18996
( 3-4 times at 20 second intervals, thread dump info will go to the wlsm.log file.)
bea921: kill -3 18996
bea921: kill -3 18996
Telcordia Technologies, Inc. Confidential — Restricted Access This document and the confidential information it contains shall be distributed, routed or made available
solely to authorized persons having a need to know within Telcordia, except with written
permission of Telcordia.
Page 7
Entries in the wlsm.log file will look like these below:
[] [04 Feb 2010 10:14:25,105] [com.granite.util.license.oracle.OracleLicenseService] [License Listener] [DEBUG]
[License 6664 is valid.]
Full thread dump [Thu Feb 04 10:14:42 EST 2010] (Java HotSpot(TM) Server VM 1.5.0.04 jinteg:07.27.06-16:18 IA64 mixed mode):
"Thread-9" daemon prio=8 tid=00d43ed0 nid=43 lwp_id=909600 waiting on condition [40300000..40300c50]
at java.lang.Thread.sleep(Native Method)
at com.granite.middle.util.MTaccess.bgCheck(MTaccess.java:1167)
at com.granite.middle.util.MTaccess.run(MTaccess.java:1013)
"[STANDBY] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=00d431a0 nid=42 lwp_id=903905 in Object.wait()
[403c0000..403c0cd0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:474)
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:165)
- locked <579057c0> (a weblogic.work.ExecuteThread)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:186)
Telcordia Technologies, Inc. Confidential — Restricted Access This document and the confidential information it contains shall be distributed, routed or made available
solely to authorized persons having a need to know within Telcordia, except with written
permission of Telcordia.
Page 8
© Copyright 2024