How to Manage Your Network with SPECTRUM Document 1909 Notice Copyright Notice Copyright © 2002-present by Aprisma Management Technologies, Inc. All rights reserved worldwide. Use, duplication, or disclosure by the United States government is subject to the restrictions set forth in DFARS 252.227-7013(c)(1)(ii) and FAR 52.227-19. Liability Disclaimer Aprisma Management Technologies, Inc. (“Aprisma”) reserves the right to make changes in specifications and other information contained in this document without prior notice. In all cases, the reader should contact Aprisma to inquire if any changes have been made. The hardware, firmware, or software described in this manual is subject to change without notice. IN NO EVENT SHALL APRISMA, ITS EMPLOYEES, OFFICERS, DIRECTORS, AGENTS, OR AFFILIATES BE LIABLE FOR ANY INCIDENTAL, INDIRECT, SPECIAL, OR CONSEQUENTIAL DAMAGES WHATSOEVER (INCLUDING BUT NOT LIMITED TO LOST PROFITS) ARISING OUT OF OR RELATED TO THIS MANUAL OR THE INFORMATION CONTAINED IN IT, EVEN IF APRISMA HAS BEEN ADVISED OF, HAS KNOWN, OR SHOULD HAVE KNOWN, THE POSSIBILITY OF SUCH DAMAGES. Trademark, Service Mark, and Logo Information SPECTRUM, IMT, and the SPECTRUM IMT/VNM logo are registered trademarks of Aprisma Management Technologies, Inc., or its affiliates. APRISMA, APRISMA MANAGEMENT TECHNOLOGIES, the APRISMA MANAGEMENT TECHNOLOGIES logo, MANAGE WHAT MATTERS, DCM, VNM, SpectroGRAPH, SpectroSERVER, Inductive Modeling Technology, Device Communications Manager, SPECTRUM Security Manager, and Virtual Network Machine are unregistered trademarks of Aprisma Management Technologies, Inc., or its affiliates. For a complete list of Aprisma trademarks, service marks, and trade names, go to: http://www.aprisma.com/support/secure/manuals/trademark-list.htm All referenced trademarks, service marks, and trade names identified in this document, whether registered or unregistered, are the intellectual property of their respective owners. No rights are granted by Aprisma Management Technologies, Inc., to use such marks, whether by implication, estoppel, or otherwise. If you have comments or concerns about trademark or copyright references, please send an e-mail to [email protected]; we will do our best to help. Restricted Rights Notice (Applicable to licenses to the United States government only.) This software and/or user documentation is/are provided with RESTRICTED AND LIMITED RIGHTS. Use, duplication, or disclosure by the government is subject to restrictions as set forth in FAR 52.227-14 (June 1987) Alternate III(g)(3) (June 1987), FAR 52.227-19 (June 1987), or DFARS 52.227-7013(c)(1)(ii) (June 1988), and/or in similar or successor clauses in the FAR or DFARS, or in the DOD or NASA FAR Supplement, as applicable. Contractor/manufacturer is Aprisma Management Technologies, Inc. In the event the government seeks to obtain the software pursuant to standard commercial practice, this software agreement, instead of the noted regulatory clauses, shall control the terms of the government's license. Virus Disclaimer Aprisma makes no representations or warranties to the effect that the licensed software is virus-free. Aprisma has tested its software with current virus-checking technologies. However, because no antivirus system is 100-percent effective, we strongly recommend that you write protect the licensed software and verify (with an antivirus system with which you have confidence) that the licensed software, prior to installation, is virus-free. Contact Information Aprisma Management Technologies, Inc., 273 Corporate Drive, Portsmouth, NH 03801 USA Phone: 603.334.2100 U.S. toll-free: 877.468.1448 Web site: http://www.aprisma.com How to Manage Your Network with SPECTRUM Page 2 Document 1909 Contents Notice ........................................................................................... 2 Preface ......................................................................................... 9 Intended Audience ..................................................................... 9 Text Conventions ....................................................................... 9 Document Feedback ..................................................................10 Online Documents .....................................................................10 Introduction ............................................................................... 11 SPECTRUM Core and Network Management ..................................12 Additional SPECTRUM Network Management .................................12 Your Network Model ................................................................... 13 Before You Begin ......................................................................13 Optimizing Your Network Model ..................................................14 Understanding Your Network Model ........................................14 Port Connection Representation ........................................16 Completing Your Network Model .............................................17 Generating an Inventory Report of Your Modeled Devices .....18 Comparing the Inventory Report to Your Inventory List ........21 Modeling Devices Manually ...............................................22 Customizing Interface Model Names ..................................22 Resolving Port Connections ...............................................24 Discovering Connections in Container Models ...........................27 Locked Connections ..............................................................27 Rearranging Your Network Model ...........................................28 Adjusting Management Capacity ............................................32 The LogPollAnalyzer Utility ...............................................33 Staggering Polling Intervals ..............................................35 Disabling Polling for a Model or a Model Type ......................37 Polling Devices that are Down ...........................................38 Customizing Your Network Model ................................................39 How to Manage Your Network with SPECTRUM Page 3 Document 1909 Using the Annotation Toolbox ................................................39 Using the Change Background Option .....................................40 Using Organizational Views ....................................................43 Creating a Network Model that Shows the Business Impact of Services on Administrative Structure ...............................44 Creating a Network Model that Shows Administrative Responsibilities ............................................................46 Maintaining Your Network Model .................................................48 Adding and Removing Device Models ......................................49 To add a device model to your network model: ...................49 To remove a device model from your network model: ..........50 Saving and Restoring the Database ........................................50 Creating a Backup ...........................................................50 Restoring a Database ......................................................51 Scheduling Backups .........................................................51 Configuring for Fault Management ............................................. 53 Setting Thresholds ....................................................................53 Alarm Thresholds .................................................................54 Rollup Condition Thresholds and Significance Levels .................54 Configuring Traps .....................................................................55 Creating New Traps ..............................................................56 Checking Devices for Trap Configuration .................................56 Configuring Alarms ....................................................................56 Clearing Old Alarms Automatically ..........................................56 Alarm Policies for Specific Interfaces .......................................57 Alarm Management on DLCI Ports .....................................58 False Management Lost or Contact Lost Alarms ........................59 Using SpectroWATCH ............................................................60 Configuring for Fault Isolation .....................................................61 Configuring General Fault Isolation Parameters ........................61 Configuring Management Neighbors .......................................64 Configuring Maintenance Mode ..............................................68 Maintenance Mode for Devices ..........................................68 How to Manage Your Network with SPECTRUM Page 4 Document 1909 Maintenance Mode for Ports ..............................................69 Configuring Cross-Landscape Fault Correlation .........................70 Cross-Landscape Fault Correlation Example ........................71 Configuring Port Fault Correlation ...........................................72 Port Fault Correlation Options ...........................................72 Port Fault Correlation Criteria ...........................................73 Port Fault Correlation Caveats ...........................................74 Port Fault Correlation Examples ........................................74 Known Port Fault Correlation Anomalies .............................80 Configuring Port Status Monitoring .........................................80 Port Status Polling Criteria ................................................82 Port Status Events and Alarms ..........................................83 Link Trap Handling ..........................................................83 Interface Trap Configuration View .....................................85 PollPortStatus Feature .....................................................86 Wide Area Link Monitoring .....................................................87 Link Fault Disposition .......................................................88 Wide Area Link Monitoring Scenarios .................................89 Port Layer Alarm Suppression ................................................90 Port Criticality .....................................................................90 Live Pipes ...........................................................................91 Enabling or Disabling Live Pipes System-Wide .....................91 Enabling or Disabling Live Pipes on Individual Links .............91 Receiving Port Alarms ......................................................92 Automatic Port Status Alarm Clearing .....................................93 Suggested Port Fault Settings for Optimal Fault Notification .......94 Other Fault Management View Settings ........................................94 Device Model Settings ..........................................................94 Port Model Settings ..............................................................95 Configuring Fault Management for Pingables .................................95 Connecting Two Pingable Models ............................................96 Connecting a Pingable Model to a Device’s Port Model ...............96 How to Manage Your Network with SPECTRUM Page 5 Document 1909 Fault Management ...................................................................... 97 Fault Isolation Overview ............................................................97 Fault Detection ....................................................................... 100 Device Condition Colors ...................................................... 100 Blue (Initial) ................................................................. 100 Green (Normal) ............................................................ 101 Yellow (Minor Alarm) ..................................................... 101 Orange (Major Alarm) .................................................... 101 Red (Lost Contact) ........................................................ 101 Gray (Unknown) ........................................................... 101 Rollup Condition Colors ....................................................... 102 How Is a Rollup Condition Detected? ..................................... 102 Fault Isolation ........................................................................ 104 Enterprise Alarm Manager ................................................... 104 Accessing the Enterprise Alarm Manager .......................... 105 Using the Enterprise Alarm Manager ................................ 106 Audible Alarms .................................................................. 107 Isolating a Fault Through Navigation ..................................... 107 Point and Click .............................................................. 108 Navigate In .................................................................. 108 Chassis Device View ........................................................... 109 Accessing the Chassis Device View .................................. 109 Device Topology View ......................................................... 110 Accessing the Device Topology View ................................ 110 Using the Device Topology View ...................................... 111 Performance View .............................................................. 111 Accessing the Performance View ..................................... 112 Using the Performance View ........................................... 112 Beyond the Performance View ............................................. 118 SPECTRUM Intelligence ............................................................ 119 Static Configuration of Device Models ........................................ 119 Dynamic Configuration of Device Models .................................... 120 How to Manage Your Network with SPECTRUM Page 6 Document 1909 The Pulled Board List .......................................................... 120 Router Reconfiguration Events ............................................. 121 Condition vs. Rollup Condition .................................................. 122 Attributes Determining Condition and Rollup Condition ............ 123 Condition and Rollup Condition Sensitivity ............................. 125 Rollup Condition Flow ......................................................... 126 Example of Rollup Condition Propagation ............................... 127 Organization Models ........................................................... 131 Fault Isolation ........................................................................ 131 How Model Category Affects Contact Status ........................... 132 Examples .......................................................................... 136 Duplicate Addresses ................................................................ 141 Automatic Naming and Addressing in SPECTRUM ......................... 143 Monitor Points ........................................................................ 144 Monitor Point Statistics ....................................................... 144 Composite Monitor Point Statistics ................................... 145 Discrete Monitor Point Statistics ...................................... 146 Monitor Point Calculations in SpectroWATCH ..................... 151 Detection of Firmware Problems ................................................ 151 Device Threshold Events .......................................................... 152 Redundancy ........................................................................... 152 Redundant SpectroSERVERs (Fault Tolerance) ....................... 152 Redundant Paths ................................................................ 152 Configuring Device Redundancy ........................................... 153 Editing the Redundancy Preferred Addresses list ................ 153 Device Redundancy Events ............................................. 155 Alternate Path Repeater Management ........................................ 156 Device Lost and Found ............................................................. 156 Device Type Verification ........................................................... 156 Device Type Mismatch ........................................................ 158 In-Band Configuration of Device Alarms ..................................... 158 Interface Intelligence .............................................................. 158 Interface Alarms ................................................................ 160 How to Manage Your Network with SPECTRUM Page 7 Document 1909 Interface Events ................................................................ 161 Index ........................................................................................ 163 How to Manage Your Network with SPECTRUM Page 8 Document 1909 Preface In This Section Intended Audience [page 9] Text Conventions [page 9] Document Feedback [page 10] Online Documents [page 10] Intended Audience This guide is intended for system administrators who are using SPECTRUM to manage their networks. Text Conventions The following text conventions are used in this document: Element Convention Used Example Variables Courier and Italic in Type the following: (The user supplies a value angle brackets (<>) for the variable.) The directory where you installed SPECTRUM <$SPECROOT> DISPLAY=<workstation name>:0.0 export display Navigate to: <$SPECROOT>/app-defaults (The user supplies a value for the variable.) Solaris and Windows directory paths On-screen text Unless otherwise noted, directory paths are common to both operating systems, with the exception that slashes (/) should be used in Solaris paths, and backslashes (\) should be used in Windows paths. <$SPECROOT>/app-defaults Courier The following line displays: on Solaris is equivalent to <$SPECROOT>\app-defaults on Windows. path=”/audit” User-typed text Type the following path name: Courier C:\ABC\lib\db How to Manage Your Network with SPECTRUM Page 9 Document 1909 Element Convention Used Example Cross-references Underlined and hypertextblue See Document Feedback [page 10]. References to SPECTRUM documents (title and number) Italic OneClick Console User Guide (5130) Document Feedback Please send feedback regarding SPECTRUM documents to the following e-mail address: [email protected] Thank you for helping us improve our documentation. Online Documents SPECTRUM documents are available online at: http://www.aprisma.com/support/secure/manuals/ Check this site for the latest updates and additions. How to Manage Your Network with SPECTRUM Page 10 Document 1909 Introduction SPECTRUM is a powerful and flexible network management software program that provides a network administrator with effective and reliable network management tools. Reliable network management has become a necessity to maintain a competitive edge in today’s market place. SPECTRUM’s diversity of features and functionality can seem intimidating to an administrator just beginning to use the product. Not only does the SPECTRUM core* product provide a wide variety of features and functionality, but an additional array of separately purchasable Aprisma or third-party add-on components offers the means for further customizing or extending SPECTRUM. Given these extensive management capabilities, the need for a book describing the use of these network management tools has become increasingly important. This document is an attempt to address the need for a “how to” approach in using SPECTRUM to effectively manage a network. The focus will be on the management features included with the core product and how to use them. * Core product refers to those components included in the basic SPECTRUM package. How to Manage Your Network with SPECTRUM Page 11 Document 1909 SPECTRUM Core and Network Management Using SPECTRUM to successfully manage your network involves reducing network downtime, improving network performance, and maintaining a clear and up-to-date model of your network. SPECTRUM functionality allows you to optimize your network model, supports proactive fault management and fault isolation, and provides network performance enhancement capabilities. The flexibility of SPECTRUM allows an administrator to: • Customize the network model to optimize ease-of-use, performance, and fault management. • Keep the network model up-to-date as devices or subnets are added or removed. • Identify and isolate network faults down to a port level. Additional SPECTRUM Network Management Ethernet IEEE 802.3 is the network management standard that is used in the SPECTRUM management examples described in this document. Additional separately purchasable SPECTRUM network management products provide management of other networking standards. Some of these products are: • Frame Relay Manager • ATM Circuit Manager • VLAN Fault Isolation • Non-Persistent Connections Manager • RMON Contact Aprisma for more information about these products. How to Manage Your Network with SPECTRUM Page 12 Document 1909 Your Network Model This section describes how to refine an auto discovered network model to improve management capabilities. In This Section Before You Begin [page 13] Optimizing Your Network Model [page 14] Customizing Your Network Model [page 39] Maintaining Your Network Model [page 48] Before You Begin In order to properly model your network and facilitate the management techniques described in this document, it is important that the following prerequisites are met: • Your network design meets all established Ethernet specifications (as outlined by the IEEE 802.3 group). • You have an accurate network diagram mapping the physical placement of your devices, nodes, and cable. (This network diagram and list will be used to verify that SPECTRUM Autodiscovery has modeled all the devices and properly resolved all port connections.) • You have an inventory list of all the devices in your network. • Your network has been auto discovered initially to allow SPECTRUM to identify and communicate with all the devices on your network. When AutoDiscovery is run, SPECTRUM may not be able to identify devices that are temporarily off the network or not allowing management communication. Completing your network model will require some subsequent manual modeling. How to Manage Your Network with SPECTRUM Page 13 Document 1909 Optimizing Your Network Model Once your network has been auto discovered, you may want to perform the tasks described in this section to complete your network model as well as optimize management capabilities. Understanding Your Network Model SPECTRUM’s network representation is based on logical relationships and rules and may not look exactly like your network diagram. AutoDiscovery uses address tables and ICMP pings to identify subnet address ranges and devices within those ranges. Once discovered, those devices and subnets are modeled by SPECTRUM. Figure 1 and Figure 2 illustrate the relationship between the actual devices, the network diagram, and SPECTRUM’s network model. These figures show how your network is represented by SPECTRUM. How to Manage Your Network with SPECTRUM Page 14 Document 1909 Figure 1: Your Actual Network WanLink Router IPClassBLAN EMM-E6 EMM-E6 LAN 132.177.117.0 Fanout A LAN 132.177.118.0 Your Actual Network How to Manage Your Network with SPECTRUM Page 15 Document 1909 Figure 2: SPECTRUM Representation of Your Network SpectroGRAPH: Topology: LAN SpectroGRAPH: Topology: Universe File View Tools File Help Bookmarks View Tools Bookmarks Help 132.177.118.0 132.177.117.0 EMM-E6 LAN_802_3 LAN_802_3 132.177.118.0 VNM EMM-E6 132.177.119.0 LAN_802_3 LAN_802_3 SpectroGRAPH: Topology: 132.177.117.0 File View Tools SpectroGRAPH: Topology: 132.177.118.0 Bookmarks File View Tools Bookmarks Workstation Workstation Workstation Workstation Bridge CSIRptr Bridge Workstation CSiRptr Workstation CSI Repeater CSIRepeater Workstation Workstation Fanout A Port Connection Representation SPECTRUM represents port level connections between devices in the Device Topology views. The Device Topology view shows you the devices that are connected to each port for the selected device. Figure 3 provides an example of a Device Topology view for the EMM-E6 device. How to Manage Your Network with SPECTRUM Page 16 Document 1909 Figure 3: Device Topology View for the EMME/EMM-E6 SpectroGRAPH: Device Topology: EMM_E6 File View Tools Bookmarks Help M I I M M I 1 M 0 F u n k n o F O R M I M E M M l E 6 BRtrCSIEMM-E6 LAN 802.3 LAN 802.3 LAN 802.3 Network A Network B Network C CSIRptr CSIRptr CSIRptr 1 FWD ETHERNET 2 FWD ETHERNET 3 FWD ETHERNET 0.0.2.1.0.5.1.A:BC 0.0.2.1.0.5.1.A:BC 0.0.2.1.0.5.1.A:BC 0 0 0 If SPECTRUM does not understand which port a device is connected to, the device model will appear in the Off-Page Reference panel. AutoDiscovery should correctly place all of these connections; if it does not, you will need to refer to Resolving Port Connections [page 24] to resolve the connections and allow SPECTRUM to properly monitor the device. Completing Your Network Model When AutoDiscovery is initially run, SPECTRUM may not be able to identify devices that are temporarily off the network or not allowing management How to Manage Your Network with SPECTRUM Page 17 Document 1909 communication. Although you may have run AutoDiscovery twice during a 48-hour period as recommended, some devices may not have been identified by SPECTRUM. To identify and properly locate any undiscovered devices or unresolved port connections, you must analyze your network diagram. The process of completing your network entails the following: • Generating an Inventory Report of Your Modeled Devices [page 18] • Comparing the Inventory Report to Your Inventory List [page 21] • Modeling Devices Manually [page 22] • Resolving Port Connections [page 24] Generating an Inventory Report of Your Modeled Devices Devices that were not contacted during the Autodiscovery process need to be identified and discovered individually or within a range. To determine what devices have been modeled, generate an Inventory Report as follows: 1. Select Reports then Generate from the Tools menu. The SPECTRUM Report Generator view will appear Figure 4. Figure 4: SPECTRUM Report Generator View SPECTRUM Report Generator File Applications Help Report Type Alarm Inventory Statistical Landscapes... Event Up/Down Time Models banshee Event Filters Report Format... Output File... Date Range Day Post Generate Script Report Depth Site Name General Output Format Graphical Tabular Display (.GRF) ASCII (.TAB) Postscript (.ps) pace/Spectrum/report.config/default.rptrc GIF (.gif) pace/Spectrum/report.config/default.rptrc How to Manage Your Network with SPECTRUM Page 18 Postscript (.ps) Document 1909 2. In the Report Type panel of the Report Generator view, select Inventory. 3. Select ASCII (.TAB) under Tabular for the output format. 4. If applicable, deselect any Graphical selections. 5. To select the models to be included in the inventory list, click on the Model Types button to open the Model Selection dialog box (Figure 5). Figure 5: Model Selection Dialog Box SRG: Model Types Scope: Under Landscape Under Model Landscapes Model Types catapult 2E253_49R 2E253_49R_Mdul 2E42_27 2E42_27_Module 2E43_47 2E43_47_Module 2E43_51 2E43_51_Module 2E48_27 2E48_27_Module 2E49_27 2E49_27_Module 2E50_28 2E50_28_Module Select All Case Sensitive Deselect All Search OK Cancel 6. For the purpose of identifying undiscovered devices, highlight your landscape in the left-hand panel, and click on Select All at the bottom of the right-hand panel to include all models within that landscape. 7. Click OK. SpectroGRAPH displays the Report Generator view. 8. Select Detailed using the Report Depth button. 9. Click the Report Format button to open the SRG:Report Format dialog box (Figure 6). How to Manage Your Network with SPECTRUM Page 19 Document 1909 Figure 6: Report Format Dialog Box Srg: Report Format Filter Spectrum/sg-support/csrib/*.rib* Directories Files .. 3comgenbdgapp 3omnetbld 3comnetbld2 3comsrcrteapp Atmifport Atm_network Selection Ace/spectrum/sg-support/csrib/* Ok Filter Cancel 10. Select Invt_Det.rib as the format under Files. This report information block file, located in the <$SPECROOT>/SG -Support/ CsRib/Inventory directory, provides a standard format for a detailed inventory report. (You can customize these reports with the Format option. See the SPECTRUM Report Generator User’s Guide (0881).) 11. Click OK. SpectroGRAPH displays the Report Generator view. 12. Click the Output File button to open the SRG:Output File dialog box. 13. Select an output file directory (the default directory is report.output). Enter an output file name in the Selections field and click OK. 14. Select Generate from the File menu of the Report Generator view. 15. Click OK in the Report Generation Invoked information box. 16. Another information box displays when the report process has completed successfully. Click OK. 17. Print the file located in the <$SPECROOT>output.report/<your file name.TAB>. How to Manage Your Network with SPECTRUM Page 20 Document 1909 Figure 7 provides an example of an inventory report. Figure 7: Example Inventory Report SPECTRUM Network Inventory Report Date: LandscapeID: User Name: Site Name: Page: 1 06/10/96 8:52:00 0x400000 arlo_admin Headquarters Model Type Model Name IP Address BdgCSINB30 Bridge #1 132.177.2.28 GnSNMPDev tutor 132.177.1.7 GnSNMPDev Workstation #2 132.177.1.5 GnSNMPDev Workstation #3 132.177.1.6 GnSNMPDev Workstation #4 132.177.2.18 GnSNMPDev Workstation #6 132.177.2.24 GnSNMPDev Workstation #8 132.177.2.42 IPClassB Automapped LAN LAN Automapped LAN LAN_802_3 Automapped LAN Hub_CSI_IRM3 Hub #1 132.177.1.52 Hub_CSI_IRM3 Hub #2 132.177.2.14 Hub_CSI_MRXi Hub #3 132.177.2.36 Pingable PsPrinter #1 132.177.1.44 Rtr_Cisco Router #1 132.177.1.1 WS_SGI Workstation #5 132.177.2.23 WS_SGI Workstation #7 132.177.2.47 MAC Address Create Date 0.0.1D.3.4A.74 06/09/94 8.0.20.B.8B.56 06/09/94 8.0.20.45.89.4D 06/09/94 8.0.20.77.8.5 06/09/94 8.0.20.0.43.AC 06/09/94 8.0.20.CC.AA.BB 06/09/94 8.0.20.12.B2.32 06/09/94 06/09/94 06/09/94 06/09/94 0.0.1D.1B.3.6 06/09/94 0.0.1D.3.34.5A 06/09/94 0.0.1D.B.8B.56 06/09/94 0 06/09/94 0.0.C.80.1.26 06/09/94 8.0.69.B.8B.56 06/09/94 8.0.69.B.8B.56 06/09/94 Contact ESTB ESTB ESTB ESTB ESTB LOST ESTB ESTB ESTB ESTB ESTB ESTB ESTB ESTB ESTB ESTB ESTB Last Contacted Vendor Object ID 06/10/94 08:51 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 Undefined Undefined Undefined 08/20/93 08:50 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 06/10/94 08:51 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 06/10/94 08:50 1.3.6.1.4.1.34 Comparing the Inventory Report to Your Inventory List The Inventory Report lists all models that have been discovered. This includes LANs, applications, and device models. Compare this list to your inventory list to determine if any devices within your network were not discovered. The applications that are listed in the report are not represented in a Topology view. For the purpose of completing your network, the applications are not needed for the comparison process. However, you may need the inventory of application models if you wish to ensure they will be polled by SPECTRUM (see the Note on [page 37]). Note the IP address or range of any missing device(s). This information will be used in Modeling Devices Manually [page 22] to discover additional devices if necessary. If the initial Inventory Report contains LANs or Fanout models, you may have to run inventory reports for each LAN or fanout to determine which models are contained within them. The inventory report provides the information shown in Figure 8. How to Manage Your Network with SPECTRUM Page 21 Document 1909 Figure 8: Using the Inventory Report The IP address for that device or LAN. The unique name of the device or LAN. SPECTRUM Network Inventory Report Date: LandscapeID: User Name: Site Name: Page: 1 06/10/96 8:52:00 0x400000 arlo_admin Headquarters Model Type Model Name BdgCSINB30 GnSNMPDev GnSNMPDev GnSNMPDev GnSNMPDev GnSNMPDev GnSNMPDev Bridge #1 tutor Workstation #2 Workstation #3 Workstation #4 Workstation #6 Workstation #8 IP Address 132.177.2.28 132.177.1.7 132.177.1.5 132.177.1.6 132.177.2.18 132.177.2.24 132.177.2.42 MAC Address Create Date Contact 0.0.1D.3.4A.74 8.0.20.B.8B.56 8.0.20.45.89.4D 8.0.20.77.8.5 8.0.20.0.43.AC 8.0.20.CC.AA.BB 8.0.20.12.B2.32 06/09/94 06/09/94 06/09/94 06/09/94 06/09/94 06/09/94 06/09/94 ESTB ESTB ESTB ESTB ESTB LOST ESTB Last Contacted Vendor Object ID 06/10/94 08:51 06/10/94 08:50 06/10/94 08:50 06/10/94 08:50 06/10/94 08:50 06/10/94 08:50 06/10/94 08:50 1.3.6.1.4.1.34 1.3.6.1.4.1.34 1.3.6.1.4.1.34 1.3.6.1.4.1.34 1.3.6.1.4.1.34 1.3.6.1.4.1.34 1.3.6.1.4.1.34 The Model Type Name for that device or LAN. Modeling Devices Manually If you have identified any devices that need to be modeled to complete your network model, they should be modeled manually. The following are examples of devices that require some manual modeling to complete your network: • MMAC-Plus and its associated modules • FDM Module • ForeRunner ATM Switch Module Refer to the specific management module guide for modeling instructions. Customizing Interface Model Names Note: All changes to the interface naming suffix configuration are done at the device model level rather than at the interface model level. By default, when an interface is modeled, its model name is created using both the device model name and the interface’s value for ifIndex as a suffix. For example, a device model named 192.168.9.17 might contain interface models named 192.168.9.17_1, 192.168.9.17_2, and so on. How to Manage Your Network with SPECTRUM Page 22 Document 1909 Using the Interface Model Name Suffix option available in the Device tab of SPECTRUM’s Global Attribute Editor for a given device model, you can select the attribute value to be used as an interface model name suffix. The following attributes can be chosen as suffixes: ifIndex, ifDescr, ifAlias, or ifName. These alternative interface model naming schemes can provide valuable information about a given interface. For example, some devices store board and port information in the value of ifName, which can be useful to have on hand when viewing an interface alarm in an alarm console. Configuration of the Interface Model Name Suffix option can be done either on a device model by device model basis, or by device model type. For example, you may want to set all Enterasys device model types to use ifDescr for an interface naming suffix, but for Rtr_Cisco device model types, you may want to set one subset of models to use ifName and another subset to use ifAlias (see Example: Configuring Interface Model Name Suffix [page 23]). This will depend on what the particular device settings for these attributes are in your network environment. In Search Manager, the interface name suffix can be set for a device model or models by selecting the model(s) from Search Manager search results and choosing Management > Set Attribute Values... and setting the Interface Model Name Suffix option on the Device tab. Note: Selecting Management > Rename Interface Models in Search Manager recalculates interface model names. This only needs to be used to update SPECTRUM when new values for interface suffix attributes have been set on the device. For more information, see the Search Manager User’s Guide (2383). Example: Configuring Interface Model Name Suffix Follow these steps to configure the Interface Model Name Suffix option for two subsets of Rtr_Cisco device models, setting one subset to ifName and the other subset to ifAlias: 1. In Search Manager, search for and select the subset of Rtr_Cisco device models you want to set to have interface models named with a suffix of ifName. 2. Choose Management > Set Attribute Values... 3. Select the Device tab in the Global Attribute Editor and set the Interface Model Name Suffix option to ifName. How to Manage Your Network with SPECTRUM Page 23 Document 1909 4. Click Apply to name this subset with a suffix of ifName. 5. Click Close in the Operation Results dialog box and the Global Attribute Editor window. 6. In Search Manager, search for and select the subset of Rtr_Cisco device models you want to set to have interface models named with a suffix of ifAlias. 7. Choose Management > Set Attribute Values... 8. Select the Device tab in the Global Attribute Editor and set the Interface Model Name Suffix option to ifAlias. 9. Click Apply to name this subset with a suffix of ifAlias. 10. Click Close in the Operation Results dialog box and the Global Attribute Editor window. Resolving Port Connections AutoDiscovery automatically identifies and resolves port connections between devices. It may be necessary to resolve some port connections manually to allow SPECTRUM to properly manage all devices on your network. To resolve port connections the following must be done: • Systematically go through all the Device Topology views to determine if you have any unresolved port connections. • Resolve each unresolved port connection. The Device Topology View The Device Topology view represents a device in terms of its ports and port connections. You can use the Device Topology view to: • Examine existing connections to a device. • Make new connections to other devices. • Navigate to other views via the icons displayed in the Device Topology view. • Access other views (Performance, Application, etc.) through the Device Icon panel and the menu. • Select which board’s ports are shown in the Connections Panel for a multi-slot device, such as a hub (Figure 9). How to Manage Your Network with SPECTRUM Page 24 Document 1909 • Resolve any undefined device connections represented by Off-Page Reference icons shown in the Unresolved Off-Page Reference Icon Panel. • Add, copy, erase, cut, paste, and destroy icons. You can access the Device Topology view using one of the following methods: • Highlight the icon and select DevTop from the Icon Subviews menu. • Double-click on the Device Topology down arrow on the icon. • Place the mouse pointer over the icon, click and hold the right mouse button and select DevTop from the pop-up Icon Subviews menu. Figure 9 provides an example of a Device Topology view. Figure 9: Example Device Topology View SpectroGRAPH: Device Topolog File View Tools Bookmarks Unresolved Off-Page Reference Icon Network Icon that contains the device connected to this port Icons for models that are connected to a specific port LAN 802.3 LAN 802.3 Network A Network B N Port Connections CSIRptr Interface Icons CSIRptr 1 FWD ETHERNET 2 FWD ETHERNET 0.0.2.1.0.5.1.A:BC 0.0.2.1.0.5.1.A:BC 0 How to Manage Your Network with SPECTRUM Page 25 0 Document 1909 Using the Device Topology View to Identify and Resolve Port Connections The Device Topology view indicates unresolved port connections between devices as Off-Page Reference icons in the Off-Page Reference Panel. Resolve the port connections as follows: 1. Open the Device Topology view for the device. If no Off-Page Reference icons appear in the Off-Page Reference Panel, continue to the next device. 2. If an Off-Page Reference icon appears for the device, use your network map to verify the physical connections between devices to ensure proper port connections. 3. Select File > Edit in the Device Topology view. 4. Highlight the Off-Page Reference icon (Figure 10) and drag it to the proper port. You may have to scroll the Port Connections Panel to view all the port connections. Release the mouse button when the icon is placed over the correct port. Figure 10: Resolving a Port Connection SpectroGRAPH : Device Topology : 111.222.333.4 File View Tools Help Bookmarks 111.222.333.4 C A Y M A N Highlight C R M 2 R E I M I M R 1 0 B T I M I M C M I M E M M - E - EMM-E6 Drag Release A ON ETHERNET B OFF ETHERNET C OFF ETHERNET D ETHE 0:0:1D:19:AC:52 0:0:1D:19:AC:53 0:0:1D:19:AC:54 0:0:1D:1 0:0:0: 111.222.333.4 0:0:0:0 0:0:0:0 How to Manage Your Network with SPECTRUM Page 26 Document 1909 5. Repeat step 4 for each Off-Page Reference icon in this view. 6. Choose File > Close Edit to exit edit mode. 7. Repeat step 2 through step 6 for each device with unresolved port connections. Discovering Connections in Container Models The Discover LANs functionality uses the parameters you have chosen in the AutoDiscovery Model Information view to discover and map the connections within the selected container model. See the Distributed SpectroSERVER (2770) guide for complete information on these parameters and their impact on how the connections will be modeled. Note: When you run this discovery, SPECTRUM maps connections for both layer 2 and layer 3 devices. To access the Container Discover Connections view, highlight the Container and choose Discover Connections from the Edit > Icon Subviews menu. Figure 11: Container Discover Connections View Locked Connections Any ports used in device connections that are manually resolved have their LockConnection attribute (0x000129f1) set to 1, the locked state. This is How to Manage Your Network with SPECTRUM Page 27 Document 1909 true of any manually configured connection, whether it is created by copying and pasting in a Topology view, through the CLI, or using the method described in the preceding procedure. The locked state preserves manually configured connections when AutoDiscovery is subsequently run on the devices. SPECTRUM will not remove such connections. All connections resolved by AutoDiscovery will have the LockConnection attribute set to 0, the unlocked state. Note: Links modeled in earlier versions of SPECTRUM (whether by AutoDiscovery or manual configuration) and migrated into SPECTRUM 7.0 and later during the upgrade process will also be locked by default. The network administrator may have a need to change this link setting in SPECTRUM, which effectively changes the state of the LockConnection attribute on the ports at either end of the link. To lock or unlock a connection: 1. Select the link in the Topology view, then choose View > Lock Connection... to open the Connection Lockdown dialog box. 2. In the Connection Lockdown dialog box, set the toggle button to the desired position and click OK. raised button is unlocked recessed button is locked Rearranging Your Network Model SPECTRUM allows you to group devices at any level to reduce the complexity of your Topology views. This is done by creating container models, such as LAN, Fanout, Network, FDDI, etc., and placing groups of modeled devices within them. SPECTRUM intelligence functions, such as rollup conditions, still apply and enable you to effectively manage those devices. In the following example, a LAN_802_3 Topology view displays a hub and its connected workstations, as shown in Figure 12. We will create a Fanout icon to represent the devices at the LAN_802_3 Topology level, then cut How to Manage Your Network with SPECTRUM Page 28 Document 1909 and paste the workstation icons into the Fanout model. The procedure below can be used for all container models. Figure 12: Example Network Topology View SpectroGRAPH : Primary Landscape File View Tools Bookmarks Help Router #1 Hub # 2 Hub_CSI_IRM3 1. Review your network and identify groups of devices that could logically be placed in a group model. Note the Hub and connected workstations in this example. 2. Select Edit from the File menu of the LAN_802_3 Topology view. 3. Select New Model... from the Edit menu. 4. Select Fanout from the Select Model Type dialog box and click OK. 5. Enter the name of the Fanout (TechPubs in Figure 13) and click OK in the Creation dialog box. 6. Select Close Edit from the File menu to exit Edit mode. 7. Open the Fanout Cablewalk view by double-clicking on the down arrow shown in Figure 13. How to Manage Your Network with SPECTRUM Page 29 Document 1909 Figure 13: Creating a Fanout Model SpectroGRAPH : Primary Landscape File View Tools Bookmarks Help Router #1 Hub # 2 Hub_CSI_IRM3 TechPubs Down arrow 8. Select Edit from the File menu for the Fanout Cablewalk view (Figure 14) and for the LAN_802_3 Topology view (Figure 12). Figure 14: Fanout Cablewalk SpectroGRAPH : Cable Walk : Fanout File View Tools Bookmarks Help TechPubs 9. In the LAN_802_3 Topology view, hold down the shift key and use the mouse pointer to highlight each workstation icon connected to the hub (Figure 15). How to Manage Your Network with SPECTRUM Page 30 Document 1909 Figure 15: Selecting the Models to be Grouped SpectroGRAPH : Primary Landscape File View Tools Bookmarks Help Router #1 Hub # 2 Hub_CSI_IRM3 TechPubs 10. Select Cut from the Edit menu in the LAN_802_3 Topology view. 11. Click OK in the Confirm dialog box. 12. Select Paste from the Edit menu in the Fanout Cablewalk view. 13. Select Close Edit from the File menu in the Fanout Cablewalk view and LAN_802_3 Topology views. These two views will appear as shown in Figure 16. How to Manage Your Network with SPECTRUM Page 31 Document 1909 Figure 16: New Grouped Views SpectroGRAPH : Cable Walk : Fanout File View Tools Bookmarks SpectroGRAPH : File View Tools Bookmarks Help Help TechPubs Router #1 Hub # 2 TechPubs Hub_CSI_IRM3 Adjusting Management Capacity SPECTRUM polls devices at regular intervals to access the management information that SPECTRUM’s intelligence uses to perform fault isolation. However, this polling activity consumes considerable bandwidth and host system resources. A key part of optimizing your network model is to selectively adjust this trade-off between bandwidth use and management capacity so that resources are deployed where they are needed most. SPECTRUM provides two main ways of doing this: • Staggering Polling Intervals [page 35] • Disabling Polling for a Model or a Model Type [page 37] Before you implement one of these methods, you may want to assess how polling is currently configured for models in your database. The The LogPollAnalyzer Utility [page 33] can provide you with this information. How to Manage Your Network with SPECTRUM Page 32 Document 1909 The LogPollAnalyzer Utility The LogPollAnalyzer utility displays information on the attributes being logged or polled by models in the database. Results are broken down by model type. The utility is located in the <$SPECROOT>/SS-Tools directory and can be run from this directory on the command line using the following syntax. LogPollAnalyzer -l <landscape handle> -c -i Where: -l <landscape handle>: Specifies the SPECTRUM landscape that you want the LogPollAnalyzer to report on. This is a required parameter. -c: specifies that models will not be listed under the same entry unless they log and poll the same attributes. This only affects models that log and poll. This is an optional parameter. -i: specifies that models will not be listed under the same entry unless their logging and polling intervals match. Note that the logging and polling intervals will not be displayed at all unless this parameter is specified. This is an optional parameter. The -c and the -i parameters can be used at the same time. Sample Output The following sample shows output generated without using the -c or -i parameter. 1 models of type 2H22_08R (0x1c80014) are polling the following attributes: contTypePhysicalChanges (0x2309c9) contTypeLogicalChanges (0x2309ca) ifNumber (0x100c3) sysUpTime (0x10245) 431 models of type SSR_PortIf (0x2c60006) are logging the following attributes: Load (0x220042) Packet_Rate (0x220093) Contact_Status (0x10004) Error_Rate (0x220054) Discard_Rate (0x220055) How to Manage Your Network with SPECTRUM Page 33 Document 1909 1 models of type 2H252_25R (0x1c8001e) are logging the following attributes: Load (0x10019) PacketRate (0x1001a) SoftErrorRate (0x1001b) Contact_Status (0x10004) HardErrorCount (0x1001c) 1 models of type 2H252_25R (0x1c8001e) are polling the following attributes: contTypePhysicalChanges (0x2309c9) contTypeLogicalChanges (0x2309ca) ifNumber (0x100c3) sysUpTime (0x10245) 1 models of type JnprRedundRtr (0x3b1000e) are logging the following attributes: Contact_Status (0x10004) 1 models of type JnprRedundRtr (0x3b1000e) are polling the following attributes: ifNumber (0x100c3) ifStackLastChange (0x11f7c) ifTableLastChange (0x11f7d) 3 models of type 8H02_16 (0x1c80007) are polling the following attributes: contTypePhysicalChanges (0x2309c9) contTypeLogicalChanges (0x2309ca) ifNumber (0x100c3) sysUpTime (0x10245) 1 models of type Dialer_IF_Port (0x22002f) are logging the following attributes: Load (0x220042) Packet_Rate (0x220093) Contact_Status (0x10004) Error_Rate (0x220054) Discard_Rate (0x220055) How to Manage Your Network with SPECTRUM Page 34 Document 1909 Staggering Polling Intervals Most SPECTRUM device models are polled every 60 seconds by default. You can change the default polling interval for an individual device model or for all models of particular model type. But keep in mind the above-mentioned trade-off between bandwidth use and management capacity, which applies as follows: • If you increase the time between polls, you will use less bandwidth for management traffic, but you will receive device status updates less frequently. • If you decrease the time between polls, you will have more frequent updates of device status. However, you will use more bandwidth for management traffic. One way to achieve both goals (i.e., reduce network management traffic and enhance fault management) is to stagger polling intervals as shown in Figure 17. If all the devices in this example were using the default polling interval of 60 seconds, they would all be using bandwidth every 60 seconds. However, by leaving the polling interval for the router at the default of 60 seconds and changing the polling interval of all the other devices to 600 seconds, the SpectroSERVER resource utilization is reduced. Moreover, you will not lose management capabilities because if anything happens to the devices downstream from the router, polling will be interrupted and an alarm will be generated. Figure 17: Staggering Polling Intervals WanLink Polling Interval=60 Router Polling Intervals=600 Polling Interval=600 Polling Interval=600 How to Manage Your Network with SPECTRUM Page 35 Document 1909 How To Change the Polling Interval You can change the polling interval for any device by entering a new value in the Poll Interval field in the device model’s Model Information view (be sure to select File > Save All Changes after you enter the new value). or You can also use a Command Line Interface (CLI) script called update_mtype to change the polling interval either for a specific model or for all models of a specific model type. The script requires you to supply three arguments: the model type name, the polling interval attribute ID (which is 0x10071), and the polling interval value in seconds. To run this script, do the following: 1. Navigate to the following location in your top-level SPECTRUM directory: /vnmsh/sample_scripts 2. Enter the following command: ./update_mtype <model type name> 0x10071 <new polling interval in seconds> For example, to set the polling interval to 600 seconds for one or more models of type Host_Sun, you would enter: ./update_mtype Host_Sun 0x10071 600. This will display a list of all models of the specified type, showing for each one the model handle, the model name, the model type handle, and the model type name, as in the following example: 0x100001 workstation1 0x60000 Host_Sun 0x100002 workstation2 0x60000 Host_Sun 0x100003 workstation3 0x60000 Host_Sun 3. To apply the new polling interval to a specific model in the list, enter the model handle (e.g., 0x100001) or the model name (e.g., workstation1). To apply the new polling interval to all models of the specified type, enter the model type handle (e.g., 0x60000) or the model type name (e.g., Host_Sun). 4. The system will indicate success or failure and disconnect you from CLI. How to Manage Your Network with SPECTRUM Page 36 Document 1909 Note: Polling intervals also apply to application models, many of which have an initial setting of zero, which in effect disables polling (although the preferred method for disabling polling for any model is to set the Polling Status attribute to FALSE as explained in Disabling Polling for a Model or a Model Type [page 37]). You can reset the polling interval for application models using the same methods described above for device models. Indeed, it is recommended that you check your inventory report (see Generating an Inventory Report of Your Modeled Devices [page 18]) for application models and run the update_mtype script on each global application model type to set a polling interval of 60 seconds. Disabling Polling for a Model or a Model Type In addition to the strategy of increasing default polling intervals for selected models to conserve bandwidth, you may decide that the status of certain devices is not worth the bandwidth that polling them requires, even at longer intervals. For example, some network administrators may choose not to model endpoint devices such as workstations so that they do not have to deal with the alarms that occur each time these devices are powered down. If you wish to have the endpoints modeled, but do not wish to expend bandwidth with network polling traffic, you can disable polling to these models (or to any models) by changing the value of the Polling Status attribute to FALSE as described below. How To Change the Polling Status You can change the polling status for any model by toggling the Polling Status selector button in the model’s Model Information view (be sure to select File > Save All Changes after you change the setting). You can also use the Command Line Interface (CLI) script called update_mtype to change the polling status either for a specific model or for all models of a specific model type. The script requires you to supply three arguments: the model type name, the polling status attribute ID (which is 0x1154f), and the polling status value (TRUE or FALSE). To run this script, do the following: 1. Navigate to the following location in your top-level SPECTRUM directory: /vnmsh/sample_scripts 2. Enter the following command: How to Manage Your Network with SPECTRUM Page 37 Document 1909 update_mtype <model type name> 0x1154f <new polling status value> For example, to disable polling for one or more models of type Host_Sun, enter: update_mtype Host_Sun 0x1154f false. This will display a list of all models of the specified type, showing for each one the model handle, the model name, the model type handle, and the model type name, as in the following example: 0x100001 workstation1 0x60000 Host_Sun 0x100002 workstation2 0x60000 Host_Sun 0x100003 workstation3 0x60000 Host_Sun 3. To apply the new polling status setting to a specific model in the list, enter the model handle (e.g., 0x100001) or the model name (e.g., workstation1). To apply the new polling interval to all models of the specified type, enter the model type handle (e.g., 0x60000) or the model type name (e.g., Host_Sun). 4. The system will indicate success or failure and disconnect you from CLI. Note: The Polling Status value takes precedence over the Polling Interval value in terms of enabling/disabling various periodic external requests to a model. Even though setting the Polling Interval to zero will automatically change the Polling Status to FALSE, if you then reset the Polling Status to TRUE, requests generated by SPECTRUM inference handlers could still occur, regardless of the fact that the Polling Interval is still set to zero. However, to enable normal SPECTRUM polling for fault isolation purposes, the Polling Interval would have to be manually reset to a non-zero value. Polling Devices that are Down When contact with a device has been lost, SPECTRUM will continue to poll the device to see if the contact status has changed. By default, SPECTRUM will poll devices that are down once every third polling interval. For How to Manage Your Network with SPECTRUM Page 38 Document 1909 example, if the device’s polling interval is set to 60 seconds, when the device is down, SPECTRUM will poll the device once every 180 seconds. To change the interval at which SPECTRUM will poll a device that is down, you can insert the following syntax into the <$SPECROOT>/SS/.vnmrc file: down_device_poll_interval_multiplier=<user_defined_multiplier> For example, if the <user_defined_multiplier>=2 and the device’s polling interval is 60 seconds, then the down device will be polled once every 120 seconds (2*60=120). Customizing Your Network Model SPECTRUM provides network model customization tools that enable you to enhance your management capabilities. The Annotation Toolbox provides a set of graphic and text tools that you can use to annotate your SPECTRUM views. For example, you could use the Annotation Toolbox to label certain devices in a view. The Change Background feature lets you adjust the background color and window size for views, or custom select an image to be used as the background. Organizational and Location Views allow you to copy device models into views that help you administer your network from a location or administrative responsibility perspective, for ease of management. Using the Annotation Toolbox The Annotation toolbox gives you access to a color palette and drawing tools (line, circle, box, and text). There are also a variety of options for changing fonts, line width, and colors. The Annotation Options window is described in more detail in Annotation Toolbox (2520). Figure 18 provides an example of how you might annotate a Topology view to help understand SPECTRUM’s representation of your network. How to Manage Your Network with SPECTRUM Page 39 Document 1909 Figure 18: Annotation of a Topology View SpectroGRAPH: Topology File View Tools Bookmarks Help Wiring Closet 45 - 2nd 132.177.117.0 132.177.118.0 EMM-E6 LAN_802_3 LAN_802_3 Technical Documentation Building 45 2nd Manufacturing Building 45 1st 132.177.118.0 LAN_802_3 Using the Change Background Option This option allows you to change background color or select a background image for views, and to set the maximum size of the window for the current GIB View. The Background Color button accesses the Select Color Index palette, from which you can select an appropriate color for your background. The Background Raster button opens the Select File dialog box, from which you can choose a background raster image file. The image you select becomes the background for the view. The window size will change to accommodate the size of the raster image you select. How to Manage Your Network with SPECTRUM Page 40 Document 1909 Figure 19: Change Background Dialog Box Change Background 220 Background Min Width = 1507 Min Height = 707 Height Width = 1544 Screen Width = 1258 757 Screen Height = 985 Background Raster OK Cancel Help To change the Background Color 1. While looking at a view in edit mode, choose Edit > Change Background… to open the Change Background dialog box (Figure 19). 2. Enter the desired window dimensions, in pixels, into the Height and Width fields of the Change Background dialog box. These entries define the maximum window size; you cannot resize the window beyond the selected size dimensions. 3. If you know the index number for the desired background color from the color palette, you can enter the index number directly into the Background Color field. Otherwise, click on the Background Color button to open the Select Color Index dialog box (Figure 20). SPECTRUM displays color index numbers 77 through 255 within the individual color blocks of the SPECTRUM Color Palette. (Only these colors are used by SPECTRUM.) 4. Select a color, then click OK. Click OK again in the Change Background dialog box to see the change take effect. How to Manage Your Network with SPECTRUM Page 41 Document 1909 Figure 20: Select Color Index Dialog Box 77 78 91 92 Select Color Index SPECTRUM Color Palette with Associated Index Numbers Click the left mouse button on the desired color and click OK to apply a color to the view. or Double-click on any color in the palette to select that color and close the palette dialog box. 247 Selected Color: OK Cancel To change the background image 1. While looking at a view in edit mode, choose Edit > Change Background… to open the Change Background dialog box. 2. Click the Background Raster button to open the Select File dialog box (Figure 21). Figure 21: Select File Dialog Box Select File Filter /space/Spectrum/SG-Support/CsImage/Background/* Directories /Spectrum/SG-Support/CsImage/Background/. /Spectrum/SG-Support/CsImage/Background/. /Spectrum/SG-Support/CsImage/Background/. Files Bd_Genbd1.csi Bd_Genbd2.csi Bd_Genbd3.csi Bk_DkBlue.csi Bk_Oat.csi Bk_LtOat.csi Bk_MdBlue.csi Default.csi Selection /space/Spectrum/SG-Support/CsImage/Background/ OK Filter Cancel How to Manage Your Network with SPECTRUM Page 42 Help Document 1909 3. Select the directory for the desired graphic file in the left-hand panel. Apply a filter to file selections if necessary. 4. Select an image file by clicking on the file name in the right-hand panel. 5. Click on OK to confirm your selection, or Cancel to leave the background unchanged. 6. Click on OK in the Change Background dialog box, or Cancel to leave the background unchanged. The Select File dialog box lists image files available in the CsImage/ Background directory. Some of these are solid color backgrounds and others are graphic images. The file select dialog box permits selecting image files written in Aprisma graphics file format (.csi) or Tagged Image File Format (TIFF). SPECTRUM supports the most commonly used (noncompressed and PackBits-compressed) TIFF file formats. Image files used as a background image must be placed in the <$SPECROOT>/SG-Support/ CsImage/Background directory. Note: The maximum size of the view is the size of the background raster image. The window view may not be enlarged to be larger than the size of the raster image selected. Using Organizational Views The Organizational view (aka Org-Chart/Services view) allows you to group subnets and device models based on organizational considerations such as: • Corporate structure; for example, the finance department network device models are grouped in their own organizational view. • Services; for example, a set of devices are essential for supporting the E-mail service which provides E-mail for several different departments. • Administrative responsibilities; for example, an operator has been assigned responsibility for all routers in your network. A variety of Org-Chart/Services views are available to help you manage your network based on your specific needs and network design. The first example below shows how to use organizational views to track the business impact of network problems by modeling how corporate services are linked to corporate structure. The second example shows how to use organizational views to track network problems by administrative responsibility. How to Manage Your Network with SPECTRUM Page 43 Document 1909 Creating a Network Model that Shows the Business Impact of Services on Administrative Structure The procedure described below is an example of how to create an OrgChart/Services view for the management of a service that one or more administrative groups depend on. 1. From the View menu of any Topology view, select New View > OrgChart/Services. 2. Select Edit from the File menu in the Org-Chart/Services view. 3. Select New Model from the Edit menu. 4. Select Department from the New Model dialog box and click OK. 5. Enter a unique name to best represent your Department model. In this example, the model name is Engineering because it will contain the devices and services that the Engineering department depends on. The Department model icon will appear in the Org-Chart/Services view. 6. Select Close Edit from the File menu. 7. Double-click on the view button on the Engineering Department model to navigate into the Engineering Department container. 8. Select Edit from the File menu. 9. Select New Model from the Edit menu. 10. Select Service_Owns from the New Model dialog box and click OK. 11. Enter a unique name to best describe the service you want to represent. In this example, the model name is E-mail because it will contain the devices that make up the E-mail service. The Service_Owns model icon will appear. How to Manage Your Network with SPECTRUM Page 44 Document 1909 12. Select Close Edit from the File menu. 13. Double-click on the view button on the E-mail Service_Owns model to navigate into the E-mail Service_Owns container. 14. Select Edit from the File menu. 15. You will need to place all device models that represent components of the E-mail service into this container. If these devices are already modeled in another view (or views), go to that view and use the Copy command to copy those models. Return to this view and paste the models into the container. If these devices have not yet been modeled, you can create models for them using the Edit > New Model command (see Adding and Removing Device Models [page 49] for further instructions). 16. Select Close Edit from the File menu. 17. You have now created a model to manage the E-mail service and have shown that the Engineering department depends on this service. Other Department container models can be created to represent additional departments. (Enterprise, Division, Landscape, or Subsidiary container model types can also be used.) The E-mail Service_Owns container can be copied and pasted into any of those Department containers that represent departments which depend on the E-mail service. The condition of a service is displayed on the Service_Owns container model. For example, if two routers contained in the E-mail Service_Owns container have gone down and a red alarm is generated on each of them, the E-mail Service_Owns container model will show a red condition. If you open the Alarm Manager from the Service_Owns container model, How to Manage Your Network with SPECTRUM Page 45 Document 1909 SPECTRUM shows all alarms for devices contained within the Service_Owns container. In addition, the rollup condition (see Rollup Condition Colors [page 102]) of the department is displayed on the Department container model. Thus, if network problems exist which affect services to a department, this will be reflected in the rollup condition shown on the Department container model. Once you have created a network model that represents your enterprise’s departments and services, the business impact of network problems can be easily determined. Network operators can correlate an alarm on a single device to the service(s) this device is critical to, and the departments that depend on that service. Creating a Network Model that Shows Administrative Responsibilities The procedure described below is an example of how to create an OrgChart/Services view for the management of routers that a given administrator is responsible for. 1. From the View menu of any Topology view, select New View > OrgChart/Services. 2. Select Edit from the File menu in the Org-Chart/Services view. 3. Select New Model from the Edit menu. 4. Select Org_Owns from the New Model dialog box and click OK. 5. Enter a unique name to best represent your Org_Owns model. In this example, the model name is ITClevelandRouters because it will contain all of the routers that the IT department in Cleveland is responsible for. The Org_Owns model icon will appear in the OrgChart/Services view. 6. Select Close Edit from the File menu. 7. Double-click on the Org_Owns view button to open the view. How to Manage Your Network with SPECTRUM Page 46 Document 1909 SpectroGRAPH: Org-Chart/Services: TopOrg File View Tools Bookmarks Help ITClevelandRouters Org_Owns 0 8. Select Edit from the File menu in the Org_Owns view. 9. Copy all routers that the IT department in Cleveland is responsible for into this view as follows: a. Open the applicable Topology view and select Edit from the File menu. b. Highlight all routers in the Topology view to be copied into the Org_Owns view. c. Select Copy from the Edit menu in the applicable Topology view. Click OK in the Confirm dialog box. d. Select Paste from the File menu in the Org_Owns view. All routers copied will appear in the view. e. You can arrange routers in the view as shown below. How to Manage Your Network with SPECTRUM Page 47 Document 1909 SpectroGRAPH: OWN: Routers File View Router 1 CiscoRtr Building 35 f. Tools Bookmarks Router 2 Help Router 3 CiscoRtr CiscoRtr Building 37 Building 38 Router 4 CiscoRtr Manufacturing Select Close Edit from the Topology views and the Org_Owns view to return to view mode. 10. Using the Annotation Tool Box, label each router as to its location or significance. This will allow identification of the router for manageability. Maintaining Your Network Model Once you have completed optimizing your network model, you will need to make sure it accurately reflects the current configuration as new devices are added or existing ones taken out of service. You will also want to be able to create a backup copy of your database so that you can restore it to a known previous condition in the event of data corruption or equipment failure. These issues are discussed in the following two sections: Adding and Removing Device Models [page 49] Saving and Restoring the Database [page 50] How to Manage Your Network with SPECTRUM Page 48 Document 1909 Adding and Removing Device Models Whenever you add or remove a device on your network, you will need to alter your network model to keep it up-to-date. You must be sure to destroy the model of the device being removed or the model will be put into Lost and Found. To add a device model to your network model: 1. Navigate to the highest level Topology view that should contain this device model. 2. Select File > Edit to put the view in the edit mode. 3. Select Edit > New Model (if you know the SPECTRUM model type you want to use) or New Model By IP (if you know the device’s IP address but want SPECTRUM to automatically select the appropriate model type). 4. If you are using the New Model option, in the Select Model Type dialog box, select a model type and click OK; then enter a name and address in the Creation View dialog box. If you are using the New Model By IP option, in the Create Model By IP Address dialog box enter the device’s IP address in the Network Address text box (see Note below). 5. When you use either the New Model or the New Model By IP option, you can have SPECTRUM automatically discover and model network elements that are connected to the device you are modeling. To do this, click the Discover Connections check box available in the Creation view dialog box when you are using the New Model option, and in the Create Model By IP Address dialog box when you are using the New Model By IP option. SPECTRUM will use the settings in the AutoDiscovery Model Information view for discovering the connections. 6. If you are using the New Model by IP option, you can choose to model the device using the SNMPv2c or SNMPv3 protocol. To choose SNMPv2c, click on the SNMP V2C Enabled button. To choose SNMPv3, click on the the SNMP V3 Enabled button, and fill in the options appropriately. For more information on the SNMPv3 options, see the SPECTRUM SNMPv3 User guide (5124). 7. Click OK. 8. Select Close Edit from the File menu. How to Manage Your Network with SPECTRUM Page 49 Document 1909 Note: You cannot use the New Model by IP option to create a model if a model with that IP address already exists in SPECTRUM. Attempting to do so will display an error message that identifies the existing model by name and model type and asks you whether you would like to open its default view (usually its Device Topology view). However, if you know a modeled device’s IP address, this same functionality can be used as a shortcut for locating the model within the network map without having to open Search Manager. If you click OK in the error message, the model’s default view will be displayed, and you can then navigate from that view as necessary to determine the model’s location. To remove a device model from your network model: 1. Navigate to the highest level Topology view that contains the device model. 2. Select File > Edit to put the view in the edit mode. 3. Select the model to be removed by clicking on it to highlight it. 4. Select Edit > Destroy. 5. Select File > Close Edit to exit the edit mode and save your changes. Saving and Restoring the Database Once you have organized your SpectroGRAPH network model the way you want, you should save your database and schedule a regular database backup using the OnLine Backup feature, which is accessed from the SPECTRUM Control Panel. A backup database will enable you to quickly return to your optimized network model if something happens to corrupt your database. This can save hours of work and should not be overlooked. Creating a Backup To backup your database from the Control Panel: 1. In the Control Panel’s Database Administration block, click the Save button. If SpectroGRAPH is not running, a dialog box will ask you what host you want to save the database against. Select the default How to Manage Your Network with SPECTRUM Page 50 Document 1909 host to save the current database. The Online Database Backup Configuration View opens. 2. Enable compression of files to save space. 3. Assign a prefix for the backup file name if desired. The default is db_. 4. Assign a backup directory if one other than the default is desired. The default is <$SPECROOT>/SS-DB-Backup. 5. The minimum disk space required is shown. Check to make sure you have enough space to save the file. 6. Click the Begin Backup Now! button. Restoring a Database To restore your database from the Control Panel: 1. Before restoring a database, it is always a good idea to backup the current database (unless this is a corrupt file). Refer to the instructions above on how to backup a database. 2. Click on the Restore button. A dialog box will ask if you want to initialize the database. If you are recovering from a corrupt file, you should choose the initialize option to add a further cleaning with the restoration of the database. If you do not choose to initialize, it will simply reload the backup database file. In either event, a dialog box will prompt you for the directory and name of the database file to load. 3. Click on the directory and file name of the database file you wish to load. 4. Click OK. Scheduling Backups To Schedule Regular Database Backups From the Control Panel: 1. Click on the Save button. The Online Database Backup Configuration View opens. 2. In the bottom half of the view, there is a section on Automatic Backup Setup. Click on the Enable button to enable automatic backups. 3. Enter the backup interval in hours and minutes. 4. Check the next backup date and time to make sure it is correct. How to Manage Your Network with SPECTRUM Page 51 Document 1909 5. Check the top section of the view for the backup directory name and whether compression is selected. Set these variables as desired. (refer to the section on how to backup a database). 6. Click the Begin Backup Now! button. How to Manage Your Network with SPECTRUM Page 52 Document 1909 Configuring for Fault Management SPECTRUM’s proactive fault management features provide the capability to set thresholds and configure traps, events, and alarms. These features work to alert you before an actual fault occurs within your network. This section describes these proactive features. In This Section Setting Thresholds [page 53] Configuring Traps [page 55] Configuring Alarms [page 56] Using SpectroWATCH [page 60] Configuring Management Neighbors [page 64] Configuring Port Status Monitoring [page 80] Suggested Port Fault Settings for Optimal Fault Notification [page 94] Other Fault Management View Settings [page 94] Configuring Fault Management for Pingables [page 95] Setting Thresholds SPECTRUM icons use color to indicate two types of status for the modeled entities they represent. All models have a condition color that reflects the current contact and/or alarm status of the model itself. Additionally, “container” models, such as Networks and LANs, have a rollup condition color that reflects the composite status of all the other models they contain (i.e., their “children”). In other words, the status of a modeled device “rolls up” to the “parent” container model. Both types of status rely on threshold values to determine when and how the associated color will change. Thresholds for condition and rollup condition are discussed separately in the following two sections. How to Manage Your Network with SPECTRUM Page 53 Document 1909 Alarm Thresholds Alarm thresholds apply to individual devices. Based on your experience with your network, you may want to set threshold critical values within some or all devices that will generate an alarm condition if these values are exceeded. For example, the alarm thresholds that can be set for Cabletron’s EMM-E6 are defined as follows: • Traffic - You can set a threshold value for the load per unit of time for the repeaters and ports managed by the EMM-E6. • Collisions- You can set a threshold value for the number of collisions per unit of time for the repeaters and ports managed by the EMM-E6. • Broadcasts- You can set a threshold value for the number of broadcasts received within a period of time by the EMM-E6. • Errors- You can set a threshold value for the number of errors per unit of time for the repeaters and ports managed by the EMM-E6. Alarm thresholds are generally set through one of the Configuration views available for the modeled device. For specific information, consult the documentation for the SPECTRUM management module used to model the device for which you wish to set alarm thresholds. Rollup Condition Thresholds and Significance Levels Rollup condition thresholds are set to display a rollup condition color based on the sum of the significance levels for the alarms generated by devices within the models that “Contain” them (Figure 2). If two devices have a combined significance level of 7, and the rollup condition threshold for the device that contains them is 3, then the containing model will be orange. Set these thresholds and significance levels in the Model Information view for each model in your network. The Significance Level attributes define the numeric value for Yellow, Orange, and Red Conditions and Rollup Conditions. Like Rollup Thresholds, Significance Level values are administrator-defined values entered on a model-by-model basis. Significance Level field labels begin with the words “value when.” The default values for Significance Level are as follows: Value_When_Yellow=1 Value_When_Orange=3 Value_When_Red=7 Typically, models (devices) are divided into two classes, “significant” or “insignificant.” A significant device is any device that requires an administrator’s attention for proper network operation. Insignificant How to Manage Your Network with SPECTRUM Page 54 Document 1909 devices are typically endpoint devices, such as a PC or workstation. Insignificant devices usually toggle between Green and Blue (Condition Value = 0). Significant models can be made insignificant by changing their Value_When_Red attribute value to 0 (zero). Many users do not change the rollup thresholds from their default values because of the complexity of options available. Figure 22: Rollup Conditions Information Alarm Model Name Rollup Condition Color With a yellow rollup threshold 4 is Yellow set to 1 and the sum of the significance 1 levels for the “Contained” models equal to 7 Landscape 3, the rollup condition color will be yellow. Minor Alarm Rollup Condition Color is Orange With an orange rollup threshold set to 3 and the sum of the significance levels for the “Contained” devices equal to LAN 802.3 7, the rollup condition color will be orange. Device Failure Device Condition Color is Red A noncritical device with a significance level of 7. HubCSIEMME Configuring Traps Traps are used to generate rollup conditions, alarms, and event messages. Some traps are automatically set by SPECTRUM. See the Event Configuration Files User Guide (5070) for information on defining how SNMP traps (unsolicited SNMP messages) are handled. You can specify how trap codes and their associated parameters are mapped into SPECTRUM events and alarms. SNMP traps are received from specific devices that support the Simple Network Management Protocol. Therefore all operations associated with How to Manage Your Network with SPECTRUM Page 55 Document 1909 editing SNMP trap information begin by selecting a specific kind of device (model type). Traps can be mapped to SPECTRUM events. An event is an indication that something has occurred on your network and SPECTRUM has taken notice of it. Events can be configured to generate alarms and can be mapped to Alarm Probable Cause messages, which provide recommended actions regarding the alarm. Note: You can disable the generation of events and alarms based on traps received from a specific device by adjusting a setting in the Fault Management view for that device model (see Figure 24: The Fault Management View [page 65]). Creating New Traps If a trap for a particular model type does not exist, you can create and map a new one. See the Event Configuration Files User Guide (5070) for more information. Checking Devices for Trap Configuration The attribute DeviceTrapsReceived (0x00012a80) tracks the total number of traps received from a device since the time the SpectroSERVER started. You can use the attribute in Search Manager to periodically check for devices which may not be properly configured to send traps. Configuring Alarms Alarms are generated by SPECTRUM to let you know when defined thresholds have been exceeded. Each device reports to SPECTRUM when an event occurs. Some events generate alarms automatically, and with some devices you can configure alarms by setting threshold values for traffic, collisions, broadcasts, and errors. Alarms that are generated automatically by SPECTRUM are listed in SPECTRUM’s SS/CsVendor directory for each device management module. Probable cause files for these alarms are listed in the SG-Support/CsPCause directory. The following sections describe how alarms can be managed to meet your specific needs. Clearing Old Alarms Automatically You can configure SPECTRUM to automatically clear any alarm which has existed for longer than a specified amount of time. It is also possible to How to Manage Your Network with SPECTRUM Page 56 Document 1909 configure SPECTRUM to clear all old alarms, or just old stale alarms. By automatically clearing the unwanted alarms, you will improve the performance of the SpectroSERVER. The VNM’s Alarm Management Information view contains two attributes that allow you to control the automatic clearing of old alarms. The AlarmAgeOutTime attribute contains the amount of time (in hours) that an alarm in the landscape can exist. Once an alarm has existed for longer than this amount of time, it will be removed. If the attribute contains a value of 0 (zero), this functionality will be disabled. The AgeOutStaleAlarmsOnly attribute contains a Boolean value that indicates whether only stale old alarms should be cleared. By default, it is set to TRUE. Every hour, SPECTRUM checks the status of all alarms in the landscape and clears alarms based on the settings of the above criteria. Therefore an alarm will not be removed at exactly the moment when its existence time has exceeded the time-out. It can be, at most, an hour “overdue.” Refer to the Alarm Management section of the Distributed SpectroSERVER (2770) guide for information on the VNM’s Alarm Management Information view and how to change the value of attributes in this view. Alarm Policies for Specific Interfaces It is possible to prevent SPECTRUM from generating alarms on specific types of interfaces. These “specific interfaces” are defined by the following criteria: IfType = ds0 (81) IfType = ppp (23) AND (ifDescr = “Serial*:*” OR ifDescr = “Async*” OR ifDescr = “Virtual-Access*”) The AlarmOnSpecificInterfaces attribute is used to control whether alarms are generated on the specific interfaces defined by the criteria above. By default, it is set to TRUE. It can be set to FALSE using SPECTRUM’s CLI, using the Global Attribute Editor, or by adding the attribute to a GIB and changing the value. When a device’s AlarmOnSpecificInterfaces attribute is set to FALSE, SPECTRUM will find all of the interfaces that meet the above criteria on the device and set the AlarmOnLinkDownTrap (0x11fc2) attribute on each interface to Never. This prevents SPECTRUM from generating alarms based on Link Down Traps on these models. Each of these interfaces will also have their PollPortStatus (0x1280a) attribute set to FALSE, preventing How to Manage Your Network with SPECTRUM Page 57 Document 1909 port status polling and alarming. In addition, each of the interfaces will have their DisableTrapEvents (0x11cd0) attribute set to TRUE, which prevents the generation of events (and alarms) for traps that are received for that interface. When AlarmOnSpecificInterfaces is set to TRUE, SPECTRUM will use the AlarmOnLinkDownTrapIfTypes (0x1290f) attribute on each interface to determine how to set AlarmOnLinkDownTrap. Note: If you change the value of AlarmOnSpecificInterfaces from FALSE to TRUE, the value of the AlarmOnLinkDownTrap, PollPortStatus, and DisableTrapEvents attributes will not be changed. These values must be changed manually. Alarm Management on DLCI Ports The AlarmOnInvalidDLCIs attribute exists for all device models and is used to control whether or not alarms are generated on invalid DLCI ports. The default value for this attribute is FALSE. It can be set to TRUE using SPECTRUM’s CLI, or by adding the attribute to a GIB and changing the value. When AlarmOnInvalidDLCIs is set to FALSE, all DLCI_Port models on the device whose circuit state is currently “invalid” will have their PollPortSatus attribute to FALSE. When AlarmOnInvalidDLCIs is set to TRUE, the DLCI_Ports will be left alone. The following bullets outline how SPECTRUM handles alarming on invalid DLCIs: • If the physical interface (i.e. lower layer interface) is down, an invalid DLCI will have a gray condition. • If the physical interface is up, and AlarmOnInvalidDLCIs is set to FALSE, a brown alarm will be generated on the invalid DLCI. • If AlarmOnInvalidDLCIs is set to TRUE, a red alarm will be generated on the invalid DLCI. The following caveats apply to this alarm management functionality: • The definition of a “specific interface” is not extendable or modifiable. Only interfaces which exactly match the above criteria will be considered “specific interfaces”. How to Manage Your Network with SPECTRUM Page 58 Document 1909 • If a pipe associated with a “specific interface” is made live, alarms will be generated. • If the PollPortStatus of a “specific interface” or DLCI had been set to TRUE, and was subsequently set to FALSE because alarm suppression was enabled for the device, it will not be reset back to TRUE when alarm suppression is disabled. It will remain FALSE. This is because it is not possible to determine the original value after a VNM restart. Manual intervention is required. • If a DLCI port receives a circuitStateChange trap and the new state is Invalid, an alarm will always be generated. There is no way to control this. • If SPECTRUM had previously set the above attributes correctly for either a “specific interface” or a DLCI, and an attribute’s value is subsequently modified, SPECTRUM will not be responsible for any associated behavior. False Management Lost or Contact Lost Alarms There is a SUN recommended security procedure which can increase the chance of a false Management Lost or Contact Lost alarm being generated. This security procedure modifies the ARP timeout on a Solaris host. This change in the ARP configuration will cause an increased delay in ARP messages being sent by the system, which will increase the overall time it takes for a SNMP packet to be sent and replied to. Since the SpectroSERVER starts the timeout timer as soon as it hands the SNMP request off to the Operating System, the overall delay easily exceeds the settings in the SpectroSERVER. One way to identify this behavior is to invoke the following command from a terminal window on the Solaris host: arp -a | wc -l This command will count the number of entries in the ARP table. The count will increase and then it will suddenly decrease back to the initial value. The time it takes for the decrease to occur will depend on the current ARP timeout setting. To work around this problem, you must return the ARP timeout value to its default. To do this, follow the instructions below: 1. Open the following file: /etc/init.d/nddconfig How to Manage Your Network with SPECTRUM Page 59 Document 1909 2. Look for an entry similar to the following: ndd -set /dev/ip ip_ire_arp_interval 600000 3. This entry modifies the default ARP timeout value. Remove this entry. 4. Type the following command in order to update your system (you must run as root): Solaris 2.8 ndd -set /dev/ip ip_ire_arp_interval 1200000 Solaris version below 2.8 ndd -set /dev/ip ip_ire_flush_interval 1200000 5. Reboot your system to allow these changes to take effect. The ARP timeout will return to the default value, which works correctly with SPECTRUM. Using SpectroWATCH You can use SpectroWATCH to monitor any attributes against thresholds you set to generate events and alarms. SpectroWATCH allows you to do the following: • Select the attribute(s) that are important to monitor for fault potential • Set the threshold value for that attribute(s) so that when the threshold is exceeded, an alarm is generated • Select the alarm severity for each threshold • Write the alarm message that will show up in the alarm view or alarm script that will be executed when the alarm is generated • Monitor either a single attribute or a complex formula of attributes, constant values, mathematical functions, and other existing formulas. For example: two threshold watches could be set on a device’s packet rate: one to generate a yellow alarm when the value exceeds 10,000 (the value determined to be close to critical), and another to generate a red alarm when the value exceeds 15,000 (the value determined to be critical.) For more information, refer to the guide SpectroWATCH Operator’s Reference (0919). How to Manage Your Network with SPECTRUM Page 60 Document 1909 Configuring for Fault Isolation The following sections cover SPECTRUM’s fault isolation configuration options: • Configuring General Fault Isolation Parameters [page 61] • Configuring Management Neighbors [page 64] • Configuring Maintenance Mode [page 68] • Configuring Cross-Landscape Fault Correlation [page 70] • Configuring Port Fault Correlation [page 72] • Configuring Port Status Monitoring [page 80] • PollPortStatus Feature [page 86] • Wide Area Link Monitoring [page 87] • Port Layer Alarm Suppression [page 90] • Port Criticality [page 90] • Live Pipes [page 91] • Automatic Port Status Alarm Clearing [page 93] • Suggested Port Fault Settings for Optimal Fault Notification [page 94] Configuring General Fault Isolation Parameters The Fault Isolation model is a model in SPECTRUM’s database which stores information about various aspects of SPECTRUM’s fault isolation functionality. The Fault Isolation Model Information view allows you to configure these settings. To access the Fault Isolation Model Information view: 1. Highlight the VNM icon and select Configuration from the Icon Subviews menu. This brings up the Landscape Configuration view. 2. In the Configure/Information panel, double-click the Fault Isolation entry, select it and click OK. The Fault Isolation Model Information view (Figure 23) is displayed. How to Manage Your Network with SPECTRUM Page 61 Document 1909 Figure 23: The Fault Isolation Model Information View The following fields can be configured: ICMP Support Enabled This field controls the value of the Fault Isolation model’s ICMP_SUPPORT attribute. This attribute is used during fault isolation to determine if an attempt should be made to contact a device using the ICMP protocol when trying to ascertain the fault status of the device. When the ICMP_SUPPORT attribute is enabled (set to TRUE) on the Fault Isolation model, SPECTRUM will look at the setting of the ICMP_SUPPORT attribute at the device level. If ICMP support is also enabled on the device, SPECTRUM will attempt to contact the device using the ICMP protocol. However, when ICMP_SUPPORT is disabled (set to FALSE) on the Fault Isolation model, this setting will take precedence over the setting at the device level and will prevent any attempt to contact the device using the ICMP protocol for fault isolation. The default setting for this field is TRUE. How to Manage Your Network with SPECTRUM Page 62 Document 1909 ICMP Timeout The amount of time (in milliseconds) that SPECTRUM will wait for a response to an ICMP ping. If a response is not received within this period of time, SPECTRUM assumes the device has timed out. The default setting for this item is 3000ms (3 seconds). ICMP Trycount This field specifies the total number of attempts to contact a device using the ICMP protocol that will be made before SPECTRUM determines the device is down. Lost Device Trycount The number of retries that SPECTRUM attempts for each SNMP request sent to a device after contact with the device is lost. The default setting for this field is 1. Contact Lost Model Destruction This field controls whether a device is automatically destroyed when its CONTACT_STATUS is set to false. When this field is set to Enabled, models that have had their CONTACT_STATUS set to lost for a specified period of time (see the Destruction Time Delay field) will be destroyed automatically. When this item is set to Disabled, models will never be automatically destroyed as a result of the value of the CONTACT_STATUS attribute. The default setting for this field is Disabled. Destruction Delay Time This field specifies the length of time (in seconds) that a model must continuously have its CONTACT_STATUS attribute set to lost before it is automatically destroyed. The default value is 604800 seconds (7 days). Destruction Event Generation This field controls whether an event message is created when a model is automatically destroyed. When this field is set to Enabled, an event will be generated each time a model is destroyed as a result of its CONTACT_STATUS being continuously set to lost for the length of time specified in the Destruction Delay Time field. When this field is set to Disabled, no event message will be generated. The default setting for this field is Enabled. Router Redundancy Retry Count This field enables you to set the number of times SPECTRUM will attempt to contact a router’s redundant IP addresses if contact is lost with its primary address (see Redundancy [page 152]). The polling interval setting How to Manage Your Network with SPECTRUM Page 63 Document 1909 determines the amount of time between each attempt. The default setting for this field is 2. Port Fault Correlation This field allows you to configure the port fault correlation feature. See Port Fault Correlation Options [page 72] for an explanation of each of the configuration options. Unresolved Fault Alarm Disposition If SPECTRUM’s information about the connectivity of your network model is incomplete, SPECTRUM may be unable to find the root cause of a network outage. In this case, the status of all devices affected by the outage is set to gray, and a red unresolved fault alarm is generated. This alarm indicates that SPECTRUM has lost contact with a group of devices, but was unable to pinpoint the cause. All of the devices to which SPECTRUM has lost contact are listed in the impact scope of the alarm. The model name and other details of the lost devices also appear in the event that generated the alarm. The Unresolved Fault Alarm Disposition field allows you to control how the unresolved fault alarm is generated. When set to Fault Isolation Model, the alarm will be generated on the Fault Isolation model. When set to Device In Fault Domain, the alarm will be generated on one of the devices with which SPECTRUM has lost contact. When determining which device to generate the alarm on, SPECTRUM looks for the device with the highest criticality. If the highest criticality is shared by two or more devices, SPECTRUM generates the alarm on the first of these devices that it finds. If all devices have the same criticality, then SPECTRUM chooses the device with the lowest model handle. Configuring Management Neighbors SPECTRUM lets you customize its fault isolation algorithm at the interface level to suppress a red alarm at the device level. This is desirable in cases where a given device still supports data traffic, but management traffic has been interrupted due to a fault with the interface between the device and a neighboring device that lies between it and SPECTRUM. In this case, the default algorithm would generate two red alarms: one for the first device and one for the interface on the neighboring device. In other words, there would be multiple alarms for what was essentially a single fault. However, you can avoid this situation by designating one or more of a device’s connected interfaces as “management neighbors” (as opposed to “dataonly neighbors”). Then, when the interface with the management neighbor How to Manage Your Network with SPECTRUM Page 64 Document 1909 is down, SPECTRUM will still generate a red alarm for the management neighbor’s interface, but will only assert a condition of “Suppressed” (gray) for the first device. To designate any of a device’s connected interfaces as a management neighbor, do the following: 1. Highlight the device icon in a SPECTRUM Topology view. 2. From the device icon’s Icon Subviews menu, select the Fault Management option. This will display the Fault Management View shown in Figure 24. Figure 24: The Fault Management View 3. In the Fault Management View, click the Fault Isolation button. (The other buttons/fields are explained under Other Fault Management View Settings [page 94].) This will display the Device Fault Isolation Configuration Options view shown in Figure 25. Note that the sample scenario depicted in this view includes two management neighbors and thus two red alarms are asserted: one for B’s interface to A and one for C’s interface to A. Device A itself is in a “Suppressed” (gray) condition. How to Manage Your Network with SPECTRUM Page 65 Document 1909 Figure 25: Device Fault Isolation Configuration Options View 4. At the bottom of the Device Fault Isolation Configuration Options view, click the Specify Management Neighbors button to display the Select Management Neighbor Interfaces dialog box shown in Figure 26. How to Manage Your Network with SPECTRUM Page 66 Document 1909 Figure 26: The Select Management Neighbor Interfaces Dialog Box 182.12.13.2_2 2 Good 0x42a003d4 5. In the Select Management Neighbor Interfaces dialog box, designate one or more management neighbors by moving the entry (which comprises the interface’s model name, OID, status, and model handle) from the Connected Interfaces list to the Management Neighbor Interfaces list. You can move an entry from one list to the other either by double-clicking it or by selecting it and then clicking the single arrow button that points to the desired list. Clicking a double arrow moves all entries from one list to the other. How to Manage Your Network with SPECTRUM Page 67 Document 1909 Note: If the selected device has a large number of connected interfaces, you can view only a specific subset of them or find a particular one by using the Filter/Search button at the bottom of either list. When Filter is selected, the list will show only those entries that contain the string you enter in the adjacent text entry box. When Search is selected, the first entry containing the entered string will be highlighted. Clicking the Next button will highlight the next entry that contains the string. 6. Click OK to put your management neighbor designations into effect and close the dialog box, or click Cancel to revert to the previous designations and close the dialog box. Configuring Maintenance Mode Maintenance Mode for Devices In a device’s Fault Management View, setting the Enable SPECTRUM Management button to FALSE places the device model into maintenance mode, suspending management traffic to the device and its components and preventing the generation of any events or alarms on their behalf. When a device has been placed into maintenance mode, its icon displays a device condition color of brown. You can use the Modeling Gateway Toolkit and the Schedule element to create a maintenance mode schedule for a particular device model. See the Modeling Gateway Toolkit Guide (5069) for more information. When a model is in maintenance mode, no events will be processed for that model. That includes events that would normally clear an alarm on the model, as well as events that would create an alarm. For example, a link_down event generated an alarm on the model prior to the model being placed in maintenance mode. If a link_up trap event is generated while the model is in maintenance mode, the alarm is not cleared as the link_up event is not processed. You can configure SPECTRUM to either hide or show secondary alarms when a device is in Maintenance mode. This is done using the parameter “Show Secondary Alarms When Device is in Maintenance” available in the Alarm Update Control tab of the Alarm Manager Preferences. If this parameter is enabled, secondary alarms will be shown when a device is in maintenance mode. If disabled (the default), secondary alarms will be hidden when a device is in maintenance mode and will be shown again How to Manage Your Network with SPECTRUM Page 68 Document 1909 when the device is taken out of maintenance mode. This setting will only apply if primary and secondary alarms are enabled in the Alarm Manager filter settings. See the Enterprise Alarm Manager User Guide (2065) for more information on the Alarm Manager. Putting a device model into maintenance mode will, by default, also put its interface models and application models into maintenance mode. Brown alarms will be generated on device models in maintenance mode but will not be generated on application and interface models that have inherited maintenance mode from devices. Optionally, you can use the SPECTRUM Command Line Interface (CLI) to change this behavior: 1. To generate brown alarms on interface models that have inherited maintenance mode, set the device model attribute 0x00012a7a (rollDownAlarmToIF) to TRUE. 2. To generate brown alarms on application models that have inherited maintenance mode, set the device model attribute 0x00012a7b (rollDownAlarmToApp) to TRUE. See the Command Line Interface (0664) guide for more information about CLI. Interface models can be put into maintenance mode independently of device models, as described in Maintenance Mode for Ports [page 69]. Maintenance Mode for Ports In the Port Fault Management View, setting the Enable SPECTRUM Management button to FALSE places the port model into maintenance mode, suspending management of the port, while still performing regular management on the rest of the device. When in maintenance mode the following items will be true for the port model: • The condition of the port model will be brown. • Alarms will not be created for the port. • Events will not be logged for the port. • No polling, logging, or other device communication will be done on behalf of the port model. Other device communications will not be affected. • Link down traps sent are ignored. An event is created on the device model indicating that the trap was received, but no alarm is generated. How to Manage Your Network with SPECTRUM Page 69 Document 1909 • If Live Pipes is enabled for a connection and one of the end points is a port in maintenance mode, the pipe on the Topology view will appear brown. Status polling for the port will stop. • If a connection is modeled with a WA_Link model connection to two ports, and one of those ports is in Maintenance Mode, a maintenance (brown) alarm will be created on the WA_Link and WA_Segment models. The WA_Link icon will be brown. If Live Pipes is enabled on this link, the pipe will remain green as long as the other port is up. If the other port goes down or is unreachable, the pipe will turn gray. If SPECTRUM loses contact with a device model connected to the port in maintenance mode, the Device Has Stopped Responding to Polls alarm will be suppressed for that device as well as all adjacent devices. If device contact lost alarms are suppressed because of their position relative to a port in maintenance mode, the maintenance mode alarm will reflect these lost devices in its Impact Severity and Impact Scope attributes. These lost devices will be viewable in the Impact Scope View for that maintenance alarm. For more information on Impact Severity and the Impact Scope view, refer to the Enterprise Alarm Manager User Guide (2065). Configuring Cross-Landscape Fault Correlation In a Distributed SpectroSERVER (DSS) environment, a network administrator may need to model a router from a local landscape in a remote landscape in order for its connections to participate in fault isolation for the remote landscape. This proxy model doesn't need to participate in alarm generation in the remote landscape because it already does so in the local landscape, where it is modeled “normally”. It is in the local landscape that alarms for the router are tracked and trouble tickets are created. (See the Distributed SpectroSERVER Guide (2770) for more information on distributed network management.) In such a scenario, Cross-Landscape Fault Correlation can prevent multiple red alarms for the same outage. With the Enable Event Creation attribute set to FALSE in the proxy model’s Fault Management View, SPECTRUM suspends the creation of events for the model (and any component models such as boards or ports). This effectively disables alarms for the model, but unlike maintenance mode (see Configuring Maintenance Mode [page 68]), SNMP communication with the proxy model continues, and so too does its participation in fault isolation. How to Manage Your Network with SPECTRUM Page 70 Document 1909 Cross-Landscape Fault Correlation Example Figure 27: Network with Multiple Landscapes Landscape A (the local landscape) contains core routers, including Router R. Landscape B (the remote landscape) contains customer routers with the core Router R modeled a second time, this time as a proxy (Figure 28: Router R in Landscapes A and B [page 71]). This proxy model has its IsEventCreationEnabled attribute set to FALSE (the default setting is TRUE). The device is being polled, but will not generate events or alarms. Figure 28: Router R in Landscapes A and B If Router R goes down (Figure 29: Alarm with Cross-Landscape Fault Correlation [page 72]), Landscape B loses contact with the proxy and customer routers. However, only one red alarm is generated, in Landscape A. The proxy router stays green in Landscape B, where the alarms are suppressed because the proxy’s IsEventCreationEnabled attribute is set to FALSE. How to Manage Your Network with SPECTRUM Page 71 Document 1909 Figure 29: Alarm with Cross-Landscape Fault Correlation Configuring Port Fault Correlation SPECTRUM lets you customize its fault isolation algorithm to resolve the root cause of a network outage to the port level. This is most desirable in cases where a single physical port, such as a Frame Relay interface, supports multiple logical connections to remote devices. If the physical port goes down, SPECTRUM can suppress the alarms on all downstream devices in favor of a single red alarm on the physical interface, thus significantly reducing the number of alarms which need attention. The impact severity and scope of the red alarm on the physical interface will contain all downstream devices, as well as the physical interface. Port Fault Correlation Options Port Fault Correlation is configurable via the Port Fault Correlation setting in the Fault Isolation Configuration View. To access this view, right click on the VNM model icon and select the Configuration option from the VNM’s Icon Subviews menu. In the Landscape Configuration view’s Configure/ Information Panel, double-click the Fault Isolation entry. The Port Fault Correlation setting has the following possible values: Disabled, All Connected Ports, Management Neighbors Only, All Connected Ports - Multiple Devices Only: • If set to Disabled, port fault correlation will not run. The root cause of a network outage will remain a red alarm on a device model. Fault Isolation will, however, still examine all of the device’s connected ports to see if they are all in Maintenance Mode. If so, the alarm on the device model will be suppressed. • If set to All Connected Ports, port fault correlation will run, examining all ports that exist on "up" neighbors which are connected to the down device as possible root causes of the outage. No additional manual configuration is required. How to Manage Your Network with SPECTRUM Page 72 Document 1909 • If set to Management Neighbors Only, the port fault correlation algorithm will run, but will only examine ports which were previously configured manually as management neighbors as possible root causes of the outage (see Configuring Management Neighbors [page 64]). • If set to All Connected Ports - Multiple Devices Only, port fault correlation will run, examining all ports that exist on “up” neighbors which are connected to the down device as possible root causes of the outage. However, SPECTRUM will only resolve the outage to the port level if there is more than one device model that would have a red alarm which can be correlated to the port alarm. If only one connected device alarm can be correlated to the port alarm, SPECTRUM will not suppress the device alarm. Instead, both the port and device alarm will be generated. Port Fault Correlation Criteria The following criteria must be met in order for the root cause of an outage to be resolved to the port level: • The down device must have one, and only one, “up” neighbor. If the down device has more than one “up” neighbor, port fault correlation will not be performed. This is done to reduce the number of alarms for a single problem. If multiple up neighbors were a valid criteria, and all connected ports were down, multiple red alarms would exist, all with the same impact severity and scope. If a device has more than one up neighbor, SPECTRUM assumes the problem lies with the device, not the upstream ports and creates a single red alarm on the device. • The down device must have at least one connected port (or management neighbor port) on an “up” neighbor that is down. • If multiple ports on the “up” neighbor connect to the down device (e.g. link aggregation), all of the ports must be down. • A port is considered “down” if it is either operationally down, or the port model has been put into Maintenance Mode. • There must be an alarm on at least one of the down ports. (Otherwise, there would be no alarm to which SPECTRUM could resolve the outage.) • If Port Fault Correlation is set to Management Neighbors Only, management neighbors for the down device must have been configured before the outage occurred (see Port Fault Correlation Options [page 72]). How to Manage Your Network with SPECTRUM Page 73 Document 1909 Port Fault Correlation Caveats The Port Fault Correlation feature overrides the Suppress Linked Port Alarms setting in the Live Pipes Configuration View (see Live Pipes [page 91]). When set to TRUE, this setting suppresses the alarm on an upstream port if it's connected to an unreachable device. If Port Fault Correlation is enabled, and the upstream port is the root cause of an outage, SPECTRUM will force the upstream port to be alarmed. Only the Criticality of the alarmed port will be used in the impact severity/ scope calculation of the root cause alarm. The Criticality of any subinterfaces (such as DLCI ports) will not be included. Port Fault Correlation is supported by Device models only. Models such as Fanouts and Unplaced do not support this feature. WA_Link models have their own mechanism for supporting port fault correlation, Link Fault Disposition, which is explained in Wide Area Link Monitoring [page 87] in this document. If multiple ports on the “up” neighbor connect to the down device (e.g. link aggregation), and all of the ports are down, multiple red alarms will exist as the root causes of the outage. Each red alarm will contain the same impact severity and scope. The root cause of the outage in this case is, in fact, all of the ports, not just one of them. Port Fault Correlation Examples The following fault scenarios describe the benefits of Port Fault Correlation. Fault Scenario 1 Figure 30: Fault Scenario 1 How to Manage Your Network with SPECTRUM Page 74 Document 1909 In the diagram in Figure 30, assume SPECTRUM must communicate through Router A in order to reach Routers 1 through N, and that this is the only means by which SPECTRUM can reach them. Each remote router is connected to Router 1 using a frame relay link. In SPECTRUM, this is modeled by connecting each DLCI port model to the other device. Assume the physical frame relay interface (FR A in Figure 30) goes down. This means that all virtual circuits provisioned on the interface will go down as well. With Port Fault Correlation disabled, the alarms shown in Figure 31 will occur. Figure 31: Fault Scenario 1: Alarms without Port Fault Correlation If a trap is received for FR A going down (or a live pipe is configured to be on), the physical frame relay interface will have a red alarm on it. In addition, all routers connected to the frame relay interface will have a red alarm on them. This could mean multiple red alarms could be generated by SPECTRUM for a single problem. Port Fault Correlation reduces the number of alarms generated for this problem down to a single alarm without requiring any manual configuration beforehand. Figure 32 shows the results with Port Fault Correlation enabled. How to Manage Your Network with SPECTRUM Page 75 Document 1909 Figure 32: Fault Scenario 1: Alarms with Port Fault Correlation A single red “Bad Link” alarm will be seen in the Alarm Manager. That alarm will have an Impact Scope and Severity which contains the following models: FR A, Routers 1 through N, and all unreachable devices that are downstream from Routers 1 through N. Fault Scenario 2 This fault scenario illustrates the benefits of setting the Suppress Linked Port Alarms and Port Fault Correlation attributes as recommended in Suggested Port Fault Settings for Optimal Fault Notification [page 94]. In the diagram in Figure 33, assume the VNM must communicate through Routers A and B in order to reach Routers C, D, and E, and that is the only means by which the VNM can reach them. In SPECTRUM, port-level connectivity is modeled as shown. Figure 33: Fault Scenario 2: Multiple “Up” Neighbors How to Manage Your Network with SPECTRUM Page 76 Document 1909 Assume Router C goes down. This will cause SPECTRUM to lose contact with Routers C, D, and E, and will make Ports A and B go down as well. If Suppress Linked Port Alarms is set to TRUE, and Port Fault Correlation is set to All Connected Ports, only a single red alarm on Router C will result (Figure 34). Figure 34: Scenario 2: Multiple “Up” Neighbors The upstream ports (Ports A and B) will have their alarms suppressed because Suppress Linked Port Alarms is TRUE. Even though Port Fault Correlation is enabled, Router C has multiple “up” neighbors, so the fault won't be resolved to the port level. When this occurs, SPECTRUM assumes the fault is with the device itself, not the connected ports. If you set Suppress Linked Port Alarms to FALSE, and Port Fault Correlation is still set to All Connected Ports, Router C and the upstream ports will be alarmed (if the status of the ports is being polled, or SPECTRUM receives a LinkDown trap), as shown in Figure 35. Figure 35: Fault Scenario 2: Multiple “Up” Neighbors How to Manage Your Network with SPECTRUM Page 77 Document 1909 Once again, the fault wasn't resolved to the port level because Router C has multiple “up” neighbors. Since Suppress Linked Port Alarms is disabled, SPECTRUM will alarm the upstream ports. If Router C had only one “up” neighbor, as in the diagram in Figure 36, even if Suppress Linked Port Alarms were set to TRUE (assuming Port Fault Correlation is still set to All Connected Ports), SPECTRUM will resolve the fault down to the port level. Port Fault Correlation forces the upstream port to be alarmed, and the alarm on Router C is suppressed (see Port Fault Correlation Criteria [page 73]). Figure 36: Fault Scenario 2: Single “Up” Neighbor Fault Scenario 3 This fault scenario demonstrates what happens when Port Fault Correlation is set to All Connected Ports - Multiple Devices Only. In the diagram in Figure 37, assume SPECTRUM must communicate through Router A in order to reach Routers 1 through N, and that this is the only means by which SPECTRUM can reach them. Each remote router is connected to Router 1 using a frame relay link. In SPECTRUM, this is modeled by connecting each DLCI port model to the other device. How to Manage Your Network with SPECTRUM Page 78 Document 1909 Figure 37: Fault Scenario 3 Assume the physical frame relay interface (FR A in Figure 37) goes down. This means that all virtual circuits provisioned on the interface will go down as well. With Port Fault Correlation disabled, the alarms shown in Figure 38 will occur. Figure 38: Fault Scenario 3: Alarms without Port Fault Correlation With Port Fault Correlation set to All Connected Ports – Multiple Devices Only, the alarms in Figure 39 will occur because multiple devices can be correlated to the frame relay interface. How to Manage Your Network with SPECTRUM Page 79 Document 1909 Figure 39: Fault Scenario 3: Port Fault Correlation Set To “All Connected Ports - Multiple Devices Only” Known Port Fault Correlation Anomalies There is one known anomaly associated with Port Fault Correlation. If a red alarm is generated on a port model as the root cause of an outage, you may then choose to put that port model into Maintenance Mode. If so, the red alarm would be replaced with a brown alarm. The brown alarm will still contain the same impact severity and scope (except the maintenance port will no longer contribute to the impact). If you then decide to take the port out of Maintenance Mode, the red alarm will reappear. It is possible, in this scenario, for the impact scope and severity of the red alarm to be lost. Configuring Port Status Monitoring SPECTRUM provides the following methods of monitoring the status of ports: • Link Traps (see Link Trap Handling [page 83]) Link traps allow the user to monitor the status of their ports without the cost of polling. However, traps are not always the most reliable notification mechanism of port status. • PollPortStatus (see PollPortStatus Feature [page 86]) The PollPortStatus feature allows users to poll the status of a port even if its connectivity is not modeled in SPECTRUM. • Live Pipes (see Live Pipes [page 91]) Live Pipes let you turn on port status monitoring for individual links. This is a more reliable monitoring method than traps because SPECTRUM will periodically poll the status of the link (with an increased How to Manage Your Network with SPECTRUM Page 80 Document 1909 cost in performance). In addition, Live Pipes let you graphically verify which links are being monitored. • WA_Link port monitoring (see Wide Area Link Monitoring [page 87]) WA_Link models automatically enable a live pipe for any connected ports. • SPECTRUM automatically maintains the port-level attribute NetworkLinkType based on the model class of the two connected devices. The attribute enables you to set up management policies based on the type of link a port is involved in. See the Policy Manager User Guide (5162) for more information on the NetworkLinkType attribute and management policies in general. The possible values for NetworkLinkType are: • 0 = Unknown Link • 1 = Router Link • 2 = Switch Link • 3 = Shared Access Link • 4 = End Station Link • 5 = Wide Area Link If the connectivity of the port is not modeled, NetworkLinkType will be set to Unknown Link. When the connectivity of the port is modeled, the value of NetworkLinkType is maintained as described in Table 1. Table 1: The NetworkLinkType Attribute in SPECTRUM Connection Attribute Value Link Type Router -> Router 1 Router Router -> Switch Router 1 Router Router -> Switch 1 Router Router -> Hub 1 Router Router -> Workstation Server 1 Router Switch Router -> Switch Router 2 Switch Switch Router -> Switch 2 Switch Switch Router -> Hub 3 Shared Access How to Manage Your Network with SPECTRUM Page 81 Document 1909 Connection Attribute Value Link Type Switch Router -> Workstation Server 4 End Station Switch -> Switch 2 Switch Switch -> Hub 3 Shared Access Switch -> Workstation Server 4 End Station Hub -> Hub 3 Shared Access Hub -> Workstation Server 4 End Station Any port connected to a WA_Link model 5 Wide Area Port Status Polling Criteria In general, in order for the status of a port to be polled, the following criteria must be met: • The Polling_Status (0x1154f) of the port model must be TRUE. • The Polling_Interval (0x10071) of the port model must be non-zero. • The Polling_Status of the port's device model must be TRUE. • Neither the port model nor the port's device model can be in Maintenance Mode (the isManaged (0x1295d) attribute for both must be set to TRUE). Note: If a port has been down since it was modeled in SPECTRUM and the Port Always Down Alarm Suppression attribute is set to Enabled (see Table 7: Port Status Monitoring Settings [page 93]), this port will only be polled once per hour (every 3600 seconds). In addition, all red alarms on these ports are suppressed and a gray condition is asserted. If the Port Always Down Alarm Suppression attribute is set to Disabled, the port will be polled according to the criteria listed above. How to Manage Your Network with SPECTRUM Page 82 Document 1909 Port Status Events and Alarms SPECTRUM's port status monitoring engine uses the events and alarms listed in Table 2 to notify the user of a change in status. Table 2: Port Status Events and Alarms Event Description Event ID Alarm Description Alarm ID Port Condition Color Port status good 0x10d10 N/A N/A Green Port status bad 0x10d11 BAD LINK 0x1040a Red Port status disabled 0x10d12 LINK DISABLED 0x1040b Brown Port status unknown 0x10d13 LINK STATUS UNKNOWN 0x1040e Gray Port status unreachable 0x10d14 UNREACHABLE LINK 0x1040c Gray Port status initial 0x10d15 N/A N/A Blue Port lower layer down 0x10d16 BAD LINK, BUT ALARM WAS SUPPRESSED 0x1040f Gray Port up, but linked with down port 0x10d17 LINK MAY NOT BE UP 0x10410 Gray Port connected to down port 0x10d18 or device PORT ALARM SUPPRESSED 0x10411 Gray 0x10d2d PORT ALARM SUPPRESSED 0x10d2d Gray Port status bad, but connected to WA_Link whose LinkFaultDisposition is LinkOnly Link Trap Handling Traps provide a means for network devices to let a management system know that a significant event has occurred on the network. Link Down and Link Up traps are perhaps the most important traps when it comes to port status monitoring. These traps tell the management system that a port has either become inoperable, or has come back up. When SPECTRUM receives a Link Down trap, it polls the status of the corresponding port once to verify its status and generates one of the Port Status Events and Alarms listed in Table 2 on the affected port. How to Manage Your Network with SPECTRUM Page 83 Document 1909 Note: SPECTRUM generates a yellow alarm on the device model to allow easy access to vendor-specific trap data, however it no longer generates the trapspecific events and alarms on the affected port model. Once a Link Down trap is received, SPECTRUM sets the OutstandingLinkDownTrap attribute on the port to TRUE. This will cause SPECTRUM to poll the status of the port regardless of the port status polling criteria. When SPECTRUM receives a Link Up trap for the port, or when the port’s status is determined to be up based on a poll, the value of the OutstandingLinkDownTrap attribute is set to FALSE and polling will take place based on the value of the port status polling criteria. (See Port Status Polling Criteria [page 82] for more information on when a port is polled.) When all of the ports for which SPECTRUM has received a Link Down trap are back up, the yellow alarm on the device will be cleared. Table 3 provides the attributes that can be used to control how SPECTRUM handles link traps. Table 3: Link Trap Attributes Attribute ID Description AlarmOnLinkDownIfTypes 0x1290f This attribute contains a mapping of ifType values to a value which determines how to handle the trap for that particular ifType and model type (0 for never, 1 for always, and 2 for check admin). This can be customized in the MTE on a per-model type basis. When a port model is created, its AlarmOnLinkDownTrap (0x11fc2) attribute is automatically populated with the value which corresponds to its particular ifType. AlarmOnLinkDownTrap 0x11fc2 This attribute (available via the port model's Interface Trap Configuration View) sets the alarm generation behavior for receipt of a LINK DOWN Trap Event. Select the desired setting from the drop down list. Possible settings are: • Never (0) = Do not generate an alarm upon receipt of LINK DOWN trap • Check Status (1) = Generate an alarm based on the current Admin Status (UP = Generate a Red alarm, otherwise generate a Brown alarm) How to Manage Your Network with SPECTRUM Page 84 Document 1909 Attribute ID AssertLinkDownAlarm Description 0x12957 This attribute is used to determine if the yellow alarm should be generated on the device model. It is read from the port model for which SPECTRUM received the trap. This attribute is available via the port model's Interface Trap Configuration View. Interface Trap Configuration View For many device models, you can configure the processing of link down traps received for individual port models either through an Interface Trap Configuration View (accessed from the port model’s Icon Subviews menu) or a Generic Interface Trap Configuration View (accessed from the Trap Configuration button in the port model’s Model Information View). Both of these views allow you to suppress link down alarms for the selected port model and/or its parent device model. For the port model, you can also select the Check Status option, in which case SPECTRUM will check the interface’s Administrative Status whenever a link down trap is received, generating a red alarm only if the status is “Up” and a brown alarm for any other status. Consult the SPECTRUM management module guide for the type of device you are interested in to see if that module supports a trap configuration view. Figure 40 shows an example of the Interface Trap Configuration View. Figure 40: Interface Trap Configuration View How to Manage Your Network with SPECTRUM Page 85 Document 1909 PollPortStatus Feature The PollPortStatus feature lets you monitor the status of a port even if the port’s connectivity is not modeled. The PollPortStatus attribute exists for both Device and Port models with a different attribute ID for each of the two model types. This lets you enable and disable port status polling at the device and/or port level. By default, PollPortStatus is set to TRUE at the device level and FALSE at the port level. Note: To reduce network traffic, SNMP reads for polled ports on the same device are grouped together into larger SNMP requests. This provides performance benefits that are most noticeable when many ports are polled by this method in a single SpectroSERVER. Utilizing PollPortStatus to Watch a Connected Port’s Status The PollPortStatus attribute at the device level (0x12809) controls port status polling on a per-device basis. If TRUE, polling is enabled for that device. If FALSE, no ports will be polled, even if a port model’s PollPortStatus attribute is TRUE. When changed to FALSE, alarms will be cleared for any port which is not involved in a live pipe. The PollPortStatus attribute at the port level (0x1280a) controls polling for each port model. If this attribute is set to TRUE (and PollPortStatus for the device is also set to TRUE), the status of the port will be polled, and alarms will be generated if needed. When changed to FALSE, any alarm on the port will be cleared. Table 4 shows that a port’s status will be polled only if PollPortStatus is TRUE for both a given port model and its device model. Table 4: Port Status Polling Conditions Device Model’s Port Model’s PollPortStatus Value PollPortStatus Value FALSE FALSE Port status is not polled for any port on device TRUE FALSE Port status is not polled for this port on device FALSE TRUE Port status is not polled for any port on device TRUE TRUE Port status is polled for this port on device Result How to Manage Your Network with SPECTRUM Page 86 Document 1909 SPECTRUM watches the polling interval to determine when to poll port status. When a port is polled, the port’s status is determined and an appropriate alarm is generated (RED, BROWN, or GRAY) if necessary. If a BAD LINK alarm is generated on a port (alarm code 0x1040a), and then polling on that port is disabled by changing the value of PollPortStatus to FALSE and disabling Live Pipes, an event will be generated to automatically clear the BAD LINK alarm. Note: Now that port status polling has been integrated in SPECTRUM, the PollPortStatus attribute can be set to TRUE while the Live Pipes functionality is enabled. This will not cause redundant network traffic. Enabling Port Status Polling You can enable port status polling for all future models of a given type using the Model Type Editor (MTE), or on a current, per-model basis using the SPECTRUM Command Line Interface (CLI). For example, use the MTE to set PollPortStatus to TRUE for both device and port model types. When polled, interface models will now generate appropriate alarms when needed. For more information see the Model Type Editor User’s Guide (0659). PollPortStatus can also be set at both the device and port level using the Global Attribute Editor. For more information, see the Search Manager User’s Guide (2383). Note: You can use the SPECTRUM CLI to enable or disable PollPortStatus for a single model. See the Command Line Interface (0664) documentation for more information. Wide Area Link Monitoring SPECTRUM polls any ports connected to a WA_Link model for status automatically. This polling is controlled by the Polling_Status and Polling_Interval of the WA_Link model. If the WA_Link model's Polling_Status is TRUE, and its Polling_Interval is non-zero, SPECTRUM will automatically make the pipes connected to a WA_Link “live” and set the PollPortStatus for port models to TRUE. The live pipe lets you visually verify that SPECTRUM is monitoring the status of the connected ports. If you choose to turn off the live pipe, but leave the WA_Link’s Polling_Status set to TRUE, Polling_Interval set to a non-zero number, How to Manage Your Network with SPECTRUM Page 87 Document 1909 and the ports’ PollPortStatus set to TRUE, SPECTRUM will continue to monitor the status of the ports. Link Fault Disposition The Link Fault Disposition setting gives you more control and flexibility over fault alarming. When a wide area connection goes down, alarms could be generated on ports and/or the link model. Using the WA_Link Fault Management View [page 88] for a WA_Link model, you can set the Link Fault Disposition (0x129e2) attribute. To access this view, right-click on a WA_Link model and choose Fault Management from the icon subviews menu. The default setting for Link Fault Disposition is BothPortsAndLink. It can be set to one of three different modes: BothPortsAndLink, PortsOnly, or LinkOnly. If this button is set to BothPortsAndLink, alarms will be generated on both the connected ports and the link model. If set to PortsOnly, only the connected ports will be alarmed, and the WA_Link will be suppressed. If set to LinkOnly, only the WA_Link model will be alarmed, and the ports will be suppressed. Figure 41: WA_Link Fault Management View How to Manage Your Network with SPECTRUM Page 88 Document 1909 Wide Area Link Monitoring Scenarios Consider the network topology shown in Figure 42. Table 5 and Table 6 illustrate two WA_Link monitoring scenarios for this topology. Figure 42: Example WA_Link Network Topology Device A Device B Port 1 Port 2 WA_link Table 5: Link Goes Down, Device B Loses Contact Link Fault Disposition Port 1 Condition WA_Link Condition Port 2 Condition BothPortsAndLink Red Red Gray PortsOnly Red Gray Gray LinkOnly Gray Red Gray Table 6: Link Goes Down, Device B Remains Reachable Link Fault Disposition Port 1 Condition WA_Link Condition Port 2 Condition BothPortsAndLink Red Red Red PortsOnly Red Gray Red LinkOnly Gray Red Gray Note: In Scenario 2, if the Suppress Linked Port Alarms [page 93] setting is set to TRUE, then only one of the ports will be alarmed (red). The other port will be suppressed (gray). How to Manage Your Network with SPECTRUM Page 89 Document 1909 Port Layer Alarm Suppression Devices that support advanced network technologies, such as Frame Relay and Link Aggregation, have logical entries in the ifTable representing higher layer interfaces. SPECTRUM models these logical layers according to the ifStackTable. When a monitored higher layer port goes down (such as a Frame Relay DLCI, or logical trunk interface), SPECTRUM will query the statuses of all lower layer interfaces before alarming the port which went down. If all of the lower layer interfaces are down as well, SPECTRUM will suppress the higher layer interface alarm. A key example is a physical Frame Relay interface going down which has multiple circuits provisioned on it. All of the higher layer DLCI port models will be suppressed, and the single red alarm will exist on the physical Frame Relay interface. Port Criticality You can assign a relative importance value to port models using the port criticality (0x1290c) attribute. The criticality of a port is used in the Impact Severity calculations of an alarm on any port which may cause a network outage. You can also display the criticality of a port in the Web Alarm View to allow prioritization of port alarms. The port criticality attribute can be set for an individual port using the port model's Port Fault Management View, or the Global Attribute Editor. For more information, see the Search Manager User’s Guide (2383). Figure 43: Port Fault Management View How to Manage Your Network with SPECTRUM Page 90 Document 1909 Live Pipes Live Pipes lets you turn on port status monitoring for individual links and display the status of a link through the use of status colors. A link is a connection between two devices that has been resolved to the port level. Live Pipes display a combined condition color from the two resolved ports representing each side of the link. When a pipe is deleted, SPECTRUM removes all of its underlying associations (such as links_with and connects_to). If the pipe being deleted represents more than one link, SPECTRUM asks the user to confirm the deletion. For more information on pipes, refer to the SPECTRUM Icons Reference Guide (2518) and Getting Started with SPECTRUM for Administrators (0985). Enabling or Disabling Live Pipes System-Wide Live Pipes are enabled system-wide by default. Without Live Pipes enabled on a system-wide basis, enabling Live Pipes for individual links is not available. To enable or disable Live Pipes system-wide: 1. In the Landscape Configuration view’s Configure/Information Panel, select the LivePipes entry and click OK to access the Live Pipe Model Information view. 2. Toggle the Live Pipes button to Enable or Disable. 3. Select Save All Changes from the File menu. 4. Exit and restart SpectroGRAPH for the changes to take effect. Enabling or Disabling Live Pipes on Individual Links Live Pipes for individual links are disabled by default. All individual pipes are gold or silver until Live Pipes are enabled. When an individual live pipe is enabled, the ok_to_poll (0x11dd8) attribute is set to TRUE for both ports involved in the link. The status of the linked ports will be monitored if ok_to_poll is TRUE and the port status polling conditions in the Port Status Polling Criteria section on [page 82] are met. To enable a live pipe for an individual link: 1. Click on the link to highlight it, then select View > Icon Subviews. 2. Select Enable Live Links from the menu. How to Manage Your Network with SPECTRUM Page 91 Document 1909 3. In the Enable Live Links dialog box, set the toggle button and click OK. To disable a live pipe for an individual link: 1. Click on the link to highlight it, then select View > Icon Subviews. 2. Select Enable Live Links from the menu. 3. In the Enable Live Links dialog box, raise (un-check) the toggle button and click OK. Note: The Enable Live Links option is not available if Live Pipes have been disabled system-wide. Receiving Port Alarms In order to receive alarms on a port model, the following conditions must be met: • Live Pipes must be enabled system-wide through the Landscape Configuration view. • Live Pipes for the link of interest must be enabled separately. Note: If there is no model on the other side of the link, an alarm can still be generated when the status of the link changes by setting a SpectroWATCH on the MIB-II ifOperStatus attribute. When the ifOperStatus attribute returns a value other than 1, an alarm is generated. With this method, an alarm can still be generated even if the port is intentionally down (the ifAdminStatus attribute has been set to OFF). Additional information about the status of a link is provided through the Link Information view. For information about the Link Information view, see the SPECTRUM Icons Reference Guide (2518). You can set the default value of ok_to_poll in the MTE for any port model type. When a port connection is made, SPECTRUM will set both ports' ok_to_poll attributes to their MTE default values so that the pipe will automatically become live if the user wishes. When the connection is removed, the value for ok_to_poll remains at the MTE default value. When SPECTRUM notices a change in the link's status, it will generate one of the Port Status Events and Alarms listed in Table 2 on the two ports involved in the link, and will also change the color of the live pipe to reflect its new status. How to Manage Your Network with SPECTRUM Page 92 Document 1909 The settings described in Table 7 allow you to control the service of Live Pipes. These settings appear in the Live Pipes Model Information View, accessible from the VNM model icon’s Landscape Configuration View. Table 7: Port Status Monitoring Settings Setting Attribute ID Description Live Pipes 0x11df9 Setting this (global) option to Disabled will turn off all pipes in the SpectroSERVER and no status polling will be performed for any ports associated with a pipe. However, any port which also has the PollPortStatus attribute (see PollPortStatus Feature [page 86]) set to TRUE will still be polled for status changes. Alarm Linked Ports 0x11fbd Setting this option to TRUE will cause ports with a good status to have a gray alarm generated on it when linked with a bad or unreachable port. Suppress Linked Port Alarms 0x11fbe Setting this option to TRUE causes ports with a bad status to suppress their red alarms and generate a gray alarm if either the linked port or the connected device is bad or unreachable. Only one port in the link will be alarmed red. The other will be gray. Port Always Down Alarm Suppression 0x12a03 Enabling this option will cause SPECTRUM to suppress red alarms and assert a gray condition if a port has always been down since first being modeled in SPECTRUM. Note: The settings described in Table 7 apply to port status monitoring throughout SPECTRUM, not just ports associated with Live Pipes. Automatic Port Status Alarm Clearing It is important to note that any port status alarm will be automatically cleared by SPECTRUM if the port's ok_to_poll and PollPortStatus attributes are both set to FALSE. How to Manage Your Network with SPECTRUM Page 93 Document 1909 Suggested Port Fault Settings for Optimal Fault Notification In SPECTRUM 7.0 and later, the default settings for both Suppress Linked Port Alarms (see Table 7) and Port Fault Correlation (see Configuring Port Fault Correlation [page 72]) have changed. The default value of Suppress Linked Port Alarms has changed from FALSE to TRUE. This setting will suppress red alarms on ports that are connected to another down port or unreachable device. The default value of Port Fault Correlation has been changed from Management Neighbors Only to All Connected Ports. This eliminates the need for you to manually configure management neighbors (see Configuring Management Neighbors [page 64]) before a fault occurs so that port fault correlation will work properly. Note: Setting Suppress Linked Port Alarms to FALSE and Port Fault Correlation to Management Neighbors Only will approximate the fault notification behavior of SPECTRUM 6.6 with Service Pack 3. Additionally, Aprisma recommends changing the default setting for Link Fault Disposition on WA_Link models (see Link Fault Disposition [page 88]). The default setting of BothPortsAndLink will result in multiple alarms if the link fails. Consider changing the setting to Link Only or Ports Only. Link Only is best in environments where the name of the WA_Link models or notes on these models is meaningful. Changing the setting to Ports Only may be better if fault notification consistency is most important. That is, regardless of the topology, you would prefer an alarm on a port model if there is a link failure. Other Fault Management View Settings The Fault Management view also provides other user-configurable settings, both for device models and port models. Device Model Settings Disable Trap Events Set this selector button to TRUE to prevent the generation of events (and alarms) for traps that are received from the selected device. How to Manage Your Network with SPECTRUM Page 94 Document 1909 Device Criticality This text entry box takes a value that indicates the relative importance of the device within the network being modeled. When SPECTRUM loses contact with the device, this value is summed for the device itself and all of its downstream neighbors for which a gray condition is now being asserted (because their actual status cannot be determined). The aggregate device criticality value is displayed as the Impact Severity value of the associated alarm in the Alarm Manager’s Impact Scope view. The default value for all devices is “1”; you can increase this value as desired depending on how important you consider the device to be (the higher the number, the more critical the device is to the network). For more information on the Impact Scope view, refer to the Enterprise Alarm Manager User Guide (2065). Redundancy Options This button allows you to control (enable/disable) whether SPECTRUM attempts to use redundant addresses in the event the device becomes unreachable via its primary address. See Redundancy [page 152] for more information. Port Model Settings Enable Event Creation When this value is set to FALSE, SPECTRUM suspends the creation of events for the model. This has the effect of disabling alarms for the model, but unlike maintenance mode (see Configuring Maintenance Mode [page 68]), SNMP communication with the model continues. The default setting is TRUE. Port Criticality You can assign a relative importance value to port models using the port criticality (0x1290c) attribute. See Port Criticality [page 90]. Configuring Fault Management for Pingables Device fault isolation relies upon device models' knowledge of each other as neighbors. When a fault occurs, each device model that is lost sends an ARE_YOU_DOWN action to all of its neighbors. Depending on the answers received from neighboring device models, the lost model will either turn gray or red. Neighbors are established by the creation of a Connects_to association between a device model and the port model of another device. How to Manage Your Network with SPECTRUM Page 95 Document 1909 The act of pasting a device model on the interface of another device model adds each device model to the other's neighbor list. Prior to SPECTRUM 6.0.3, this presented a problem for Pingable models. They don't have ports, and so they could not be made neighbors without the use of an inferred connector, like a Fanout. However, it is preferable for the relationship between models to be resolved directly, rather than by placing them both in a Fanout. This is because Fanouts and other inferred connector model types resolve faults differently and less directly during fault isolation. Connecting Two Pingable Models It is now possible to connect pingables using one of the following methods: • You can establish neighbors simply by drawing pipes between device models. Drawing a pipe between two pingables establishes a Connects_to association between the two models and thus establishes them as neighbors. • You can create a Connects_to association between two Pingable models via CLI. The syntax is: ./create association rel=Connects_to lmh=<model handle of Pingable A> rmh=<model handle of Pingable B> Once the Connects_to association has been established, you will see a gold pipe between the two models. When a fault occurs, each Pingable will send the other an ARE_YOU_DOWN action. Connecting a Pingable Model to a Device’s Port Model When creating a neighbor relationship between a pingable and a device model that has ports, you must paste the pingable onto the device’s port model. See Resolving Port Connections [page 24] for further instructions. How to Manage Your Network with SPECTRUM Page 96 Document 1909 Fault Management This section describes the SPECTRUM features that provide rapid fault detection. This chapter also describes how to access SPECTRUM views to pinpoint a fault’s location. In This Section Fault Isolation Overview [page 97] Fault Detection [page 100] Fault Isolation [page 104] Fault Isolation Overview A Fault Isolation Flow Chart (Figure 44) is included as an example for using the detection and pinpointing features as described in this chapter to isolate a fault. This flow chart is not intended to cover all circumstances and fault conditions that may be experienced within your network but is for example only. The following applications, views, and navigational features can be useful in diagnosing network faults. • Device Condition Colors [page 100] • Rollup Condition Colors [page 102] • Enterprise Alarm Manager [page 104] • Audible Alarms [page 107] • Isolating a Fault Through Navigation [page 107] • Chassis Device View [page 109] • Device Topology View [page 110] • Performance View [page 111] How to Manage Your Network with SPECTRUM Page 97 Document 1909 Figure 44: Fault Isolation Flow Chart Rollup Condition Alarm Generated LAN/Landscape Icon indicates a rollup condition. Refer to Fault Detection in the previous section. Open the LAN/Landscape Alarm Manager to identify the device, read the probable cause and recommended action. Refer to the Alarm View Section. Use Navigate In to access the LAN Topology containing the faulty device. Refer to the Navigate In section. Yes Is the alarm significant enough to require further investigation? Is the fault generating a condition color on the device? No Yes Acknowledge the alarm to silence it. Open the Alarm Manager for the device and read the probable cause and recommended action for the specific alarm. Refer to the Alarm View section. No Continued How to Manage Your Network with SPECTRUM Page 98 Document 1909 Continued Open the Device View to identify the faulty port or module managed by the device. Refer to the Device View section. No Is there a rollup condition color on any of the LANs connected to the device? Yes Access the Device Topology View or open the LAN Topology View to identify the faulty device. Access the Alarm Manager for the device generating the alarm and read the probable cause and recommended action. Refer to the Alarm Manager section. Open the port or module performance view. Refer to the Performance View section. Is the fault caused by a device connected to a port (e.g., excessive %Load). No Access the Alarm Manager for the device generating the alarm and read the probable cause and recommended action. Refer to the Alarm Manager section. Disable the port from the Device View. Check the physical device for hardware, cable, or firmware failure. Yes Use the source address table, MALT, RMON, or a LAN analyzer to identify the offending device. Shut down the device if necessary. How to Manage Your Network with SPECTRUM Page 99 Document 1909 Fault Detection Faults are detected through traps received from the device or configured through SPECTRUM. These traps are used to generate rollup conditions, device conditions, alarms, and event messages for any active port and any device. Device Condition Colors Condition colors are displayed on an icon to reflect the status of the model represented by the icon. The colors appear in the background of the label that indicates the kind of device or other network entity being modeled, as shown in Figure 45. If an event causes the condition color to change, the event will be registered in the Event Log. The Alarm Manager will indicate the Symptom/Probable Cause and recommended actions for that event. Figure 45: Device Condition Colors Blue - No contact with device has yet been established. Green - Contact established and condition is normal. Yellow - First level of marginal operation or duplicate IP address. HubCSIEMME Orange - Device is functioning but is unresponsive to management requests. Red - Contact with this device has been lost. Gray - Condition suppressed; device cannot be reached due to a known error condition that exists on another device. Blue (Initial) If SPECTRUM has not made contact with the device since the model was created, its condition color is blue. To ping such a device to check its status, highlight the device icon and select Ping from the Icon Subviews > Utilities menu. The following message will appear on your SpectroGRAPH console indicating that pinging is in progress. Pinging 132.177.118.24 PING 132.177.118.24 64 data bytes How to Manage Your Network with SPECTRUM Page 100 Document 1909 Green (Normal) If SPECTRUM has made contact with the device and operation is normal, the condition color will be green. Note: Location fault isolation does not work exactly the same as container fault isolation. If a site (Location) contains devices with gray conditions, the site is still shown with a green condition. However, if a container model (LAN for example) contained these same devices, the container would show a gray condition. Site models generally represent a building or location which give an idea as to what models (devices) are inside that location, but the location itself (its condition) is not reflected in the same way that LAN models are reflected. For Site models, the Rollup condition is displayed. LAN models have both Rollup condition and model condition displayed. Yellow (Minor Alarm) A yellow alarm signifies a situation that does not impact overall network operation, but for which SPECTRUM has received information. An alarm with the probable cause A module within this hub has been removed or An IP (Internet Protocol) address or physical (Ethernet) address assigned to this model has already been assigned to another model is an example of a yellow alarm. Orange (Major Alarm) If SPECTRUM cannot retrieve management information from a device, but the devices connected to this device have a normal condition (green), then the condition color will be orange. For example, an alarm with the probable cause A firmware problem exists in the device is an orange alarm. Red (Lost Contact) If SPECTRUM detects a complete failure of the device or has lost contact with the device, the condition color will be red. Gray (Unknown) When device status is unknown, alarms for the device are suppressed. For example, if contact with a device is lost due to network problems on an How to Manage Your Network with SPECTRUM Page 101 Document 1909 upstream device (i.e., between the device and SpectroSERVER), the actual condition of the device cannot be determined and its condition color will be gray. Condition color for any devices connected downstream from this device would also be gray. Rollup Condition Colors When a fault occurs with a device, a trap is generated by the device or by SPECTRUM for network management purposes. SPECTRUM receives the trap and generates a device condition color which indicates the status of the device. The importance of the device to your network determines the significance value you have placed on the condition. Figure 46 shows the location of the rollup condition color and defines the relationship between the color and the severity. Figure 46: Rollup Condition Colors Yellow - Information Alarm Orange - Minor System or Subsystem Failure Red - Major System or Subsystem Failure For example, you would want to associate a high significance number with a red condition color for a device that is critical to network performance; then, if the device failed, the red condition color would roll up to the model that contains the device, thus indicating that a critical device or devices have failed within that model. Conditions roll up to the next level in the hierarchy as determined by the significance level of the alarm and by the threshold level set for the next model in the hierarchy. For details on configuring rollup condition colors, thresholds, and significance levels, see Configuring for Fault Management [page 53]. How Is a Rollup Condition Detected? Rollup condition colors (Figure 47 and Figure 48) are used to indicate that a fault has been detected as described in the following example: How to Manage Your Network with SPECTRUM Page 102 Document 1909 Figure 47: Example Rollup Condition File View Tools Bookmarks Help Model Name 4 1 The Red rollup color indicates that a critical system or subsystem failure has been detected within this landscape network. 7 Landscape Major alarm - rollup condition color is red. File View Tools Bookmarks Help IPClassB The rollup threshold has been set for this critical LAN to rollup a major system or subsystem failure condition from the models it contains. LAN_802_3 Major alarm - rollup condition color is red. File View Tools Bookmarks Help Automapped_LAN Bridge LAN_802_3 This critical device has failed and generated an alarm. The significance level assigned to this device will cause a rollup condition based on the threshold set in the LAN 802.3 that contains this device. Failed device generates a red condition color. How to Manage Your Network with SPECTRUM Page 103 Document 1909 Figure 48: Example Rollup Condition Colors Model Name Information Alarm Rollup Condition Color is Yellow 4 1 7 With a yellow rollup threshold set to 3 and the sum of the significance levels for the “Contained” models equal to 3, the rollup condition color will be yellow. Landscape Minor Alarm Rollup Condition Color is Orange With an orange rollup threshold set to 3 and the sum of the significance levels for the “Contained” devices equal to 4, the rollup condition color will be orange. LAN 802.3 Device Failure Device Condition Color is Red A noncritical device with a significance level of 4. HubCSIEMME Fault Isolation Fault detection is the first step in fault isolation. Utilizing the proactive fault management features described in Fault Detection [page 100], SPECTRUM detects that a fault has occurred and generates an alarm. Once the fault has been detected, you can use the pinpointing features of SPECTRUM to navigate to the view that identifies the specific network, device, channel, or port that generated the alarm. Enterprise Alarm Manager Enterprise Alarm Manager (EAM) views exist for the whole network, LAN icons, and devices. It is recommended that you keep the views for models essential to your network open and minimized at all times. The Enterprise Alarm Manager provides the Symptom/Probable Cause of the alarm, suggests actions to correct the error, and allows you to assign a repair person to the task. How to Manage Your Network with SPECTRUM Page 104 Document 1909 Accessing the Enterprise Alarm Manager To access the EAM, choose Tools > Alarm Manager or, to view alarms on a specific model, highlight the applicable model icon and select Model Alarms from the Icon Subviews menu. SPECTRUM provides a feature called Auto Raise which automatically opens the EAM if it has been minimized on your work station. This feature is enabled from the EAM Options menu (Figure 49). Figure 49: Enterprise Alarm Manager Alarm Information Panel Tool Bar Buttons (provides the same functions as the menus) Menu selections Enterprise Alarm Manager: Main File View Model Alarms Troubleshooter Options Help Filter Sort Order Select All Ack Unack Clear Assign Unassign Ping Telnet MIB Tools What’sThis? Hints System Probable Cause Events* History* Alarm Status Trouble TicketID Location Device Notes 111.222.3.4. DIFFERENT TYPE MODEL GnSNMPDev SYMPTOMS: Device may be missing some device-specific intelligence. PROBABLE CAUSES: Severity Date/Time Critical Critical Critical Major Major Major Minor Minor Minor Model Name Fri 12 May 2000 15:11:08 Wed 10 May 2000 15:11:08 Wed 3 May 200012:59:33 Sat 29 Apr 2000 16:28:34 Wed Apr 12 2000 17:00:00 Mon 15 May 200008:48:11 Thu 20 Apr 2000 11:34:23 Wed 19 Apr 2000 12:34:21 Wed 12 Apr 2000 17:00:00 Search golem golem praetorius praetorius Administrator AlarmMgmt 111.222.3.4 golem praetorius Network Address Vendor Name 111.222.3.4 Model Class Cabletron Systems Unknown Model Type GnSNMPDev Prev Shown Critical: 3 Next Displayed 9 of 9 Filtered by: Severity Major: 3 Total: 6 Minor: 3 Servers Alarm Condition Panel (totals of each severity) Alarm List How to Manage Your Network with SPECTRUM Page 105 Document 1909 Using the Enterprise Alarm Manager Once an alarm has been generated, use the Enterprise Alarm Manager to isolate the fault as follows: 1. If multiple alarms exist, click on an alarm from the list within the view (Figure 50) to display that model’s alarm information and its icon in the Alarm Information panel. The icon provides the double-click zones and Icon Subviews menu selections for that model. Figure 50: Selecting the Alarm to be Isolated Severity Critical Critical Critical Major Major Major Minor Minor Minor Date/Time Model Name Fri 12 May 2000 15:11:08 Wed 10 May 2000 15:11:08 Wed 3 May 200012:59:33 Sat 29 Apr 2000 16:28:34 Wed Apr 12 2000 17:00:00 Mon 15 May 200008:48:11 Thu 20 Apr 2000 11:34:23 Wed 19 Apr 2000 12:34:21 Wed 12 Apr 2000 17:00:00 golem golem praetorius praetorius Administrator AlarmMgmt 111.222.3.4 golem praetorius Network Address Vendor Name 111.222.3.4 Model Class Cabletron Systems Unknown Model Type GnSNMPDev 2. In the Alarm Information panel, click on the Probable Cause tab (Figure 51) to read a complete description for that alarm. This view identifies the symptom, suggests a probable cause, recommends an action, and provides the name of an assigned troubleshooter. Figure 51: Reading the Probable Cause for the Alarm System Probable Cause Events* History* Alarm Status Trouble TicketID Location Device Notes 111.222.3.4. DIFFERENT TYPE MODEL GnSNMPDev SYMPTOMS: Device may be missing some device-specific intelligence. PROBABLE CAUSES: Severity Critical Critical Critical Major Major Major Minor Minor Minor Search Date/Time Fri 12 May 2000 15:11:08 Wed 10 May 2000 15:11:08 Wed 3 May 200012:59:33 Sat 29 Apr 2000 16:28:34 Wed Apr 12 2000 17:00:00 Mon 15 May 200008:48:11 Thu 20 Apr 2000 11:34:23 Wed 19 Apr 2000 12:34:21 Wed 12 Apr 2000 17:00:00 Model Name golem golem praetorius praetorius Administrator AlarmMgmt 111.222.3.4 golem praetorius Network Address Vendor Name 111.222.3.4 Model Class Cabletron Systems Unknown Model Type GnSNMPDev Prev Shown How to Manage Your Network with SPECTRUM Page 106 Next Document 1909 3. Depending on the type of fault and size of your network, you may want to use one of the following pinpointing options to isolate the fault. • To see the impact the fault has on the network and other devices in the network before proceeding directly to the fault, refer to the Point and Click [page 108] and Navigate In [page 108] sections for details. • If the fault is related to an individual port, you can go directly to the Chassis Device View to identify the port and view port-specific performance information. Refer to Chassis Device View [page 109] for more details. Audible Alarms When new alarms occur, an audible sound indicates which severity of alarms have entered the system. The default sound is a voice stating the severity, e.g., “Critical Alarm,” or “Minor Alarm.” Refer to the Enterprise Alarm Manager User’s Guide (2065) to learn how to customize audible alarms. If you have the EAM view for your network open and minimized (Figure 52), an audible alarm will sound to indicate an alarm has been generated. If Auto Raise is selected in the Options menu, the EAM will open automatically when an alarm is generated. Figure 52: Minimized Alarm View Isolating a Fault Through Navigation There are two ways to navigate through the topology hierarchy to locate a device that is generating an alarm, point and click and Navigate In. How to Manage Your Network with SPECTRUM Page 107 Document 1909 Point and Click You can navigate through an affected model by pointing and clicking the down arrow contained in its icon, as shown in Figure 53. Figure 53: Point and Click SpectroGRAPH File View Tools Help Bookmarks Model Name 4 1 7 Landscape Down Arrow Double-click the Down Arrow Navigate In To use the Navigate In feature, follow these steps: 1. Highlight the Landscape or LAN icon in a Topology view (Figure 54) then select Navigate > Navigate In from the Icon Subviews menu. 2. Double-click the IP Address of type LAN_802_3 showing the rollup condition to access its Navigate In list. Repeat this process until you have opened the list containing the faulty device indicated by the rollup condition colors. The LAN containing the device is listed at the top. 3. From this Navigate In list you may do one of the following: • Double-click the LAN containing the faulty device to access the LAN Topology view. This view will show the network affected by the faulty device. The faulty device and any affected devices will be indicated by rollup and device condition colors. Refer to Fault Detection [page 100] for details on condition colors. • Double-click the faulty device listed to open the Device Topology view for that device (see Device Topology View [page 110]). How to Manage Your Network with SPECTRUM Page 108 Document 1909 Figure 54: Navigate In View Step 1 Highlight the Landscape Icon with the middle mouse button and select Navigate and Navigate In. Step 2 Double-click opens the Navigate In view for that LAN indicated by the arrow. SpectroGRAPH: Topology: Universe File View Tools Help Bookmarks IP Address of type Landscape IP Address of type IPClassB Model Name 4 IP Address of type LAN_802_3 IP Address of type LAN_802_3 1 7 IP Address of type LAN Landscape IP Address of type LAN_802_3 IP Address of type HubCSIEMME IP Address of type LAN_802_3 IP Address of type LAN_802_3 IP Address of type HubCSIEMM_E6 IP Address of type GnSNMPDev Step 3 Double-click opens the Device Topology view for that device. LAN Containing the devices and LANs listed below Chassis Device View Use the Chassis Device view (Figure 55) to identify the faulty module or port. The Chassis Device view will provide you with port and module information at a glance. Refer to the management module guide for the particular device for Device view information. Accessing the Chassis Device View Access the Chassis Device view using one of the following methods: • Click on the Device View button on the device icon. • Highlight the device icon and select Device View/Chassis from the Icon Subviews menu. How to Manage Your Network with SPECTRUM Page 109 Document 1909 Figure 55: Example Chassis Device View Logical Modules Module Channel or Isolation Station Module Number Module Type Port Number Port Status Frame Rate 2 Network A 1 FOM-22 1 SEG Pkts 0 2 LNK Pkts 0 3 LNK Pkts 0 4 LNK Pkts 0 5 SEG Pkts 0 6 ON Pkts 487 7 LNK Pkts 0 8 LNK Pkts 0 9 SEG Pkts 0 EMM-E6 10 SEG Pkts 0 11 ON Pkts 147 Bridge Port Bridge Status Bridging A FWD B FWD C FWD D1 FWD D2 OFF Repeaters A UNLOCKED B UNLOCKED C UNLOCKED Redundant Port Repeater Ports Network Port Security FDDI 12 SEG Pkts 0 Device Topology View The Device Topology view for a device provides you with port status and access to specific port performance and error views. Accessing the Device Topology View There are several ways to access the Device Topology view: • Double-click on the device in the Navigate In list. • Double-click on the down arrow on any device icon. • Highlight the icon and select DevTop view from the Icon Subviews menu. How to Manage Your Network with SPECTRUM Page 110 Document 1909 Using the Device Topology View The Device Topology view can be used to isolate the fault to the port level. For example, if a port on a device is bad, the Device Topology view (Figure 56) for the device will show the devices connected to a particular port or interface. If the device connected to the port is gray (suppressed), the port is bad and contact to the device connected to the port is lost. Figure 56: Using the Device Topology View Click the module to display its port connections below. SpectroGRAPH : Device Topology : 111.222.333.4 The condition color status label is gray indicating a suppressed network downstream of the device this Device Topology view is for. 1 0 B T I R M 3 The condition color status label for the icon that represents the device connected to this port is gray (suppressed), indicating a bad port 4.1 on the IRM 3. Port Connections A ON B ON C ON ETHERNET ETHERNET ETHERNET 0:0:1D:19:AC:52 0:0:1D:19:AC:53 0:0:1D:19:AC:54 0:0:0:0 0:0:0:0 111.222.333.4 Interface Icons Performance View The Performance view for a device model provides you with the following general performance information for isolating faults: • Multi-Attribute Line Graph How to Manage Your Network with SPECTRUM Page 111 Document 1909 • Error Breakdown Pie Charts (some management modules) • Frame Breakdown Pie Charts (some management modules) • Port Frame Size Breakdown Pie Charts (some management modules) • Protocol Breakdown Pie Charts (some management modules) Accessing the Performance View Access the Performance view using one of the following methods: • Highlight the device icon and select Performance view from the Icon Subviews menu. • Double-click on the Performance View label on the device icon. • Highlight the port icons and select Performance view from the Icon Subviews menu. Using the Performance View The device, modules, and ports have individual Performance and Detail views. A Performance view summarizes statistics corresponding to the chosen primary application as color-coded entities, and gives a statistical and graphical breakdown for current, average, and peak values. The Performance view provides statistical and graphical information on network, device, application, and port performance as follows: Multi-Attribute Line Graph The Multi-Attribute Line Graph will give you a quick indication of problems (spikes) with load, frame rate, collision rate, error rate, and channel packet errors, but for more detailed information, you must access the Detail Views with error and frame breakdown pie charts. The Multi-Attribute Line Graph (Figure 57) provides a general indication of device or application activity. The attributes displayed are pre-selected and use colors to represent different statistics. Buttons allow you to modify the statistical presentation of the Multi-Attribute Line Graph. For more information on the Multi-Attribute Line Graph, refer to the SPECTRUM Views (2517) document. How to Manage Your Network with SPECTRUM Page 112 Document 1909 Figure 57: Multi-Attribute Graph Graph Properties Scroll to Date-Time Detail Pie Charts Color-coded pie charts, accessed by clicking the Detail button, present breakdowns of application statistics (Figure 58). Each statistic is presented as a total amount since the device was initialized, and as a percentage of overall traffic. The pie charts will show the current (total) attribute values, the accumulated attribute values (accumulated from the time the Accum button is clicked), or the difference (delta) between the current and previous attribute values for each related model, depending on which mode is selected. The models presented are determined by the relation selected when the field is created or modified. When Delta is selected, this pie chart can be used to compare the difference between the previously polled value and the current value of an attribute from several models, and shows each model’s contribution (percentage) to the total of the delta values. When Total is selected, this pie chart provides the current attribute values for each of the related models and each model’s contribution (percentage) to the total of values. When Accum is selected, this pie chart provides the accumulated attribute values for all related models since the Accum button was clicked and each model’s contribution (percentage) to the total accumulation of values. To restart the counter, select the Clear button. The values will continue to accumulate until clear is selected or another presentation mode is chosen. How to Manage Your Network with SPECTRUM Page 113 Document 1909 Figure 58: Pie Chart Packet Breakdown Delivered 1515660 73.85% Forwarded 67818 3.30% Transmitted 22398 1.09% Errors 9996 0.49% Discards 436528 21.27% Total 2052400 Total Delta Accum Clear Error and Frame Breakdown Pie Charts Error Breakdown pie charts can be useful for isolating problems. The total errors will give you a general indication of performance, but the accumulated errors can show you what is happening at the moment. To see the accumulated errors follow these steps: 1. Access the Error or Frame Breakdown pie charts from the performance view. 2. Click on the Accum button to start the accumulated count of errors or frames. 3. Click on the Clear button to clear the errors and start at zero. 4. Wait for about 60 seconds and then look at the accumulated errors (which will be the current errors from the time of zeroing). Troubleshooting with the Performance View When you begin to look at error rates and other traffic information, keep in mind that most of the statistics provided have to do with usage levels, packet formation problems, access problems, or network design problems. Knowing which statistics indicate which sorts of problems can help to narrow the search for your network troubles to one general area. For How to Manage Your Network with SPECTRUM Page 114 Document 1909 example, high numbers of active users and high utilization rates may explain why your network is experiencing high collision levels, and may indicate a need for bridging or routing to control traffic levels. Protocol and frame size statistics can help you identify the portions of your network that might best lend themselves to isolation. You may want to group users by protocol, or provide a group of users with an appropriate server if they are consistently using a server outside their network segment. An evaluation of frame size statistics may tell you where and/or when big file transfers are clogging up the cable, or where management packets make up the bulk of traffic (indicating a need for a DLM or RMON device). Error Rates are a function of frame rates. A high number of errors may actually be a very small percentage of overall traffic, resulting in no significant network disturbance. Looking at error rates as a function of traffic levels can be a good way to detect errors that are occurring simply because your network is getting busier. The following descriptions will help to isolate the problem: Collisions (and sometimes runts) are a natural by-product of a busy network; if you notice the levels of these two errors gradually increasing over time, or suddenly increasing at certain times of the day (accompanied by an increase in traffic levels), you may want to consider using bridges or routers for traffic management. High collision counts can also indicate other problems, such as a data loop (redundant active network connections) or a bad NIC (someone transmitting without listening first). CRC and Alignment errors can indicate some MAC layer/packet formation problem — either packets are not being properly formed in units of 8 bits, or the element responsible for either the transmit or receive CRC calculation is malfunctioning, causing perfectly good packets to be discarded as errors. These error types can also indicate some transmission medium problem — noisy or otherwise unreliable cable that is corrupting the packet as it is transmitted. Remember, according to the error priority schemes employed by Cabletron devices, packets counted as alignment errors may also have CRC errors; packets counted as CRC errors, however, do not contain truncated bytes. Out-of-Window (OOW) Collisions can either indicate that some portion of your network is out of spec — that is, the furthest distance between two nodes exceeds the 51.2 µs collision window— or that you have a bad transceiver which is transmitting without first listening for carrier sense (either intermittently or consistently). If your network begins to experience OOW collisions, use the port-level statistics screens to determine which nodes are recording these errors. Note that these nodes are not How to Manage Your Network with SPECTRUM Page 115 Document 1909 necessarily the ones which are the source of the problem: the only node that will record an OOW collision is the one that was transmitting for more than 51.2 µs before the collision occurred. If you have an established network which suddenly begins to experience OOW collisions, make sure the maximum propagation delay has not been violated and that your network is still in spec; be especially sure to check the distance between the node or nodes that are recording the errors and the nodes furthest away from those, and between the nodes recording the errors and the newest nodes. If you find that your network does meet the propagation delay spec, chances are you have a node which is transmitting without first listening for carrier sense. This can be a more difficult problem to track, since the malfunctioning node may only fail to listen occasionally, and since not all of its failures to listen will result in OOW collisions — some may simply result in collisions (if the 51.2 µs window has not closed), and some will get through fine (if no one else happens to be transmitting). The best way to track this kind of problem is to gradually bridge your network until you have isolated the OOW collisions on one segment, then check each node on that segment until you find the culprit. You might also look for a segment which is experiencing both OOW collisions and a sudden increase in regular collisions. Runts and Giants can both be caused by packet formation problems or by corrupted packets (packets being truncated or lengthened, or frame size bits being corrupted); as stated above, runts can also result from collisions. Most frequently, giant packets are the result of a jabbering node — one that just transmits continuously, either for long periods of time or only occasionally — and indicate a bad transmitter on a NIC. Framesize Statistics Aside from runt and giant packets, which are caused by errors, the sizes of the packets being transmitted on your network can give you an indication of what kind of traffic your network is seeing — many large packets can mean a lot of file transfers; small packets can mean a lot of management traffic — and can also give you an indication of the efficiency of your network — too many small packets indicates an inefficient use of bandwidth (too much overhead for too little data); too many large packets can slow your network down. You can use the framesize data to effectively bridge and/or route your network for a more evenly balanced flow of traffic. % Load, Bytes, and Packets provide a good overall sense of network usage. You can use these statistics to pinpoint peak usage times or segments of your network where usage is particularly heavy, which in turn can alert you to upcoming needs to expand, bridge, or route your network. How to Manage Your Network with SPECTRUM Page 116 Document 1909 One thing to note about packet and byte counts: on the newer Cabletron management devices (such as the EMME and the MRXI-24), packet and byte counts include errors; on older devices (such as the IRM2 or IRM3), they do not. High Broadcast packet counts can indicate the presence of a broadcast storm — an unending flood of ARP and RARP packets transmitted by a defective or improperly configured bridge or router. These storms are a serious matter and can completely shut down your network by monopolizing the bandwidth. Packet Size data can provide valuable information about how efficiently your network bandwidth is being used — too many small packets indicates a high overhead cost for the data being transmitted; too many large packets can clog your network and hurt performance. Packet size statistics can also give you a general sense of the types of data being transmitted: many large packets may indicate large file transfers; many small packets may indicate high levels of management traffic. Many packet size-related problems can be solved by bridging or routing to isolate network segments with special needs. Protocol Statistics breakdowns can be helpful if you are planning to add bridges, routers, or gateways to your network; you may want to group users by protocol so traffic needn’t pass through multiple segments to reach a designated protocol server or to avoid any problems of incompatibility. Also, different protocols have different overhead costs and may tend to use either very large or very small packets, so protocol statistics may give you some insight into other network performance issues: You may discover, for example, that a protocol which tends to have large packets also receives heavy use at a particular time of day or on a particular network segment, causing a noticeable slowdown in network traffic. Such problems can be solved by isolating those users on their own bridge- or router-protected network segment. Protocol Type data can be very useful in planning for bridges, routers, gateways, and even additional servers; you may want to group and/or isolate users based on protocol, especially if users are sending packets beyond their own network segment to reach a specific protocol server on another segment or if there are any issues of incompatibility. In addition, different protocols have different overhead requirements and transmission characteristics, which may have different effects on overall network performance. How to Manage Your Network with SPECTRUM Page 117 Document 1909 Beyond the Performance View These views provide you with information on network, device, port activity and application performance to be used to isolate a fault to a particular network segment, module, port, and/or node. You may need to use packet analyzers or other hardware-related diagnostic tools to isolate to the packet level. Depending on the significance of the fault, you may decide to disconnect that port, module, or segment to keep the rest of the network operating or choose to acknowledge the alarm and continue operation. How to Manage Your Network with SPECTRUM Page 118 Document 1909 SPECTRUM Intelligence SPECTRUM comes with the patented Inductive Modeling Technology™ (IMT), which consists of a suite of intelligence circuits that work with the Virtual Network Machine to help configure, manage, and monitor your network. This section describes the functionality provided by SPECTRUM’s Inductive Modeling Technology. In This Section Static Configuration of Device Models [page 119] Dynamic Configuration of Device Models [page 120] Condition vs. Rollup Condition [page 122] Fault Isolation [page 131] Duplicate Addresses [page 141] Automatic Naming and Addressing in SPECTRUM [page 143] Monitor Points [page 144] Detection of Firmware Problems [page 151] Device Threshold Events [page 152] Redundancy [page 152] Alternate Path Repeater Management [page 156] Device Lost and Found [page 156] Device Type Verification [page 156] In-Band Configuration of Device Alarms [page 158] Interface Intelligence [page 158] Static Configuration of Device Models Many network devices are configured during their manufacturing process and are difficult to modify later. For SPECTRUM modeling purposes, these devices are considered to have a static configuration—i.e., once SPECTRUM How to Manage Your Network with SPECTRUM Page 119 Document 1909 intelligence models these devices, they are not reconfigured. SPECTRUM creates models for the ports, matching the types of port models to the types of ports on the device (such as T1 or Ethernet). For each port model that is created, an association is established between the port and the device via the “HASPART” relation and may be typically viewed with the device model’s Application view. When the model is destroyed, all of the port models associated with the device are also destroyed. Dynamic Configuration of Device Models Some network devices can be configured dynamically by removing boards and installing new boards without removing the device from the network. SPECTRUM intelligence provides for automatic modeling of these devices and their connections upon their creation and then performs verification and remodeling if necessary after each VNM polling cycle. Thus SPECTRUM continuously monitors and changes these models to match the actual device on the network. To view the configuration of the model, use the Device, Device Topology, or Application View for the model. Whenever a model is created, SpectroSERVER polls the device and creates an appropriate configuration, including the number, type, and order of modules and ports. For each device model created, a relation exists between that model and the parent device via the “HASPART” model type relation rule. This relation also exists between the boards and any ports on the boards. After each polling cycle, SpectroSERVER re-examines and, if necessary, changes the parent model and its related models to match changes to the device configuration. Whenever you add a new board to a dynamically configured device, SPECTRUM creates a model to represent that board and each of the ports on the board. If a board model is destroyed, all of the port models that form part of the board are also destroyed. The Pulled Board List Creating and destroying models is time consuming. When you remove a board from the device, SPECTRUM does not destroy the board model, but rather keeps a copy of the board model in a “pulled board list.” The board model’s “HASPART” relation to the hub model is removed. SPECTRUM reassociates this model to the device if you re-install the old board. If you add a new board to replace the old board, SPECTRUM associates the new board model to the device and the old board model is placed in the Lost and Found view. If you remove a board model from the pulled board list, How to Manage Your Network with SPECTRUM Page 120 Document 1909 the board model is destroyed from the Lost and Found view and no longer exists. Here is a general list of pulled board list attributes: • Max_Pulled_Bd_Cnt The maximum number of models allowed to exist in the pulled board list. When the value is exceeded, the oldest model in the list is removed from the list. • Pulled_Bd_Cnt The current number of models in the pulled board list. • Pulled_Bd_List The Pulled_Bd_List is not readable by the user. It contains a list of board models that have been pulled from a dynamically configured device. When a board is re-installed in the device, SPECTRUM removes the board model from the Pulled_Bd_List. Attributes for the pulled board list are usually displayed in the Configuration view. If they are not present there, they can be added using the Generic Information Block (GIB) Editor. To change the maximum number of models allowed in the pulled board list, use the Update feature. GIB Editor features are described in the GIB Editor User’s Guide (0660). Router Reconfiguration Events Router reconfiguration actually involves two separate processes: interface reconfiguration (which ensures the device’s interfaces are properly modeled), and device discovery (which ensures proper modeling of other devices, LANs, etc. that are connected to those interfaces). Depending on whether both or either one of these processes occurs, SPECTRUM generates one of the following events to help you keep track of the configuration changes: ROUTER_RECONFIG_EVENT (0x1001c) - This event is generated when a device is reconfigured and both interface reconfiguration and device discovery occur. INTERFACE_RECONFIG_EVENT (0x1001d) - This event is generated for a device whenever interface reconfiguration occurs. DEVICE_DISCOVERY_EVENT (0x1001e) - This event is generated for a device whenever device discovery occurs (i.e., connections off the device’s interfaces are being rediscovered). How to Manage Your Network with SPECTRUM Page 121 Document 1909 Condition vs. Rollup Condition SPECTRUM provides intelligence circuits that allow you to see changes in your network devices and their performance by simply glancing at the icons that represent them. The icons use color to indicate two different types of status: Condition and Rollup Condition. Condition reflects the contact and alarm status of the modeled device represented by the icon (see Table 8: Contact Status vs. Condition Color [page 123]). Rollup Condition (Table 9: Rollup Condition Colors [page 125]) is the composite status of models that are “children” of the model represented by the icon. (Child models are related to parent models through the “collects” relation in the Topology hierarchy and through the “contains” relation in the Location hierarchy.) The Rollup Condition generally changes as you move up in the hierarchy, because at each level it reflects the blending of the Rollup Conditions from a greater number of individual models. The location of the Condition and Rollup Condition colors varies according to the type of icon, as shown in Figure 59: Condition and Rollup Condition Colors on Icons [page 122]. For device and topology (LAN) icons, Condition is displayed in the diagnostic double-click zone and the Rollup Condition is displayed in the down-arrow double-click zone for the icon. The circle at the base of location model icons displays either the Condition or the Rollup Condition, whichever is more critical. Figure 59: Condition and Rollup Condition Colors on Icons Condition Color Location Flashing Condition colors indicate a change in condition. Model Name Model Name Model Name Model Name Model Type Model Type Model Type Rollup Condition Color When a model goes into an alarm state (yellow, orange, or red) its condition indicator flashes (unless you have changed the default setting of SpectroGRAPH’s iconBlink resource, which is “True”—see Defining SPECTRUM Resources (2559) for more information on the iconBlink How to Manage Your Network with SPECTRUM Page 122 Document 1909 resource). You can acknowledge the alarm, and stop it from flashing, by selecting Acknowledge from the model’s Icon Subviews menu. When an alarm automatically clears itself, it is logged in the Event view and the condition indicator does not flash. If you need it to flash to indicate that it has cleared an alarm, select Flash Green Enabled from the model’s Icon Subviews menu. The model will then flash green when the alarm automatically clears, making it obvious that the model transitioned from an alarm state to a normal state without any user interaction. Alarm acknowledgements roll up to container models just as other status changes do, but only if there is at least one other model with no alarms in the same container. Note that the rolled-up alarm on the container will only clear once you have acknowledged a sufficient number of models within the container to add up to the threshold for the indicated color. In other words, if your Red_Threshold is 10, and you only acknowledge one red model whose Value_When_Red is 7, the container will continue to flash. If you acknowledge a second red alarm, the flashing would stop. All external models have at least one polled attribute, which is read by SpectroSERVER, which then reads all polled attributes at the frequency specified for the polling interval. For information on adjusting or disabling polling, see Adjusting Management Capacity [page 32]. Attributes Determining Condition and Rollup Condition There is a unique set of attributes that are related to Condition and Rollup Condition. Their values are used in determining Condition and Rollup Condition for models: Condition The Condition attribute value reflects the contact status as well as any more specific alarm in effect for a device model. This value determines the Condition color on topology and location icons as explained in Table 8. Table 8: Contact Status vs. Condition Color Contact Status Condition Color Description Initial Initial BLUE Either the model has not yet established contact with the device it represents, or it represents an insignificant device with which contact has been lost. How to Manage Your Network with SPECTRUM Page 123 Document 1909 Contact Status Condition Color Description Established Normal GREEN The model has successfully established contact with the device it represents, and the device is functioning normally. Established Minor YELLOW This is the first level of marginal operation. Either the model has successfully established contact with the device it represents, but there is an abnormal condition that does not affect overall network operation (e.g., a module has been removed from the device), or the IP address assigned to this model was already assigned to another model Lost Major ORANGE This is the second level of marginal operation. The management agent on the device has failed and is not responding to any communication from SPECTRUM, but the device is still relaying data to its downstream neighbors. This condition occurs only on data-relay type devices such as hubs and is typical of a firmware failure. Lost Critical RED This condition indicates a total failure of the device and requires management’s attention to repair or replace it. Lost Suppressed (Unknown) GRAY Contact has been lost with this device AND with a device that is upstream from this device (i.e., between this device and SPECTRUM), thus the actual condition of this device is unknown and alarms for the model representing it are suppressed. The gray condition color will also be displayed for all models that are downstream from this device. All adjacent (directly connected) models, whether upstream or downstream, will have a contact status of “Lost.” Lost Maintenance BROWN SPECTRUM cannot contact the device because the model has been placed into maintenance mode (see Configuring Maintenance Mode [page 68]). How to Manage Your Network with SPECTRUM Page 124 Document 1909 Condition_Value The Condition Value is a numeric value that represents a model’s overall condition. This value is passed up to a parent model and included in the Composite Condition. The model’s overall condition is either the Condition or the Rollup Condition, whichever is more severe. (The Condition Value indirectly receives the value of the administrator-defined Significance Level corresponding to a model’s overall condition.) Composite_Condition The sum of all Condition Values for models that are contained by a location model or collected by a topology model. Rollup Condition SPECTRUM computes the Rollup Condition attribute value using the administrator-defined Rollup Threshold and Composite Condition. The resulting attribute value determines the color that will be displayed to indicate the overall condition of models that are contained by a location model or collected by a topology model. The Rollup Condition for a model is displayed as a color on an icon, or the dot in a Location View icon. If the iconBlink resource is set to TRUE, the condition color flashes. Refer to Defining SPECTRUM Resources (2559) for more information on the iconBlink resource. The possible colors are described in Table 9. Table 9: Rollup Condition Colors Color Meaning Green The value of the Composite Condition attribute for this model’s children is less than the Yellow (Rollup Condition) Threshold. Yellow The Composite Condition Value for this model’s children equals or exceeds the Yellow Threshold but is less than the Orange Threshold. Orange The Composite Condition Value for this model’s children equals or exceeds the Orange Threshold but is less than the Red Threshold. Red The Composite Condition Value for this model’s children equals or exceeds the Red Threshold. Condition and Rollup Condition Sensitivity The following two attributes serve as parameters that can be used to emphasize or diminish the impact on Rollup Condition from the Condition Values of particular models. By adjusting these attribute values you can How to Manage Your Network with SPECTRUM Page 125 Document 1909 control when the Rollup Condition color changes. These attributes are typically found in the Information (GIB) View for a model. The values for these attributes can be changed by the network administrator by updating the value shown in each field. Rollup Thresholds The Rollup Thresholds are the three attributes that control the Rollup Condition color (Yellow, Orange, and Red) for a model. Rollup Thresholds are administrator-defined values that are entered on a model-by-model basis. The Composite Condition value received from the model’s children is compared with these attributes to determine a Rollup Condition Color. For example, if a model’s Composite Condition value is equal to or greater than its Orange Threshold (but less than its Red Threshold), the model’s Rollup Condition Color is Orange. The default values for Rollup Thresholds are: Yellow Threshold=3 Orange Threshold=6 Red Threshold=10 Significance Level The Significance Level attributes define the numeric value for Yellow, Orange, and Red Conditions and Rollup Conditions. Like Rollup Thresholds, Significance Level values are administrator-defined values, and are entered on a model-by-model basis. Significance Level field labels begin with the words “value when.” The default Significance Level values are: Value_When_Yellow=1 Value_When_Orange=3 Value_When_Red=7 Typically, models (devices) are divided into two classes, “significant” or “insignificant.” A significant device is any device that requires an administrator’s attention for proper network operation. Insignificant devices are typically end-point devices, such as a PC or workstation. Insignificant devices usually toggle between Green and Blue (Condition Value = 0). Significant models can be made insignificant by changing their Value_When_Red attribute value to 0 (zero). Rollup Condition Flow An overview of the Rollup Condition process is illustrated in Figure 60. Read the flow from bottom to top, but keep in mind that it shows a single How to Manage Your Network with SPECTRUM Page 126 Document 1909 path in the propagation of Rollup Condition and that there may be many children passing Condition Values to a parent model. Figure 60: Rollup Condition Process To Parent model Determine the Rollup Condition. Compare the Composite Condition to Rollup Thresholds. Set the Rollup Condition Color on the icon. Obtain the Significance Level for the current Condition of the device. Set the Condition Value to the Significance Level. Yes Use the Condition Values from children to calculate the Composite Condition Determine the Condition for the device. Set the Condition Color on the icon. Condition more severe than Rollup Condition ? No Obtain the Significance level for the current Rollup Condition. Condition Values from other children Determine the Rollup Condition. Compare the Composite Condition to Rollup Thresholds. Set the Rollup Condition Color on the icon. Obtain the Significance Level for the current Condition of the device. Set the Condition Value to the Significance Level. Yes Use the Condition Values from children to calculate the Composite Condition Determine the Condition for the device. Set the Condition Color on the icon. Condition more severe than Rollup Condition ? No Obtain the Significance level for the current Rollup Condition. Condition Values from children of this model Example of Rollup Condition Propagation Figure 61 illustrates the propagation of a Rollup Condition in the Topology hierarchy. How to Manage Your Network with SPECTRUM Page 127 Document 1909 Figure 61: Rollup Condition Process 6. Determine the Rollup Condition. Compare the Composite Condition to Rollup Thresholds. Yellow Threshold= 3 Orange Threshold=6 Red Threshold=10 7. Set the Rollup Condition color to Orange on the icon (Composite Condition > Orange Threshold but < Red Threshold). Rollup Condition: OVERALL_ NET Condition: Overall Net Orange 1 5. Calculate the Composite Condition (sum of the Condition Values from children = 8). 4. Apply the Significance level for the current Rollup Condition to set the Condition Value. (7) No Condition more severe than Rollup Condition ? Green Network Condition Values: 8= 7+1+0 Rollup Condition (Red) is more severe than Condition (Green). Yes Rollup Conditions: Yellow Red FINANCE ENGRG TWLAN TW LAN Engrg LAN Finance LAN LAN_802_3 LAN_802_3 LAN_802_3 Conditions: Green 3. Set the Rollup Condition color to Red on the icon (Composite Condition > Red Threshold) Green (No Color) Green Green Condition Values: 2. Determine the Rollup Condition Compare the Composite Condition to Rollup Thresholds. Yellow Threshold= 3 Orange Threshold=6 Red Threshold=10 1. Calculate the Composite Condition (sum of the Condition Values from children = 11). 11 = 7 + 0 + 1 + 3 + 0 Rollup Conditions: Red Green (No Color) Conditions: Green Green Yellow Orange Green Green Green (No Color) Green This example depicts two layers of a Topology hierarchy. The example assumes the use of default Rollup Thresholds and Significance Levels. At the lowest level in the figure there are five devices: two hubs, a router, and two end-point devices. These are contained by TWLAN, a LAN of type How to Manage Your Network with SPECTRUM Page 128 Document 1909 802_3_LAN (as are the two other LANs, FINANCE and ENGRG). The network group model named OVERALL_NET collects these three LAN models. The following Rollup Conditions and Conditions, at lower levels, determine the top-level Rollup Condition for the network group model OVERALL_NET: Devices collected by TWLAN Hub#1 Condition = Green Rollup Condition = Red Condition Value = 7 Hub#2 Condition = Green Rollup Condition = Orange Condition Value = 3 Router#1 Condition = Green Rollup Condition = Yellow Condition Value = 1 End-Point Devices PC#1 & PC#2 Condition = Green Rollup Condition = Green Condition Value = 0 LAN named FINANCE Condition = Green Rollup Condition = Green Condition Value = 0 LAN named ENGRG Condition = Green Rollup Condition =Yellow Condition Value = 1 Every model within a Topology View receives its Collects relation from the model that it is collected by. Therefore, all models contribute to the Rollup Condition of the network group model. The following steps provide a detailed flow of condition values that contribute to the Rollup Condition for the model named OVERALL _NET shown in the example of Figure 61. How to Manage Your Network with SPECTRUM Page 129 Document 1909 1. Determine the Composite Condition for the TWLAN network group model. Composite Condition is the sum of the collected models’ Condition Values. In this case, the device models have Condition Values of: “Orange” Condition hub model has a Condition Value of 3. “Red” Condition hub model has a Condition Value of 7. “Green” Condition PC#1 model has a Condition Value of 0. “Yellow” Condition router model has a Condition Value of 1. “Green” Condition PC#2 model has a Condition Value of 0. Therefore, for the TWLAN model: Composite Condition = (3 + 7 + 0 + 1 + 0) = 11 2. Determine the Rollup Condition for TWLAN. In this case: Composite Value = 11 TWLAN Yellow Threshold = 3 TWLAN Orange Threshold = 6 TWLAN Red Threshold = 10 Composite Value > Red Threshold Therefore: Rollup Condition for TWLAN = Red 3. Assign Significance Levels to TWLAN Condition and Rollup Condition. In this case, Significance Levels are: Value When Yellow = 1 Value When Orange = 3 Value When Red = 7 Therefore: Rollup Condition = Red Condition = 7 4. Set Condition Value for TWLAN model. In this case: Rollup Condition more severe than Condition Therefore: Condition Value = Rollup Condition Significance Level = 7 The three network models TWLAN, ENGRG, and FINANCE pass their Condition Values up to the network group model OVERALL_NET. Changes in the Condition or Rollup Condition for the device models at lower levels can produce changes in the topology models further up in How to Manage Your Network with SPECTRUM Page 130 Document 1909 the Topology hierarchy. The Rollup Conditions for these three networks produce the following Rollup Condition for OVERALL_NET. 5. Determine the Composite Condition for the OVERALL_NET network group model. Composite Condition is the sum of the collected models’ Condition Values. In this case, the network models have Condition Values of: Red Condition TWLAN model has a Condition Value of 7. Green Condition FINANCE model has a Condition Value of 0. Yellow Condition ENGRG model has a Condition Value of 1. Therefore, for the TWLAN model: Composite Condition = (7 + 0 + 1) = 8 6. Determine the Rollup Condition for OVERALL_NET. In this case: Composite Value = 8 Yellow Threshold = 3 Orange Threshold = 6 Red Threshold = 10 Composite Value > Orange Threshold, but < Red Threshold Therefore: Rollup Condition for OVERALL_NET = Orange Organization Models Unlike device, location, and network group icons that indicate status using colors, icons that are unique to the Organization hierarchy do not provide any visual status indication. However, device, location, and network group icons can appear in the Organization hierarchy, and when they do, they reflect the same Condition and Rollup Condition as that displayed when they appear in the location and topology hierarchies. Fault Isolation Fault Management is one of the key requirements of network management. A fault is different from an error because it is an abnormal condition that requires management attention and repair. Problems that give rise to a fault could be caused by bad firmware, bad hardware, or a bad network. Each of these problems requires a different response from the network manager. Thus the goal is to determine the exact location of How to Manage Your Network with SPECTRUM Page 131 Document 1909 the fault and to get the attention of the network administrators as quickly as possible. SPECTRUM intelligence has the capability of isolating a network problem to the most probable faulty component. To speed up fault isolation and to reduce unnecessary traffic, two actions occur: Are-You-Down Action Upon losing contact with the device it represents, a model sends the Are-You-Down action to all of its neighbors to determine its own condition. If all of the neighbors return a response of TRUE, the model’s condition color will turn gray (meaning “my device might be down, but it is impossible to tell because all the neighbors are down.”). However, if any of the neighbors return a response of FALSE, the model’s condition color will turn red (meaning “my device must be down, because one of the neighbors is up.”). Are-You-Up Action Upon re-establishing contact with the device it represents, a model sends the Are-You-Up action to its neighbors to speed up the fault isolation. Upon receiving this action, each neighbor will return TRUE if it has an established contact status. If the model’s contact status is lost, and the next-time-to-poll is more than 60 seconds, then the model pings the device for quicker fault isolation. Every time a model’s status changes, or the information available to SPECTRUM changes, a new assessment occurs. SPECTRUM intelligence keeps the SpectroGRAPH presentation as up to date and accurate as possible, but it depends on correct modeling to accurately assess the contact status and determine device failures on the network. Correct modeling includes placing your VNM model in proper relation to the other models that represent your network (i.e., it must have a resolved connection in the Device Topology view of a model that represents a device to which the VNM host is actually connected). When the VNM model is properly connected and SPECTRUM loses contact with a model, the icon representing that model will display a condition color of Gray, Orange, or Red, which helps the network administrators to locate the faults immediately. How Model Category Affects Contact Status Each fault is associated with a particular condition, which is represented by a particular color that displays on the icon representing the model where the fault occurs. The condition color reflects both the contact status and How to Manage Your Network with SPECTRUM Page 132 Document 1909 the alarm status of the model, as was shown in Table 8. However, the contact status and condition color asserted for a model also depend upon which of the following categories a model belongs to. Table 10 summarizes how the categories to which a model and its neighbors belong will influence its contact status and condition color. Significant Device Models Any device that requires an administrator’s attention for the smooth operation of the network is called a significant device. To change an insignificant model into a significant model change the value of the attribute Value_When_Red (0x01000e) to Seven. Insignificant Device Models An insignificant device (e.g., end user PC) toggles between Blue and Green contact states and does not generate alarms or event messages to get the attention of the administrator. To change a significant model into an insignificant model change the value of the attribute Value_When_Red (0x01000e) to Zero. Inferred Connectors These are dumb models that do not poll, but keep track of a list of their Data Relay neighbors. Possible inferred connectors are: WA_Segment, Fanout, etc. SPECTRUM will automatically enable Live Pipes for all ports connected to a WA_Segment. Note: SPECTRUM intelligence does not expect Fanout models to be connected to each other, thus this configuration will result in inaccurate contact status displays. If two Fanouts are connected to each other and each of them is in turn connected to a device with a contact status of green, the fanouts will nonetheless turn red. If two fanouts are connected to each other with no other devices connected to either, both fanouts will turn gray. Composite and Discrete Topology Models The contact status of LAN, LAN 802.3, LAN 802.5 etc. models is determined by the contact status of its collected children. A LAN model with lost contact status will turn either Red or Grey, depending on the condition of its collected models. How to Manage Your Network with SPECTRUM Page 133 Document 1909 Wide Area Links In SPECTRUM, Wide Area Links (WA_Links) are modeled in conjunction with fanout or wide area segment (WA_Segment) models (see Note below). This allows for proper roll-up of the Wide Area Link condition. For instance, in using a fanout, devices at either end of the WA_Link are connected to the fanout model. You can view this by navigating into the fanout’s cablewalk view. In turn, the fanout model must be connected to the appropriate port of the device model. This cross-connection is very important for fault isolation to work. WA_Link models work best when modeling point-to-point types of Wide Area connections (such as T1 and T3 lines). Note: WA_Link models can accommodate only one WA_Segment model. If you attempt to paste more than one WA_Segment model into a WA_Link model’s Topology view, the second one will be destroyed immediately and an alarm will be generated. Figure 62: A Typical Wide Area Network Modeled in SPECTRUM: VNM Rtr1 Rtr1 WA_Link WA_Segment Rtr2 Rtr2 Wide Area Segments WA_Segments poll the InternalPortLinkStatus (IPLS) attribute of each interface model which Connects_To the WA_Segment. This is an active poll, meaning that the IPLS of each connected interface is read at every polling interval rather than simply watched for a change in the attribute. Therefore, SPECTRUM does not have to lose contact with one of the connected routers in order for a Fault Isolation alarm to be generated on a WA_Link. The polling of the connected ports’ IPLS will be regulated by the WAL_Link model’s Polling_Interval and Polling_Status attributes. When the How to Manage Your Network with SPECTRUM Page 134 Document 1909 Polling_Interval changes to zero (0) or Polling_Status goes to FALSE, polling of the connected port’s IPLS is stopped. If one of the connected interfaces has an IPLS of BAD (for example, Admin Status is ON, but Oper Status is OFF), then the WA_Segment’s Contact_Status is set to LOST and the WA_Segment will go GRAY. The WA_Link will go RED. If one of the connected interfaces has an IPLS of DISABLED (for example, Admin Status is OFF), then the WA_Segment’s Contact_Status is set to LOST and the WA_Segment will go GRAY. The WA_Link will go ORANGE. This is because the alarm must be severe enough to be viewed in the Alarm Manager, but it is not a “Contact Lost” alarm. If the DISABLED interface causes SPECTRUM to lose contact with the remote router, then the WA_Link will go RED. This is the regular InferConnector-type Fault Isolation working. Table 10: Model Category vs. Condition Color Model Category Model’s Neighbors (connected Condition Color models) Significant Devices connected to a VNM... (Modeling Hub-types only.) turn Red after losing contact. Significant Devices with no connections to other models (a zero connector count)... Significant Devices connected to an established Data Relay neighbor... Composite and Discrete Topologies in which all of the collected children have a lost contact status and at least one of those collected children is Red... Inferred Connectors where the fanout model has lost contact but one of its neighbors is good and the associated port has bad port link status, then it... Significant Devices, Inferred Connectors, and WA_Links where all neighbors have also lost turn Gray after losing contact status... contact. Composite and Discrete Topologies in which all of the collected children have a lost contact status and none of those collected children are Red... How to Manage Your Network with SPECTRUM Page 135 Document 1909 Model Category Model’s Neighbors (connected Condition Color models) Significant Devices (Modeling Hub-types only.) connected to an end-point neighbor (e.g., PC) that has established contact status... WA_Links WA_Segment (or fanout) is good and one of the routers is lost then... Significant Devices connected to a model with an Established contact status... Composite/Discrete Topologies and WA_Links in which any of the collected children has established contact status, then the LAN will also... Inferred Connectors connected to a model with an established contact status where at least one of its neighbors is Good and its associated port (port connected to the Fanout) status is Good... Significant and Insignificant not yet connected to other Devices devices... Composite/Discrete Topologies and WA_Links turn Orange after losing contact. turn Green. turn Blue. when all collected children of a LAN have initial contact status, then the LAN will also have the initial contact status... Examples The following examples illustrate how SPECTRUM fault isolation operates with various network configurations and problem scenarios. Example 1 This example demonstrates that fault isolation is a pro-active mechanism, and does not depend upon polling all of the connected models. Consider a simple network topology as shown in Figure 63. The device H1 is connected to the VNM model. Devices H1, H2 and H3 poll every 3 minutes. H4 polls every 5 minutes. The PC polls every 30 minutes. How to Manage Your Network with SPECTRUM Page 136 Document 1909 Figure 63: Network Topology Example 3 H1 VNM 3 3 H3 H2 5 30 H4 PC Assume H2 is BAD. As a result H2 would turn red, H4 would turn gray, PC (insignificant model) would turn blue, while H1 and H3 should remain green. Fault isolation is initiated as soon as H2, H4 or PC polls. If H4 is lost, it sends an Are-You-Down action to H2. If H2 is lost by then, it sends TRUE to H4, otherwise it pings itself and then sends the response to H4. This causes H4 to turn gray. Now H2 is lost, and it sends Are-You-Down action to H1. Since H1 is established, H2 has to decide between orange and red conditions. H2 pings PC. Since PC can not respond H2 will turn red. The ping from H2 puts PC in a lost state. Since PC is an insignificant device it will turn blue. Note: H1 and H2 are connected by dragging H2 onto a port of H1 and H1 onto a port of H2. Example 2 This example demonstrates fault isolation when modeling a fanout. Assume fanout is red, D2 and D3 are gray, and then D3 polls successfully. Figure 64 depicts this scenario. How to Manage Your Network with SPECTRUM Page 137 Document 1909 Figure 64: Network Topology with a Fanout 2 VNM D1 P1 Fanout 4 P2 D2 4 P3 D3 30 PC D3 will have an established contact status and turn Green. Then D3 will send an Are-You-Up action to the fanout. The fanout reads device P3’s (D3’s port connection to the fanout) internal link port status. Assuming the port has good status the fanout will turn green with established contact status. This means that as long as P1 (D1’s port connection to the fanout) has good internal link port status, the contact status of the inferred connector will remain good. What if: D2 goes bad? D2 will lose its contact status and sends an Are-You-Down action to the Fanout. The Fanout will ping D1, and finds D1 to be good. The intelligence then examines the status of P1. Assuming Link-Status of P1 is good, the Fanout will return FALSE to model D2. This causes D2 to turn Red. What if: P1 is bad? This is the same case as disconnecting the network connection to the Fanout. If D3 polls first, it will lose its contact status and send an Are-YouDown action to the Fanout. The Fanout will ping D1 as finds it as a good neighbor. Fanout then reads the internal-port-link-status of the port P1. Since P1 is bad, the Fanout will lose its contact status and turns Red. The Fanout will return TRUE to the model D3. This causes D3 to turn Gray. D2 will also turn Gray in the same way as D3. PC being the insignificant device will turn Blue immediately after losing its contact status. How to Manage Your Network with SPECTRUM Page 138 Document 1909 Example 3 This example shows how SPECTRUM manages devices using redundant paths if a link is shut down administratively (i.e., admin-status equals down). Figure 65 depicts a network with redundant WA Links. Here VNM manages Rtr3 through link WL-1 and Rtr2 using link WL-2. Now let us assume that the network administrator shuts down the WL-1 link. This causes WL-1 to turn gray. Rtr3 will turn red because VNM can not talk to it through WL-1. The redundancy intelligence of Rtr3 will modify its agent address, so that VNM can talk to it using links WL-2 and WL-3. This causes Rtr3 to turn green again. The link WL-1 will still have the gray condition. Figure 65: Redundant Wide Area Links New York VNM WL-1 Rtr1 London Rtr3 WL-2 WL-3 Rtr2 San Francisco Example 4 This example demonstrates that fault isolation for an Inferred Connector requires specific modeling. Assume that two routing devices, Rtr1 and Rtr2, are connected at both ends of the WA_Link and that their ports are P1 and P2 respectively. WA_Link models need to be associated with a WA_Segment (or fanout) model through the Collects relation to enable the proper roll-up of the WA_Link condition. The devices at either end of the WA_Link need to be connected to the WA_Segment collected by the WA_Link model. You do this by navigating into the device’s Device Topology view and resolving the WA_Segment off-page reference icon to the appropriate port. You can view How to Manage Your Network with SPECTRUM Page 139 Document 1909 the connections by navigating into the WA_Segment’s Cablewalk view. This cross-connection is very important for fault isolation to work, as shown in Figure 66. Figure 66: Network Topology P2 P1 Rtr1 VNM Rtr2 WA_Link Rtr1 WA_Segment Rtr2 WA_Link Assume P1 is the port on Rtr1 and P2 is the port on Rtr2. The routers connected to the WA_Segment will cause it to behave as described in the following table. Note that the port link status becomes important in determining the status of the WA_Link only when both Routers are “contact established.” Table 11: WA_Segment Conditions Rtr1 Rtr2 WA_Link initial initial blue established lost red lost lost gray established established Check Port States (See Note below). How to Manage Your Network with SPECTRUM Page 140 Document 1909 Note: If both Rtr1 and Rtr2 have a contact status of established then the port status of P1 and P2 will determine the condition of the WA_Link. If any port is BAD, the WA_Link will be RED. If any port is DISABLED, the WA_Link will be ORANGE. Otherwise, the WA_Link will be GREEN. Duplicate Addresses SPECTRUM intelligence automatically detects when duplicate Internet Protocol (IP) addresses are entered in the SpectroSERVER database. Although some devices are allowed to have a duplicate IP address, Cabletron hub devices should be configured with only one IP address per device. Alarm condition colors warn you of duplication, as shown in Table 12 [page 142]. Note: Beginning with SPECTRUM version 6.6, the policy for modeling duplicate IPs in SPECTRUM has changed in the following way. SPECTRUM is now capable of modeling different devices that share IPs, provided that each device has at least one IP that is unique to that device. This was done to accommodate certain networking technologies such as load balancing that create identical IP addresses across a range of devices. Devices that share some interface IP addresses can now be modeled manually or by using AutoDiscovery. Devices that share all of their interface IP addresses in common cannot be modeled manually or by using AutoDiscovery in SPECTRUM. How to Manage Your Network with SPECTRUM Page 141 Document 1909 Table 12: Duplicate Address Alarms and Color Status Alarm Color Same MAC Address & Different IP Address Yellow This alarm occurs when there are two or more Models with the same MAC address and at least one model with a different IP address. Same IP Address & Different MAC Address Orange This alarm occurs when there are two or more models with the same IP address and at least one model with a different MAC address. Same IP Address & Same MAC Address Yellow This alarm occurs when there are two or more models with the same IP address and the same MAC address (duplicate addresses). Duplicate MAC Address Yellow The special case alarm for duplicate models where at least one of the models does not have an IP address. Only a Physical_Address model type can have this characteristic. To get these alarms, a model type needs both the MAC address and the IP address. For example, the model types Pingable and PhysicalAddress do not have both addresses, so you will not see the above alarms. To manually clear a duplicate address: 1. Open the Enterprise Alarm View. 2. Select the model that has the duplicate IP address alarm so that its icon is visible in the Alarm View. Determine if you want to allow two devices to have the same IP address. If not, you will need to change one of the devices to a unique IP address, and then use the Update feature to change the IP address within SPECTRUM. 3. To clear the duplicate, select the Clear button in the Alarm View. The alarm is removed. Once the duplicate IP address alarm clears, the status color on the model’s icon returns to a normal green condition, unless another alarm is present HardErrorCount = TRMonHardErrorCount for the model. For detailed information on the Enterprise Alarm Manager, ( HardErrorCou n t = TRMonHardErrorCount ) How to Manage Your Network with Document 1909 SPECTRUM Page 142 refer to the Enterprise Alarm Manager User’s Guide (2065). For more information on the Update feature, refer to the GIB Editor User’s Guide (0660). Automatic Naming and Addressing in SPECTRUM SPECTRUM implements an automatic model naming and addressing feature through the AUTO_NAME attribute (attribute ID 0x00011979). The value for this attribute is set to TRUE by default for each model type in your modeling catalog. You can disable automatic naming and addressing on a model type basis using the Model Type Editor (MTE) to set the value to FALSE. Otherwise, the feature functions as described below. If you create a new model using only the IP address, SPECTRUM automatically attempts to supply a name for the model in one of three ways: • Using NIS (Network Information System) or DNS (Domain Naming Service) to get the name from the modeled device • Checking the local /etc/hosts file for the name associated with the modeled device’s IP address • Using the IP address as the model name Note: The priority order of the source that will be used to supply a name for the model (IP Address, Name Service, or sysName) is dictated by the Model_Naming_Options attribute on the VNM model. This attribute can be modified on the VNM model's control view. See the Distributed SpectroSERVER User Guide (2770) for more information on configuring landscapes. Likewise, if you create a new model and supply only a model name, SPECTRUM will attempt to use NIS, DNS, or the local /etc/hosts file to retrieve the modeled device’s IP address. In either case, as long as the value for AutoName is TRUE for a particular model type, SPECTRUM will automatically maintain the names for models of that model type as follows: in the event the IP address for a model changes and the original model name was supplied by SPECTRUM, a new name will be supplied using one of the three methods listed above. However, if the original name was supplied by the user and differs from the How to Manage Your Network with SPECTRUM Page 143 Document 1909 name that would be supplied through automatic naming, then the original name will be preserved. Board and port models are also automatically named by default, each being assigned the name of the parent device model suffixed with the board/port Instance ID. For example, if a model of model type Hub_CSI_IRM2 is named IRM2_UK, and the modeled device has a port with the Instance ID of 2.5, the name of the port will be IRM2_UK.2.5. If the device name is changed to IRM2_US, then the name of the port becomes IRM2_US.2.5. However, if the device name was user-specified and the user then changes the port name from IRM2_US.2.5 to LAB_PORT, then the automatic naming will not be not used for that port in the event of subsequent IP address changes. Some boards (mainly standalone MIMs) contain their own intelligence. In such cases, setting AUTO_NAME to FALSE as previously described will disable the autonaming intelligence and allow the board’s own intelligence to work. Monitor Points Network group models (FDDI, LAN_802_3, or LAN_802_5) are container models that represent a logical collection of actual devices and/or other network groups. A Monitor Point is a specific device within a network group model that provides the statistics that are used to calculate network activity in the network group model’s Performance View. Statistics, such as Packet Rate and Bits Per Second, are gathered by a Monitor Point and used to calculate total network activity. The specific Monitor Point device is selected by SPECTRUM based on the value of its Monitor Precedence attribute (0x11a44). Within a network, the device model having the highest Monitor Precedence value becomes the monitor point for that network group. If more than one device has the same Monitor Precedence value, the first device seen is chosen. Typically, the highest precedence is for network analyzing devices, followed by successively less intelligent devices. (Default precedence values can be altered by updating them in a Network’s Information View.) Monitor Point Statistics The collection, rollup and calculation of monitor point statistics is determined by two base model types: Composite (model type handle = 0x1002c) and Discrete (model type handle = 0x102e1). Each of these model types provide the intelligence (inference handlers) that produce a How to Manage Your Network with SPECTRUM Page 144 Document 1909 specific set of monitor point statistics. Model types that inherit these model types, inherit the ability to gather and calculate the monitor point statistics. Figure 67 shows the monitor point model type hierarchy and an example of model type inheritance by network group model types. Figure 67: Monitor Point Model Type Hierarchy Entity_Types Composite Network Topology Discrete LAN_Types LAN_802_3 Collects Monitors HUB HUB LAN_802_3 PC Lantern PC HUB Potential Monitor Points In general, monitor point calculations for model types derived from the Discrete model type consist of reading attribute values from a Monitor Point device model, then using those values to establish the attribute values of a Discrete network group model (WA_Link, FDDI, LAN_802_3, LAN_802_5). Discrete network group models (derived from the Discrete model type) are, in turn, collected by a composite network group model (derived from the Composite model type). Composite model types use the statistics gathered in the Discrete model types to calculate their own statistics. Composite Monitor Point Statistics Five model types are derived from the Composite model type: • BroadBand_Net • LAN How to Manage Your Network with SPECTRUM Page 145 Document 1909 • Network • Universe • WAN The Composite model type calculates its monitor point statistics by comparing or summing attribute values from each of the models that it COLLECTS. Four attributes are calculated by finding the corresponding maximum value in each of the collected models: Load Loadx100 PacketRate SoftErrorRate One attribute is found by summing the values of the corresponding attribute from each of the collected models: Hard_Error_Count Discrete Monitor Point Statistics Four network group model types are derived from the Discrete model type: • WA_Link • LAN_802_3 • LAN_802_5 • FDDI These model types use a more sophisticated calculation for monitor point statistics, and each calculates the statistics in a slightly different way. In the following tables, the calculations performed by a model type’s inference handlers to are shown as equations. The values on the left are the attributes of the network group model type level. The values on the right are values read from the Monitor Point device model. The calculation of Load is multiplied by a factor of 100 to show Load as a percentage. In certain models, where Load is a relatively small value, the Loadx100 attribute is used to obtain a better graphical presentation of the actual value. Loadx100 multiplies the value of Load by another factor of 100. WA_Link (model type handle = 0x102e2) The Wide Area Link (WA_Link) model type is used to model a connection between the interfaces of two connected routers. The monitor point calculations are computed using one of the two interfaces that are How to Manage Your Network with SPECTRUM Page 146 Document 1909 connected to this model. Figure 68 shows how the interfaces are connected to the WA_Link model and illustrates the MONITORS association. Figure 68: WA_Link Relations WA_link Router 1 Interface Router 2 Interface fanout Monitors Connects Collects HASPART The monitor point calculations that are performed for this model use information calculated from the data_relay_prt model type. The data_relay_prt model type is the base model type for router interfaces. WA_Bandwith is calculated by an inference handler attached to the WA_Link and is set to the if_speed of the data_relay_prt model. If this value is zero, then the default of 1540000 (T1 interface) is used. Table 13 shows monitor point calculations for the WA_Link. Table 13: WA_Link Monitor Point Calculations Attribute = Formula Attribute = Formula WAMonMaxBitsUsedPerSecIn × 100 × 100 Loadx100 = -----------------------------------------------------------------------------------------------------WABandwidth WAMonMaxBitsPerSecIn × 100 Load = --------------------------------------------------------------------------WABandwidth SoftErrorRateIn = WAMonSoftErrorRateIn PacketRateOut = WAMonPacketRateOut How to Manage Your Network with SPECTRUM Page 147 Document 1909 Attribute = Formula Attribute = Formula WAMonBitsUsedPerSecOut × 100 LoadOut = --------------------------------------------------------------------------------WABandwidth Lan_MP_SysUpTime = MonSysUptime WAMonBitsUsedPerSecIn × 100 LoadIn = ----------------------------------------------------------------------------WABandwidth SoftErrorRate = WAMon_Max_SoftError_Rate PacketRateIn = WAMonPacketRateIn PacketRate = WAMon_Max_Packet_Rate SoftErrorRateOut = WAMonSoftErrorRateOut WAMonBitsUsedPerSecOut × 100 LoadOut = --------------------------------------------------------------------------------WABandwidth Lan_MP_SysUpTime = MonSysUptime PacketRateIn = WAMonPacketRateIn WAMonBitsUsedPerSecIn × 100 LoadIn = ----------------------------------------------------------------------------WABandwidth SoftErrorRate = WAMon_Max_SoftError_Rate PacketRate = WAMon_Max_Packet_Rate How to Manage Your Network with SPECTRUM Page 148 Document 1909 Attribute = Formula Attribute = Formula SoftErrorRateOut = WAMonSoftErrorRateOut FDDI (model type handle = 0x10032) The FDDI model type is used to model an FDDI network. The monitor point calculations for this model type are computed using the model that is associated to the LAN model with the MONITORS relation. The following table shows the monitor point calculations performed for this model type. Table 14: FDDI Monitor Point Calculations Attribute = Formula Attribute = Formula Load = UtilizationRate LAN_MP_SysUpTime = Uptime Packet_Rate = FrameRate RingStatus = RingStatus Soft_Error_Rate = ErrorRate Note: Loadx100 is not calculated for the FDDI model type. LAN_802_5 (model type handle = 0x1003d) The LAN_802_5 model type is used to model a Token Ring network. The monitor point calculations for this model type are computed using the model with the highest precedence that is associated to the LAN_802_5 model with the MONITORS relation. The following table shows the monitor point calculations performed for this model type. Table 15: LAN_802_5 Monitor Point Calculations Attribute = Formula Attribute = Formula LAN_Bandwidth = Mon_Bandwidth TR_Errs_Per_MFrame = TR_Mon_Errs_Per_MPacket How to Manage Your Network with SPECTRUM Page 149 Document 1909 Attribute = Formula Attribute = Formula TR_Band_Util = TR_Mon_Utilization LAN_MP_SysUpTime = Mon_Sys_Uptime TR_Bytes_Per_Second = TR_Mon_Bytes_Per_Second TR_Active_Stations = TR_Mon_Active_Stations TR_Frame_Rate = TR_Mon_Frame_Rate Soft_Error_Rate = TR_Mon_Errs_Per_MFrame TR_Ring_Speed = TR_Mon_Ring_Speed Note: Packet_Rate is not calculated for the LAN_802_5 model type. LAN_802_3 (model type handle = 0x1003c) The LAN_802_3 model type is used to model an Ethernet network. The monitor point calculations for this model type are computed using the model with the highest precedence that is associated to the LAN_802_3 model with the MONITORS relation. LAN_Bandwidth is an attribute in the LAN_802_3 model type. This attribute is set to 10,000,000. The following table shows the monitor point calculations performed for this model type. Table 16: LAN_802_3 Monitor Point Calculations Attribute = Formula Attribute = Formula CollisionRate = CollisionRate HardErrorRate = MonHardErrorRate LAN_MP_SysUpTime = Mon_Sys_Uptime PacketRate = MonPacketRate How to Manage Your Network with SPECTRUM Page 150 Document 1909 Attribute = Formula Attribute = Formula MonBitsUsedPerSec × 100 Load = ---------------------------------------------------------------MonBandwidth MonBitsUsedPerSec × 100 × 100 Loadx100 = ------------------------------------------------------------------------------MonBandwidth SoftErrorRate = MonErrorRate HardErrorCount = MonHardErrorCount Monitor Point Calculations in SpectroWATCH SpectroWATCH is an intelligent data collection system used to manage networks. Many of the monitor point calculations for SPECTRUM are now in watches in SpectroWATCH. This makes it easier to view the existing formulas and add new ones. The monitor point calculations in the preceding sections require reading attributes from other models within the database and are not yet implemented as watches in SpectroWATCH. However, in a future release, even these will be moved to SpectroWATCH, and all of the monitor point calculations will reside in watches, making them more readily accessible. Refer to the guide SpectroWATCH Operator’s Reference (0919) for a list of monitor point calculations performed in SpectroWATCH. Detection of Firmware Problems SPECTRUM allows for automatic detection of problems in some device firmware. Using the connects_to and collects model type relations formed via the Device Topology view, SPECTRUM detects if a network management firmware problem exists in the hub device. The connects_to relation denotes that one model is attached or connected to another (for example, a PC model connects to a hub model via a port on the hub). The collects relation denotes that one model collects information from another model (for example, the Topology view model collects information from the hub devices contained within that view). If SPECTRUM cannot retrieve management information from a hub device, but can still contact devices connected to this hub, then a hub firmware management problem can be deduced and the hub icon’s condition color is set to orange. You should then use the Alarm View along with the Diagnostic View and the Performance View to help isolate and correct this problem. How to Manage Your Network with SPECTRUM Page 151 Document 1909 Device Threshold Events When a device’s internal threshold values are exceeded, its firmware generates an unsolicited alert message, or trap. SPECTRUM intelligence evaluates these trap messages and generates an appropriate message in the Event Log View and/or Alarm View. Additionally, this process also sets the device icon’s condition to yellow or orange indicating a minor alarm. For information on alarms and events, refer to the SPECTRUM management module guide for the specific device. Redundancy SPECTRUM allows you to configure redundancy not only for the network management system itself, but for paths between devices. Redundant SpectroSERVERs (Fault Tolerance) SPECTRUM provides for the loss of a critical server by allowing you to have up to two backup servers. These backup SpectroSERVERs are ready to go within minutes of the loss of the main server. The backup SpectroSERVERs have the same landscape handles, but different precedences, and do not actually poll any devices on the network until they are made the primary server. Frequently scheduled on-line backups ensure the database is as close as possible to the primary server’s database. Any traps should be configured to be sent to all backup SpectroSERVERs so that in the event of the loss of the primary SpectroSERVER, traps will still be sent to the backup server as it is operating. For more information on setting up redundant SpectroSERVERs, refer to the section on fault tolerance in the manual Distributed SpectroSERVER (2770). Redundant Paths Devices that have more than one way of being reached have redundancy. If the redundancy feature is enabled on a device and it cannot be reached directly, SPECTRUM attempts to reestablish contact using the IP addresses of the device’s other interfaces (see Configuring Device Redundancy [page 153]). This provides an alternate line of communication by forging a connection over a redundant path. If SPECTRUM still cannot reach the device, its icon changes to condition red (contact lost) and the alarm is recorded in the Alarm View. How to Manage Your Network with SPECTRUM Page 152 Document 1909 Note: The redundancy feature is not supported by all device models. To determine whether a given device supports the feature, consult its SPECTRUM management module guide. Configuring Device Redundancy Redundancy is enabled by default in the Redundancy Administrative Status field. To access the field, click the Redundancy Options button in the The Fault Management View [page 65] of the modeled device. (The field can also be accessed in the Redundancy and Model Reconfiguration Options view, accessible from the device’s Configuration view.) Disabling the Redundancy feature changes the value of the RedundancyEnabled (0x00011d2c) attribute to FALSE. The default value is TRUE. Editing the Redundancy Preferred Addresses list In addition to the RedundancyEnabled (0x00011d2c) attribute, each router model in SPECTRUM also has the following attributes: • Primary Address (PA): (0x00011d2a) - This is the IP address of the interface SPECTRUM uses to communicate with the device. This address is initially used to model the device. • Agent ID or Network Address: (0x0001027f) - This is the IP address of the interface through which communication with the device is active. Initially, the network address and the PA are the same, but later may be different from the PA at a given time when redundancy determination is in progress. • Redundancy Preferred Addresses (RPA): (0x00011d2b) - This is a list of the device’s interface IP addresses and is used in determining a redundant connection. The RPA list is created when the router is first modeled. Note: The SPECTRUM modeling process includes a loopback functionality. The first valid loopback address detected for a device will be used as the value for the device model’s PrimaryAddress, Network_Address, and Model_Name attributes. To enable this functionality, you must add use_loopback=true to the .vnmrc file. See Defining SPECTRUM Resources (2559) for additional information on the .vnmrc file and its associated resources. How to Manage Your Network with SPECTRUM Page 153 Document 1909 When contact with the device is lost, redundancy is activated in two ways. One way is that the addresses in the RPA list are checked every polling interval to determine their availability. Once an address from the RPA is reached, this stops. The other way is that a timer is started and attempts to reach the PA upon its expiration. When the PA is reached, this second action stops. You can configure the router model’s Primary Address and the Redundancy Preferred Addresses list through the Preferred Addresses button, found in the Redundancy and Model Reconfiguration Options view. Click the button to display the Preferred Addresses dialog box (Figure 69). Figure 69: Preferred Addresses Dialog Box Redundancy Preferred Addresses Each IP address in this list is numbered according to the order in which it will be used by SPECTRUM. You can change the list to suit your network needs. • To add an address to the RPA list, click the Insert... button and enter the IP Address and desired position. • Highlight an address in the RPA list and click Delete to remove it from the list. • Highlight an address in the RPA list and click Move to change the order in which it is accessed. In the dialog box that appears, enter the source and destination positions for the address that you want moved. How to Manage Your Network with SPECTRUM Page 154 Document 1909 • Highlight an address in the RPA list and click Primary to make the address the Primary Address (PA). Click Save to save the modifications you make to the RPA list, or click Cancel to exit the Preferred Addresses dialog box without saving the changes. Primary Address This field displays the value of the current PA (the field is not editable). Redundancy Excluded Addresses The Redundancy Excluded Addresses list displays those IP addresses you want SPECTRUM to exclude from use in a redundant path. To add an address to the Redundancy Excluded Addresses list, highlight the address in the RPA list on the left and click the right arrow button. Or, you can click the Insert... button under the Redundancy Excluded Addresses list to insert an address at a given position in the list. Device Redundancy Events Events are logged in the Event Log when a redundancy related action occurs. They are generated when: • A Primary Address that is in use has been determined to be no longer valid and the Network Address, which is valid, becomes the new Primary Address (0x00010914). • It is determined that a preferred address in the RPA list is no longer valid and thus deleted from the RPA (0x00010914). • A given Primary Address is not reachable and does not belong to the device modeled (0x00010915). • The Network Address differs from the Primary Address (0x00010916). • A model is incapable of redundancy, which happens when the PA used to model the device can be reached but the agent on the device does not provide the necessary MIB-II information to be able to determine redundancy (0x00010917). How to Manage Your Network with SPECTRUM Page 155 Document 1909 Alternate Path Repeater Management Alternate Path Repeater Management (APRM) prevents loss of contact with repeater models caused by the activation of router redundancy. APRM is designed for devices capable of functioning as IP routers and that contain the following repeater models types: CSIRptr, FddiSMT, and MMACPlusRptr. You can enable/disable APRM from the Alternate Path Repeater Mgmt. button in the Model Information View of the repeating application. The default state for APRM is disabled. Note: As the repeater model types of bridges and other non-routing devices do not inherit router redundancy, there is no need to enable APRM for them. When contact with a device is lost and router redundancy activates, SPECTRUM attempts to contact the device through one of the model’s preferred addresses. The address used to successfully recontact the device becomes the model’s network address. When APRM is enabled for the relevant repeater models their network addresses change to match the device model’s. This insures contact with the repeaters as long as there is contact with the device. When APRM is disabled the new network address is not rolled down to the device’s repeater models, which causes SPECTRUM to lose contact with the relevant repeater models. APRM has three possible states: enabled, disabled or activated. Should the APRM state change for a repeater model, an event is generated and logged to that repeater’s Event Log. Device Lost and Found SPECTRUM has a lost/found intelligence circuit which automatically detects a device that does not relate to any Location, Topology, or Organization View. Whenever an existing device becomes unrelated to any view, the lost/found intelligence sends it to the Lost and Found view. This can happen when a device has been erased from a view but never pasted into another view, or when the exact physical location of the device is unknown. Device Type Verification SPECTRUM intelligence automatically verifies the device type for any SNMP device that you add to your network via the Edit menu’s New Model How to Manage Your Network with SPECTRUM Page 156 Document 1909 option, or by AutoDiscovery. The SNMP device verification process works this way: 1. SPECTRUM communicates with the newly modeled device to read the device’s SysObjectID and SysDesc attributes. 2. SPECTRUM intelligence goes through several identification methods to identify the device. • Identify Device by OID Method - If the value of SysObjectID contains the DeviceID, then this method is used. This method relies on the SysObjectID being a unique identifier for a device type. • Identify Device by System Descriptor Method - If the SysObjectID does not contain the DeviceID, then this method is used. This method searches the SPECTRUM database for the model type(s) whose System_Desc_Verify attribute (0x110c6) matches the value of SysDesc that was returned from the device. • Identify Device by Keyword Method - If the Identify Device by System Descriptor and Identify Device by OID methods fail to identify the device, then this method is used. This method uses both the SysObjectID and SysDesc values from the device to identify it. • If the device has not been identified by any of these methods then it is modeled as a generic SNMP device (given that the SNMP Management Module is installed). 3. If multiple model type handles are returned, then SPECTRUM intelligence uses the following method to choose the appropriate model type. • The Disposable_Precedence attributes of each model type is compared and the one with the highest Disposable_Precedence is selected as the model type to be used. • If no model type has a greater Disposable_Precedence, an event (event code 0x10644) is generated containing the IP Address and two model type names. Should this occur you should use the MTE to modify the Disposable_Precedence attribute of one of the two model types so that one of the values is larger. Then, the next time the model type with the highest Disposable_Precedence will be selected. How to Manage Your Network with SPECTRUM Page 157 Document 1909 Device Type Mismatch If the device type can be determined from these attributes, it is compared against the device type selected when the new model (New Icon) was added to the database. If the device is not the same as the new device, the device icon’s condition is set to yellow and an entry is recorded in the Alarm View and Event Log. When the SPECTRUM administrator is issued a device mismatch alarm, there are two things to check. First, check the IP address of the device that is being modeled. If the wrong IP address was assigned, use the Update feature to correct it. Next, check the actual device type to make sure that the correct model type was chosen initially. If not, select the mismatched model and choose the Destroy option from the Edit menu. Then recreate the model using the correct model type. The Model Mismatch intelligence circuit can be enabled or disabled by modifying attribute Verify_Model_Mismatch (0x00011197c). To disable the feature for a given model type, set Verify_Model_Mismatch to FALSE using MTE. To turn the intelligence circuit on, set it to TRUE. In-Band Configuration of Device Alarms SPECTRUM lets you manually configure alarm thresholds for many device models. Normally this is done via a GIB View. For some devices this is a Config Alarms View, accessed through the model’s Configuration View. Refer to the appropriate SPECTRUM management module guide for your device for more information on the configuration process. For more information on alarm thresholds, refer to your device’s hardware/software manuals. Interface Intelligence Interfaces are ports that have both a physical address and a network address. These types of ports are found on routers and bridges, where network identity is important (unlike a port on a repeater, which may have a physical address, but not a network address). Figure 70 illustrates the derivation of the interface port: How to Manage Your Network with SPECTRUM Page 158 Document 1909 Figure 70: Derivation of the Interface Port Port (0x1001c) SnmpPif (0x1006b) Inet_If_Port (0x1001e) CSI_Rptr_Port (0x1001f) (other ports) data_relay_prt (0x10256) cisco_If_Port (0x100cb) (other data_relay_prts) Port is the base model type for all ports. Two basic types of ports are derived from Port: repeater ports and interface ports. • Inet_If_Port is derived from Port and Enet Monitor model types. Inet_If_Port is the base class for all interface ports that are potential monitor points for network statistics. • Data_relay_prt is derived from Inet_If_Port and SnmpPIf. SnmpPIf is the base class for all model types that communicate with SNMP agents. Data_relay_prt is the base model type for all interface ports that do their own reading and polling. It is an instantiable model type and is used to model a generic interface port. More specific types of interface ports, such as Cisco_If_Port, are derived from data_relay_prt. The inference handler CsIHInterfaceIntLinkStatus, which computes InternalPortLinkStatus, is attached at Inet_If_Port so that all interfaces will inherit the desired functionality. The intelligence for interfaces that use ifAdminStatus and ifOperStatus are defined in the MIB-II definition of the interface group in RFC 1158. These variables can have the state of ON or OFF, and are defined below: • ifAdminStatus: desired interface state - This is the state that the administrator wants the interface to be in. This attribute shows whether or not the interface has been shut off. The values of this attribute are ON and OFF. How to Manage Your Network with SPECTRUM Page 159 Document 1909 • ifOperStatus: current interface state - This attribute shows the actual state of the interface. The values of this attribute are UP and DOWN. UP means that the interface is communicating with the network properly. DOWN means the interface has lost connection with the network. These two variables are used to calculate a SPECTRUM internal attribute named INTERNAL_PORT_LINK_STATUS (IPLS - 0x101fb) This attribute’s possible values are LINK_STATUS_GOOD (LSG)), LINK_STATUS_BAD (LSB), and LINK_STATUS_UNKNOWN (LSU). This attribute is used to create and clear alarms, both on the interface and the device it is part of. It is also used to generate events concerning the attempt to reach of the interface. Table 17 shows how these variables are calculated. Table 17: Internal Port Link Status Conditions IfAdminStatus IfOperStatus INTERNAL_PORT_LINK_STATUS ON ON LINK_STATUS_GOOD ON OFF LINK_STATUS_BAD OFF ON LINK_STATUS_UNKNOWN OFF OFF LINK_STATUS_UNKNOWN The interface’s INTERNAL_PORT_LINK_STATUS is set to LSU when SPECTRUM has lost contact with the device. Interface Alarms IPLS (Internal Port Link Status) is used to generate alarms for both the device model and the interface model. The only interface alarm is GRAY. These alarms can only be seen by going into the Detailed Alarm View of an interface model. It is interesting to note that the alarm for an interface with an IPLS of LSB will have a gray alarm. All models with alarms are displayed in the General Alarm View. This is undesirable for interfaces. Interfaces are considered to be a part of a larger device such as a router. When a router goes down, all of its interfaces goes down as well. A red alarm is generated for the router. It would be confusing to have all of the interfaces producing red alarms as well, cluttering the Alarm View and making it difficult to locate the router. A device will watch the IPLS of each of its interfaces. If any of the interfaces has an IPLS of LSB, then the device will generate a YELLOW How to Manage Your Network with SPECTRUM Page 160 Document 1909 alarm with a probable cause of CS_ALARM_CAUSE_PORT_LINK_STATUS_BAD. This alarm, once set, will not be reasserted until it is cleared. Only one alarm will be asserted for all ports. The first interface with an IPLS of LSB creates an alarm. The second and successive interfaces with an IPLS of LSB are put into a bad port list. Once the list is clear the YELLOW alarm is dissasserted. Each of the interfaces watches its own IPLS. Whenever the interface has an IPLS of LSB or LSU it will generate a GRAY alarm. This alarm will only show up in the interfaces Alarm View. When the device becomes unreachable the interface’s IPLS is set to LSU. A GRAY alarm with a probable cause of CS_ALARM_CAUSE_DEV_CONTACT_STATUS_LOST is generated. When the interface has been administratively shut off the interface’s IPLS is set to LSU. A GRAY alarm with a probable cause of CS_ALARM_CAUSE_ADMIN_SHUT_OFF is generated. When the interface becomes unreachable, its IPLS is set to LSB. A GRAY alarm with a probable cause of CS_ALARM_CAUSE_PORT_LINK_STATUS_BAD is generated. Interface Events The interface generates two events that deal with its status. The events contain information about the attempts to reach the interface. These events will be generated when a device has a YELLOW alarm due to a bad interface. Each event message contains the interface number and IP address. For example: Tue 20 Jul, 1994 - 13:31:50 Interface 2 (IP address = 129.128.127.2, type = Gen_IF_Port) on device cisco1 of type Rtr_CiscoMIM is unreachable. - (event [00010623]) As stated before, IPLS has three possible values. This makes it important to know the last two states of an interface’s IPLS to make a proper judgement about its current state. If the interface IPLS is LSB, and it was previously LSU, it is important to know if it was previously LSB or LSG. Due to this, each interface keeps the values of the last two states of IPLS. Events are generated based on these two saved values, and the current value. Table 18 shows which states generate events based on the IPLS. How to Manage Your Network with SPECTRUM Page 161 Document 1909 Table 18: Events Generated from IPLS States 2 Previous Previous Current Event GOOD UNKNOWN GOOD none GOOD UNKNOWN BAD UNREACHABLE GOOD BAD GOOD REACHABLE GOOD BAD UNKNOWN none BAD GOOD BAD UNREACHABLE BAD GOOD UNKNOWN none BAD UNKNOWN GOOD REACHABLE BAD UNKNOWN BAD none UNKNOWN GOOD BAD UNREACHABLE UNKNOWN GOOD UNKNOWN none UNKNOWN BAD GOOD REACHABLE UNKNOWN BAD UNOWNN none How to Manage Your Network with SPECTRUM Page 162 Document 1909 Index Symbols % Load, Bytes, and Packets [116] .csi files used in backgrounds [43] .vnmrc [153] A Accum Button [113] acknowledgement rollups [123] Adding or Changing Devices on your Network [49] address tables [14] Addresses clearing duplicate [142] duplicate [141] Adjusting Management Capacity [32] AgeOutStaleAlarmsOnly attribute [57] Alarm Symptom/Probable Cause [104] Alarm generation on DLCI Ports [58] Alarm generation on interfaces [57] Alarm Thresholds [54] AlarmAgeOutTime attribute [57] AlarmOnInvalidDLCIs [58] AlarmOnLinkDownIfTypes [84] AlarmOnLinkDownTrap [84] AlarmOnSpecificInterfaces [57] Alarms in SPECTRUM [56] All Connected Ports - Multiple Devices Only [73], [78] AssertLinkDownAlarm [85] Audible Alarms [107] colors [107] AutoDiscovery and locked links [28] Autodiscovery [13], [17] Automatic Addressing [143] Automatic Naming of models [143] How to Manage Your Network with SPECTRUM Page 163 Document 1909 B Background changing color of [41] changing image used as [42] Raster [40], [42] backup servers [152] Broadcasts [54] C Change Background Dialog Box [41] channel packet errors [112] Chassis Device View [109] Clear Button [113] Clearing duplicate addresses [142] old alarms [56] collision rate [112] Collisions [54], [115] Colors device condition [100] rollup condition [102] Completing Your Network Model [17] Condition adjusting [125] attributes [123] colors indicating [100] device model [122] icon color display areas [122] Organization Models [131] Rollup [122] Configuring alarms [56] traps [55] connecting a Pingable to a device model [96] connecting two Pingables [96] Connection Lockdown dialog box [28] Connections locked [27] locking and unlocking [28] Contact Lost alarm [59] contact lost polling [38] container models [53] How to Manage Your Network with SPECTRUM Page 164 Document 1909 core [11] CRC and Alignment [115] Criticality [74] Cross-Landscape Fault Correlation [70] D Delta Button [113] Device View [11], [53], [97] Device Condition Colors Blue (Initial) [100] Gray (Unknown) [101] Green (Normal) [101] Orange (Major Alarm) [101] Red (Lost Contact) [101] Yellow (Minor Alarm) [101] Device Models dynamic configuration [120] insignificant [133] significant [133] static configuration [119] Device Topology (DevTop) View [16], [110] Discover Connections [27] Disposable_Precedence [157] down_device_poll_interval_multiplier [39] Duplicate Addresses [141] clearing [142] E Enable Event Creation for a device model [70] for a port model [95] Enterprise Alarm Manager [104] error rate [112] Error Rates [115] Errors [54] Excluding an IP address from the RPA list [155] F fanout [134] Fault Correlation How to Manage Your Network with SPECTRUM Page 165 Document 1909 Cross-Landscape using Enable Event Creation attribute [70] Fault Detection [100] Fault Isolation [104], [131] Configuration [61] configuring using port fault correlation [72] Fault Isolation model [61] Fault Isolation Model Information view [61] Firmware detecting problems [151] frame rate [112] Framesize Statistics [116] G Generating an Inventory Report [18] GIB View [40] group subnets [43] H Height [41] High Broadcast [117] I ICMP [62] ICMP pings [14] Identify Device Methods [157] IEEE 802.3 [13] IMT [119] In-Band Configuration of Device Alarms [158] Inductive Modeling Technology [119] Inferred Connectors [133] insignificant models [54] Intelligence alarm thresholds [158] device type OID verification [156] duplicate address detection [141] firmware problem detection [151] lost and found [156] Interface connected [67] events [161] Intelligence [158] How to Manage Your Network with SPECTRUM Page 166 Document 1909 management neighbor [67] Model Name customizing [22] Suffix [23] Interface Trap Configuration View [85] inventory list [13] IP Address [108] IP addresses Redundancy Preferred Addresses (RPA) [153] L landscape handles [152] link down traps [85] Link Fault Disposition [88] Link traps [80] Links locked [27] locking and unlocking [28] Live Pipes [80], [91] enabling/disabling [91] load [112] Locked Connections [27] Locked Links [27] logical relationships [14] LogPollAnalyzer [33] loopback [153] Lost & Found [49] M Maintaining Your Network Model [48] Maintenance Mode Applications [69] Devices [68] Interfaces [69] Management Lost alarm [59] Management Neighbors Configuration [64] Max_Pulled_Bd_Cnt [121] Modeling Services [44] Monitoring Points [144] Multi-Attribute Graph Example [113] How to Manage Your Network with SPECTRUM Page 167 Document 1909 Multi-Attribute Line Graph [112] N neighbors [64] Network Address [153] network diagram [13] network model [14] O Off-Page Reference panel [17] Online Database Backup Configuration view [51] Optimizing Your Network Model [14] Org_Owns model [46] Out-of-Window (OOW) Collisions [115] OutstandingLinkDownTrap [84] P Packet Size [117] Performance View [111] Pie Chart Relation with Instance ID Example [114] Pie Charts [113] Pingables [95] Pipes Live Pipes [91] Point and Click [108] polling devices that are down [38] polling intervals [35] PollPortStatus [80] port alarms [92] Port Always Down Alarm Supression [82], [93] Port Connection Representation [16] Port Criticality Setting [90] Port Fault Correlation [72], [94] Caveats [74] Criteria [73] Examples [74] setting configuration options [72] How to Manage Your Network with SPECTRUM Page 168 Document 1909 port level connections [16] Port Status Configuring monitoring of, [80] Events and Alarms [83] Polling Criteria [82] Preferred Addresses Dialog Box [154] Primary Address [153] changing in the Preferred Addresses dialog box [155] Protocol Statistics [117] Protocol Type [117] Pulled Board List [120] Pulled_Bd_Cnt [121] Pulled_Bd_List [121] R Redundancy Administrative Status [95] Redundancy Excluded IP Addresses [155] Redundancy Preferred Addresses (RPA) editing the list of [154] Rename Interface Models [23] Report Depth [19] Format [19] Generator [18] Type [19] Rollup Condition adjusting [125] example [127] process flow [126] See also Condition [123] Rollup Condition Color Orange [102] Red [102] Yellow [102] Rollup Condition Colors [102] Rollup Condition Thresholds [54] RPA list [153] Runts and Giants [116] S Scheduling Database Backups [50] Maintenance Mode [68] How to Manage Your Network with SPECTRUM Page 169 Document 1909 Select Color Index Dialog Box [42] Select File Dialog Box [42] Service container model [45] Show Secondary Alarms When Device is in Maintenance [68] Significance Levels [54] significant models [54] Simple Network Management Protocol [55] SNMPv2c [49] SpectroWATCH [60], [92] subnet address range [14] Suppress Linked Port Alarms [74], [94] SysDesc [157] SysObjectID [157] T Threshold Events device internal threshold [152] TIFF files used in backgrounds [43] Topology Models composite [133] discrete [133] Total Button [113] Traffic [54] Traps in SPECTRUM [55] Troubleshooting [114] U unresolved fault alarm [64] Unresolved Fault Alarm Disposition [64] Using Organizational Views [43] Using SpectroWATCH [60] W WA_Links [134] WA_Segment [134], [139] Wide Area Links [134] Width [41] How to Manage Your Network with SPECTRUM Page 170 Document 1909
© Copyright 2025