Atrium Integration Engine: What Is It? How Do You Use It? › Atrium Integration Engine (AIE) Overview – What is it? – Overview of latest features › Tell me how to do it – Under the hood details of creating CMDB Integration using AIE 1 4/9/2010 BMC Atrium Integration Engine › What is it? – A mechanism to manage data transfers between external data sources and CMDB (also AR Forms) – Bi-directional and multi-threaded data transfer engine – An Enterprise Application Integration tool with data transformation capability – It supports event-based or scheduled information flow – Ideal for bulk, incremental, data synchronization transfers – Provides interfaces to external data source and the CMDB – Provides an intuitive graphical interface for field mapping configuration 2 © Copyright 4/9/2010 BMC Software, Inc. BMC Atrium Integration Engine › How is it used? – Transform data to the CMDB data model – Efficiently load data in bulk – Can create relationships between instances as they are inserted in the CMDB. › Advantages – – – – Designed for complex integrations Easily configurable Can be used against a variety of applications Extensible › When Is AIE a Fit? – Complex transfers requiring data transformation – Handling large amount of data transfers – Multiple integration points 3 © Copyright 4/9/2010 BMC Software, Inc. BMC Atrium Integration Engine 7.1 › New Name and Repackaging for EIE › Released in late August 2007 › Broad Enhancement Areas – Performance and Scalability • Improved performance of Relationship exchanges • Better memory management for all update operations – Usability Improvements • Lot of usability improvements (please see the release notes for details on new features) › Packaging – Ships with Atrium CMDB 2.1 to integrate with Atrium CMDB – One product, multiple adapters • • • • Oracle SQL DB2 XML – XML adapter only supports class based relationships (not vendor) • Flat file 4 New! New! Improved! © Copyright 4/9/2010 BMC Software, Inc. How to Create AIE Based Integrations to CMDB: The Ingredients To Success › Data Structure Knowledge › External Data from Discovery/Inventory › Atrium Integration Engine › Atrium CMDB › Reconciliation Rules › Workflow Filters – if Needed 5 © Copyright 4/9/2010 BMC Software, Inc. AIE Terminology › Data Exchange Application (DEA) – AR System based application at the heart of AIE › AIE Service – DEA passes the service instructions on how to interact with the adaptor › Data Mapping – CMDB Class/AR Form mapped to External Table – CMDB Attributes (or fields of AR Form) mapped to Columns of External Table – Values mapped to External Table Columns Data › Exchange – Collection of “similar” Mappings – Definition of when to run – “Pipe that manages the flow” › Adapter – The Plug at the end of the wire 6 © Copyright 4/9/2010 BMC Software, Inc. AIE Service Adapter Exchange Application Data Exchanges Data Mappings Configuration SQL Example Discovery Environment Remedy Environment AR System Discovery SQL Database AIE Service Data Exchange Application Adapter Data Exchanges (can be on either server) Data Mappings CMDB Discovery Dataset CMDB Reconciliation Engine CMDB BMC.ASSET Dataset Data Cleansing Workflow CMDB Reconciliation Rules Remedy Asset Management Database for AR System 7 © Copyright 4/9/2010 BMC Software, Inc. External Data Source › Database data › XML data › Flat file data › Source Relationships can be stored as – foreign key – relationship tables 8 © Copyright 4/9/2010 BMC Software, Inc. Atrium CMDB › Configuration Management Database – – – – Store multiple datasets Reconcile datasets Store Relationships Base data for OOTB Applications › Data Sets – Data value in a field... but so much more – Data Sets you should use • BMC.ASSET – The “Gold Standard” • BMC.SANDBOX – Temp location for GUI changes to Assets • VENDOR.PURPOSE.PRODUCT – custom datasets – Examples: BMC.IMPORT.MARIMBA, MS.IMPORT.SMS, ACME.IMPORT.LEGACY 9 © Copyright 4/9/2010 BMC Software, Inc. AIE Main Console 10 © Copyright 4/9/2010 BMC Software, Inc. Configuration > Database Field Menus › Populates the “vendor” table and column menus › Run before you start any mapping work 1. Load 3. Create Menus 2. Add Desired Tables 11 © Copyright 4/9/2010 BMC Software, Inc. How To – Data Mapping › › › › › › › › 12 Mapping Name = name for new mapping Direction = External Data into CMDB Namespace = BMC.CORE in most cases Class = Destination for data, pick from the list External Data Store = SqlServer, Flatfile or Oracle Table Name = Source of data (pick from list) – if not in the list, run rulehelper.exe to create the menu items. Row Element = (XML only) the element that represents a row Click “Create” to save © Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping How To – Data Mapping If the Table Name does not show in the drop down list, it is because you have not configured the Database Field Menus from the main console Data Exchange Data Mapping Relationship Exchange Relationship Mapping 13 © Copyright 4/9/2010 BMC Software, Inc. How To – Data Mapping › Used to map source and destination side primary/foreign keys › Keys must be unique on both sides – If multiple keys, stored as AND to define Unique 14 © Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping How To – Data Mapping › Map appropriate CDMB Class.Attribute to a Data Store Attribute › Make sure to specify CMDB mandatory fields such as – – – – – – 15 Fields use in Key mappings Data Set ID (automatic if “Use Dataset” is checked) Short Description Submitter Name = anything that identifies the entry AssetLifeCycleStatus = 3 (if Asset Management is installed) © Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping How To – Data Mapping › Rules › 16 – Works like “Function” key in Remedy User and Administrator Tools – Select by highlighting the row and click “Add” – Rule is added – Refer to the Rules box for hints on how to complete the rule Available Rules © Copyright 4/9/2010 BMC Software, Inc. How To – Data Mapping › Only works if you can write to the Vendor / Source Database › Can be any value › Usually is completion status Data Exchange – 0 = transfer with no error – N = error # - corresponds to ARERROR 17 © Copyright 4/9/2010 BMC Software, Inc. Data Mapping Relationship Exchange Relationship Mapping How To – Data Mapping › If you “Update Row to Indicate Delete” i.e. Soft Delete – Set MarkAsDeleted to 1 (Yes) – Other attributes may be set as desired › If you choose to “Permanently Delete” – Hard Delete – No field to set. AIE will do trigger the deletion 18 © Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping How To – Data Mapping - Use to limit amount of data in a given 19 transfer. Important if you are experiencing performance or memory issues If you specified “Restrict to Dataset” on the Main Tab, you don’t need to include it here. © Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping How To – Data Mapping 1 2 3 4 5 6 1. 2. 3. 4. 5. 6. If set to all data that can be transferred will be. This could result in partial data records. Transfer as NULL cleans up any data that might be mistaken for null in a field If trailing blanks are possible in your key values, choose to consider Stores change history in a diary field you designate. Attribute must exist in CMDB. If Yes, updates are only made when a change is detected in the source data › You must select a field that can store a negative integer to store the checksum Data Exchange Allows you to retrieve new or change data since last transfer › You must select a date/time field to store last transfer time Relationship Exchange Data Mapping Relationship Mapping 20 © Copyright 4/9/2010 BMC Software, Inc. How To - Data Exchange › › › › › Exchange Name – Name you create to identify the exchange Instance Name – Specify the AIE thread to use Data Exchange Type – Specify the type of mapping contained in the exchange External Data Store – Must match the mapping(s) Data Exchange Execute Exchange As Relationship Exchange Data Mapping – Schedule Only • Use Schedule tab Relationship Mapping – Event Driven Only • run eiexfer.exe from command line, filter or active link 21 © Copyright 4/9/2010 BMC Software, Inc. How To - Data Exchange › Select the mapping(s) to include in the Exchange container. › Multiple Mappings › 22 – Cuts down on queries for source or target Main Mapping – Need to designate if multiple mappings included – Key values only come from Main Mapping © Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping How To - Data Exchange › Interval – Works like the interval function in escalation workflow › On Completion Section Data Exchange – When this exchange is done, you can configure another exchange to run 23 © Copyright 4/9/2010 BMC Software, Inc. Data Mapping Relationship Exchange Relationship Mapping How To - Data Exchange › Time Data Exchange – Works like the time function in escalation workflow – Set day and time to run 24 © Copyright 4/9/2010 BMC Software, Inc. Data Mapping Relationship Exchange Relationship Mapping How To - Data Exchange › To Be Triggered – Use this option when you want to exchange to run on completion of some other exchange – Must list the exchange in the On Completion section on the other exchange 25 © Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping How To - Data Exchange Use if the AR Server involved in the exchange is NOT the server that DEA is installed on set to “No” and fill in the info. Advantage – 1 license/DEA can service Dev, Test & Production You can either set the external connection information on each exchange or on the Connections link from the Data Exchange Console 26 Alternate Method © Copyright 4/9/2010 BMC Software, Inc. How To - Data Exchange › While Testing – Limit records so you have less to undo › In Production Data Mapping – Set Enable Debug to No 27 Data Exchange Relationship Exchange Relationship Mapping © Copyright 4/9/2010 BMC Software, Inc. How To – Relationship Mapping V_Computer BMC_ComputerSystem Data Mapping BMC_DiskDrive V_DiskDrive Data Mapping Vendor Relationship Table BMC_HostedSystemComponent Vendor Relationship Mapping •Map using External Data Store Mapping. •Trigger on Complete of Data Exchange. Data Exchange Data Mapping Relationship Exchange Relationship Mapping 28 © Copyright 4/9/2010 BMC Software, Inc. How To – Relationship Mapping BMC_ComputerSystem V_Computer Data Mapping BMC_DiskDrive V_DiskDrive Data Mapping Class Relationship Mapping •Map using CI Class Relationship Mapping •Put in an Event Driven Only Exchange* BMC_HostedSystemComponent *In this case, the relationship data exchange only runs after the data transfer for the two configuration item classes has completed. 29 © Copyright 4/9/2010 BMC Software, Inc. Data Exchange Data Mapping Relationship Exchange Relationship Mapping How To - Relationship Mapping › This is where you define which 2 classes to relate and which relationship › Must Select Relationship Source on Create › External Data into CMDB is the only valid Direction › If you choose Relationship Source of “External Data Store” Data Exchange – must specify DB2, FlatFile, Oracle, SqlServer – Table Name › It is best to Restrict to DatasetID and specify 30 © Copyright 4/9/2010 BMC Software, Inc. Data Mapping Relationship Exchange Relationship Mapping How To - Relationship Mapping › Define mapping criteria that defines the relationship › Works just like creating Relationship in CMDB – Source on Left, Target on Right Data Exchange Data Mapping Relationship Exchange Relationship Mapping 31 © Copyright 4/9/2010 BMC Software, Inc. How To - Relationship Mapping › Used to define values for a relationship instance › Mandatory attributes shown above must be mapped › To get a list of Attributes of each CI Class choose class from drop down. Data Exchange Data Mapping Attributes list will then appear 32 © Copyright 4/9/2010 BMC Software, Inc. Relationship Exchange Relationship Mapping How To - Relationship Mapping › Used to define values for a relationship instance › Mandatory attributes shown above must be mapped › To get a list of Attributes of each CI Class choose class from drop down. Data Exchange Data Mapping Attributes list will then appear 33 © Copyright 4/9/2010 BMC Software, Inc. Relationship Exchange Relationship Mapping How To - Relationship Mapping › If you “Update Row to Indicate Delete” i.e. Soft Delete › 34 – Set MarkAsDeleted to 1 (Yes) – Other attributes may be set as desired If you choose to “Permanently Delete” – Hard Delete – No field to set. AIE will do trigger the deletion © Copyright 4/9/2010 BMC Software, Inc. How To - Relationship Mapping › Use if you are doing class based mapping › Should match query used on class mapping › If you specified “Restrict to Dataset” on the Main Tab, you don’t need to include it here. Data Exchange Data Mapping Relationship Exchange Relationship Mapping 35 © Copyright 4/9/2010 BMC Software, Inc. How To - Relationship Mapping › Use if you are doing class based mapping › Should match query used on class mapping › If you specified “Restrict to Dataset” on the Main Tab, you don’t need to include it here. Data Exchange Data Mapping Relationship Exchange Relationship Mapping 36 © Copyright 4/9/2010 BMC Software, Inc. How To - Relationship Mapping › Use if you are doing class based mapping › Should match query used on class mapping › If you specified “Restrict to Dataset” on the Main Tab, you don’t need to include it here. Data Exchange Data Mapping Relationship Exchange Relationship Mapping 37 © Copyright 4/9/2010 BMC Software, Inc. How To - Relationship Mapping › Used only if a External Data Store mapping › This is used to limit the external records returned Data Exchange Data Mapping Relationship Exchange Relationship Mapping 38 © Copyright 4/9/2010 BMC Software, Inc. How To - Relationship Exchange › Works the same as a normal Data Exchange › Data Exchange Type = Relationship Class Data Exchange Data Mapping Relationship Exchange Relationship Mapping 39 © Copyright 4/9/2010 BMC Software, Inc. General AIE Hints › When testing an exchange – – – – – Test one exchange at a time (when done chain them to be triggered) Use Run Now button to trigger an exchange immediately Set Limit = 1 (so that you only run once) Be sure you know your debug path › When writing a Vendor Query – – Test it with Direct SQL – If it doesn’t work there, it won’t work here – Use Indexed fields if possible This won’t work – MacAddress != NULL This will – MacAddress IS NOT NULL 40 © Copyright 4/9/2010 BMC Software, Inc. General AIE Hints › When setting an AR System field in a mapping – – Doesn’t work like a setfield or pushfield operation – Must Use Rules to convert functions, keywords, etc. Example of an Integer Field This won’t work – 0 This will – Constant|0 Example of Concatenation This won’t work – $Manufacturer0$+”-”+$MediaType0$ This will – function|concat($Manufacturer0$,”-”,$MediaType0$) 41 © Copyright 4/9/2010 BMC Software, Inc. Atrium CMDB › Reconciliation Rules CMDB 3rd Party Dataset – Identify CMDB Master Dataset • Look for potential duplicates – Compare – Merge – Cleanup / Purge CMDB Reconciliation Engine • Can delete entire data set from all tables • Can delete selected items – Configuration Item Set field MarkedForDelete = Yes – Create Reconciliation Rule to Purge if field is set » Define how to handle related items – Run as a Reconciliation Job 42 © Copyright 4/9/2010 BMC Software, Inc. CMDB Reconciliation Rules