› Atrium Integration Engine: What Is It? How Do You Use It?

Atrium Integration Engine: What Is It?
How Do You Use It?
› Atrium Integration Engine (AIE) Overview
– What is it?
– Overview of latest features
› Tell me how to do it
– Under the hood details of creating CMDB Integration using AIE
1
4/9/2010
BMC Atrium Integration Engine
› What is it?
– A mechanism to manage data transfers between external data sources and
CMDB (also AR Forms)
– Bi-directional and multi-threaded data transfer engine
– An Enterprise Application Integration tool with data transformation capability
– It supports event-based or scheduled information flow
– Ideal for bulk, incremental, data synchronization transfers
– Provides interfaces to external data source and the CMDB
– Provides an intuitive graphical interface for field mapping configuration
2
© Copyright 4/9/2010 BMC Software, Inc.
BMC Atrium Integration Engine
› How is it used?
– Transform data to the CMDB data model
– Efficiently load data in bulk
– Can create relationships between instances as they are inserted in the CMDB.
› Advantages
–
–
–
–
Designed for complex integrations
Easily configurable
Can be used against a variety of applications
Extensible
› When Is AIE a Fit?
– Complex transfers requiring data transformation
– Handling large amount of data transfers
– Multiple integration points
3
© Copyright 4/9/2010 BMC Software, Inc.
BMC Atrium Integration Engine 7.1
› New Name and Repackaging for EIE
› Released in late August 2007
› Broad Enhancement Areas
– Performance and Scalability
• Improved performance of Relationship exchanges
• Better memory management for all update operations
– Usability Improvements
• Lot of usability improvements (please see the release notes for details on new features)
› Packaging
– Ships with Atrium CMDB 2.1 to integrate with Atrium CMDB
– One product, multiple adapters
•
•
•
•
Oracle
SQL
DB2
XML
– XML adapter only supports class based relationships (not vendor)
• Flat file
4
New!
New!
Improved!
© Copyright 4/9/2010 BMC Software, Inc.
How to Create AIE Based Integrations to CMDB:
The Ingredients To Success
› Data Structure Knowledge
› External Data from Discovery/Inventory
› Atrium Integration Engine
› Atrium CMDB
› Reconciliation Rules
› Workflow Filters – if Needed
5
© Copyright 4/9/2010 BMC Software, Inc.
AIE Terminology
› Data Exchange Application (DEA)
– AR System based application at the heart of AIE
› AIE Service
– DEA passes the service instructions on how to interact with the adaptor
› Data Mapping
– CMDB Class/AR Form mapped to External Table
– CMDB Attributes (or fields of AR Form) mapped to Columns of External Table
– Values mapped to External Table Columns
Data
› Exchange
– Collection of “similar” Mappings
– Definition of when to run
– “Pipe that manages the flow”
› Adapter
– The Plug at the end of the wire
6
© Copyright 4/9/2010 BMC Software, Inc.
AIE
Service
Adapter
Exchange
Application
Data
Exchanges
Data
Mappings
Configuration SQL Example
Discovery Environment
Remedy Environment
AR System
Discovery
SQL Database
AIE
Service
Data
Exchange
Application
Adapter
Data
Exchanges
(can be on either server)
Data
Mappings
CMDB
Discovery
Dataset
CMDB
Reconciliation
Engine
CMDB
BMC.ASSET
Dataset
Data
Cleansing
Workflow
CMDB
Reconciliation
Rules
Remedy
Asset
Management
Database for AR System
7
© Copyright 4/9/2010 BMC Software, Inc.
External Data Source
› Database data
› XML data
› Flat file data
› Source Relationships can be stored as
– foreign key
– relationship tables
8
© Copyright 4/9/2010 BMC Software, Inc.
Atrium CMDB
› Configuration Management Database
–
–
–
–
Store multiple datasets
Reconcile datasets
Store Relationships
Base data for OOTB Applications
› Data Sets
– Data value in a field... but so much more
– Data Sets you should use
• BMC.ASSET – The “Gold Standard”
• BMC.SANDBOX – Temp location for GUI changes to Assets
• VENDOR.PURPOSE.PRODUCT – custom datasets
– Examples: BMC.IMPORT.MARIMBA, MS.IMPORT.SMS, ACME.IMPORT.LEGACY
9
© Copyright 4/9/2010 BMC Software, Inc.
AIE Main Console
10
© Copyright 4/9/2010 BMC Software, Inc.
Configuration > Database Field Menus
› Populates the “vendor” table and column menus
› Run before you start any mapping work
1. Load
3. Create Menus
2. Add Desired Tables
11
© Copyright 4/9/2010 BMC Software, Inc.
How To – Data Mapping
›
›
›
›
›
›
›
›
12
Mapping Name = name for new mapping
Direction = External Data into CMDB
Namespace = BMC.CORE in most cases
Class = Destination for data, pick from the list
External Data Store = SqlServer, Flatfile or Oracle
Table Name = Source of data (pick from list)
– if not in the list, run rulehelper.exe to create the menu items.
Row Element = (XML only) the element that represents a row
Click “Create” to save
© Copyright 4/9/2010 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
If the Table Name does not show in the drop down list,
it is because you have not configured the Database
Field Menus from the main console
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
13
© Copyright 4/9/2010 BMC Software, Inc.
How To – Data Mapping
› Used to map source and destination side primary/foreign keys
› Keys must be unique on both sides
– If multiple keys, stored as AND to define Unique
14
© Copyright 4/9/2010 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
› Map appropriate CDMB Class.Attribute to a Data Store Attribute
› Make sure to specify CMDB mandatory fields such as
–
–
–
–
–
–
15
Fields use in Key mappings
Data Set ID (automatic if “Use Dataset” is checked)
Short Description
Submitter
Name = anything that identifies the entry
AssetLifeCycleStatus = 3 (if Asset Management is installed)
© Copyright 4/9/2010 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
› Rules
›
16
– Works like “Function” key in Remedy User and Administrator Tools
– Select by highlighting the row and click “Add”
– Rule is added
– Refer to the Rules box for hints on how to complete the rule
Available Rules
© Copyright 4/9/2010 BMC Software, Inc.
How To – Data Mapping
› Only works if you can write to the Vendor / Source Database
› Can be any value
› Usually is completion status
Data Exchange
– 0 = transfer with no error
– N = error # - corresponds to ARERROR
17
© Copyright 4/9/2010 BMC Software, Inc.
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
› If you “Update Row to Indicate Delete” i.e. Soft Delete
– Set MarkAsDeleted to 1 (Yes)
– Other attributes may be set as desired
› If you choose to “Permanently Delete” – Hard Delete
– No field to set. AIE will do trigger the deletion
18
© Copyright 4/9/2010 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
- Use to limit amount of data in a given
19
transfer. Important if you are experiencing
performance or memory issues
If you specified “Restrict to Dataset” on
the Main Tab, you don’t need to include it
here.
© Copyright 4/9/2010 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To – Data Mapping
1
2
3
4
5
6
1.
2.
3.
4.
5.
6.
If set to all data that can be transferred will be. This could result in partial data records.
Transfer as NULL cleans up any data that might be mistaken for null in a field
If trailing blanks are possible in your key values, choose to consider
Stores change history in a diary field you designate. Attribute must exist in CMDB.
If Yes, updates are only made when a change is detected in the source data
› You must select a field that can store a negative integer to store the checksum
Data Exchange
Allows you to retrieve new or change data since last transfer
› You must select a date/time field to store last transfer time
Relationship Exchange
Data Mapping
Relationship Mapping
20
© Copyright 4/9/2010 BMC Software, Inc.
How To - Data Exchange
›
›
›
›
›
Exchange Name – Name you create to identify the exchange
Instance Name – Specify the AIE thread to use
Data Exchange Type – Specify the type of mapping contained in the exchange
External Data Store – Must match the mapping(s)
Data Exchange
Execute Exchange As
Relationship Exchange
Data Mapping
– Schedule Only
• Use Schedule tab
Relationship Mapping
– Event Driven Only
• run eiexfer.exe from command line, filter or active link
21
© Copyright 4/9/2010 BMC Software, Inc.
How To - Data Exchange
› Select the mapping(s) to include in the Exchange container.
› Multiple Mappings
›
22
– Cuts down on queries for source or target
Main Mapping
– Need to designate if multiple mappings included
– Key values only come from Main Mapping
© Copyright 4/9/2010 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Data Exchange
› Interval
– Works like the interval function in escalation workflow
› On Completion Section
Data Exchange
– When this exchange is done, you can configure
another exchange to run
23
© Copyright 4/9/2010 BMC Software, Inc.
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Data Exchange
› Time
Data Exchange
– Works like the time function in escalation workflow
– Set day and time to run
24
© Copyright 4/9/2010 BMC Software, Inc.
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Data Exchange
› To Be Triggered
– Use this option when you want to exchange to run on
completion of some other exchange
– Must list the exchange in the On Completion section
on the other exchange
25
© Copyright 4/9/2010 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Data Exchange
Use if the AR Server involved in
the exchange is NOT the server
that DEA is installed on set to
“No” and fill in the info.
Advantage – 1 license/DEA can
service Dev, Test & Production
You can either set the
external connection
information on each
exchange or on the
Connections link from
the Data Exchange
Console
26
Alternate Method
© Copyright 4/9/2010 BMC Software, Inc.
How To - Data Exchange
› While Testing
– Limit records so you have less to undo
› In Production
Data Mapping
– Set Enable Debug to No
27
Data Exchange
Relationship Exchange
Relationship Mapping
© Copyright 4/9/2010 BMC Software, Inc.
How To – Relationship Mapping
V_Computer
BMC_ComputerSystem
Data Mapping
BMC_DiskDrive
V_DiskDrive
Data Mapping
Vendor Relationship Table
BMC_HostedSystemComponent
Vendor Relationship Mapping
•Map using External Data Store Mapping.
•Trigger on Complete of Data Exchange.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
28
© Copyright 4/9/2010 BMC Software, Inc.
How To – Relationship Mapping
BMC_ComputerSystem
V_Computer
Data Mapping
BMC_DiskDrive
V_DiskDrive
Data Mapping
Class
Relationship Mapping
•Map using CI Class Relationship Mapping
•Put in an Event Driven Only Exchange*
BMC_HostedSystemComponent
*In this case, the relationship data
exchange only runs after the data
transfer for the two configuration
item classes has completed.
29
© Copyright 4/9/2010 BMC Software, Inc.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Relationship Mapping
› This is where you define which 2 classes to relate and which relationship
› Must Select Relationship Source on Create
› External Data into CMDB is the only valid Direction
› If you choose Relationship Source of “External Data Store”
Data Exchange
– must specify DB2, FlatFile, Oracle, SqlServer
– Table Name
› It is best to Restrict to DatasetID and specify
30
© Copyright 4/9/2010 BMC Software, Inc.
Data Mapping
Relationship Exchange
Relationship Mapping
How To - Relationship Mapping
› Define mapping criteria that defines the relationship
› Works just like creating Relationship in CMDB
– Source on Left, Target on Right
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
31
© Copyright 4/9/2010 BMC Software, Inc.
How To - Relationship Mapping
› Used to define values for a relationship instance
› Mandatory attributes shown above must be mapped
› To get a list of Attributes of each CI Class choose class from drop down.
Data Exchange
Data Mapping
Attributes list will then appear
32
© Copyright 4/9/2010 BMC Software, Inc.
Relationship Exchange
Relationship Mapping
How To - Relationship Mapping
› Used to define values for a relationship instance
› Mandatory attributes shown above must be mapped
› To get a list of Attributes of each CI Class choose class from drop down.
Data Exchange
Data Mapping
Attributes list will then appear
33
© Copyright 4/9/2010 BMC Software, Inc.
Relationship Exchange
Relationship Mapping
How To - Relationship Mapping
› If you “Update Row to Indicate Delete” i.e. Soft Delete
›
34
– Set MarkAsDeleted to 1 (Yes)
– Other attributes may be set as desired
If you choose to “Permanently Delete” – Hard Delete
– No field to set. AIE will do trigger the deletion
© Copyright 4/9/2010 BMC Software, Inc.
How To - Relationship Mapping
› Use if you are doing class based mapping
› Should match query used on class mapping
› If you specified “Restrict to Dataset” on the Main Tab, you
don’t need to include it here.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
35
© Copyright 4/9/2010 BMC Software, Inc.
How To - Relationship Mapping
› Use if you are doing class based mapping
› Should match query used on class mapping
› If you specified “Restrict to Dataset” on the Main Tab, you
don’t need to include it here.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
36
© Copyright 4/9/2010 BMC Software, Inc.
How To - Relationship Mapping
› Use if you are doing class based mapping
› Should match query used on class mapping
› If you specified “Restrict to Dataset” on the Main Tab, you
don’t need to include it here.
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
37
© Copyright 4/9/2010 BMC Software, Inc.
How To - Relationship Mapping
› Used only if a External Data Store mapping
› This is used to limit the external records
returned
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
38
© Copyright 4/9/2010 BMC Software, Inc.
How To - Relationship Exchange
› Works the same as a normal Data Exchange
› Data Exchange Type = Relationship Class
Data Exchange
Data Mapping
Relationship Exchange
Relationship Mapping
39
© Copyright 4/9/2010 BMC Software, Inc.
General AIE Hints
› When testing an exchange –
–
–
–
–
Test one exchange at a time (when done chain them to be triggered)
Use Run Now button to trigger an exchange immediately
Set Limit = 1 (so that you only run once)
Be sure you know your debug path
› When writing a Vendor Query –
– Test it with Direct SQL – If it doesn’t work there, it won’t work here
– Use Indexed fields if possible
This won’t work – MacAddress != NULL
This will – MacAddress IS NOT NULL
40
© Copyright 4/9/2010 BMC Software, Inc.
General AIE Hints
› When setting an AR System field in a mapping –
– Doesn’t work like a setfield or pushfield operation
– Must Use Rules to convert functions, keywords, etc.
Example of an Integer Field
This won’t work – 0
This will – Constant|0
Example of Concatenation
This won’t work – $Manufacturer0$+”-”+$MediaType0$
This will – function|concat($Manufacturer0$,”-”,$MediaType0$)
41
© Copyright 4/9/2010 BMC Software, Inc.
Atrium CMDB
› Reconciliation Rules
CMDB
3rd Party
Dataset
– Identify
CMDB
Master
Dataset
• Look for potential duplicates
– Compare
– Merge
– Cleanup / Purge
CMDB
Reconciliation
Engine
• Can delete entire data set from all tables
• Can delete selected items
– Configuration Item Set field MarkedForDelete = Yes
– Create Reconciliation Rule to Purge if field is set
» Define how to handle related items
– Run as a Reconciliation Job
42
© Copyright 4/9/2010 BMC Software, Inc.
CMDB
Reconciliation
Rules