1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The following is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.
2
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Key Patterns Converge: Data
Integration and Data
Virtualization for SOA
(CON8514)
Alex Kotopoulis and Ayush Ganeriwal
Data Integration Product Management
3
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template
Agenda
 Oracle Data Integration Solutions Overview
 Combining SOA and Oracle Data Integrator
 Data Virtualization with Oracle Data Services
 Q&A
4
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template
Oracle Data Integration
Overview
5
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template
Oracle Data Integration Solutions
Thousands of customers including Fortune
500 companies, representing Financial
Services, Communications, Healthcare, Public
Sector, Retail, Utilities industries
• 4 of top 5 largest commercial banks
• 3 of top 3 busiest ATM networks
• 5 of top 5 financial data services
companies
• 5 of top 5 telecommunications providers
• 3 of top 5 largest food & drug stores
6
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template
Oracle Data Integration 12c
Delivering Real-Time Integration for Big Data and Cloud
Cloud
Oracle Data Integrator
Apps
Oracle GoldenGate
Database
Big Data
7
Oracle Enterprise Data
Quality
Oracle Data Service
Integrator
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
• Real-time replication; optimized for
Database 12c and Oracle Exadata
• End-to-end integrated with simplified
deployment
• Unified tooling for both structured data
sources and Hadoop / NoSQL
• Flexible deployment on-premise or in
the Cloud for heterogeneous systems
• Expanded support for 3rd party systems
and Oracle Applications in real-time data
integration and continuous availability
solutions
Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template
Oracle Data Integrator
Bulk Data Processing and Data Transformation
Cloud
Apps
Oracle Data Integrator
High Performance E-LT
Declarative Design
Database
Extensible Knowledge Modules
 Certified for leading
technologies to deliver fast
time to value
 High-performance, low cost
of ownership E-LT
architecture
 Lightweight deployment
Big Data
CEP Data Services
Capable of Integrating Structured &
Unstructured Data
8
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template
 Flexible, easy to enrich
functionality
Oracle GoldenGate
Real-time Data Integration
Cloud
Oracle GoldenGate
Apps
Log-based Change Data Capture
and Delivery
Active-Active Replication
Database
Big Data
Heterogeneous Source and Targets
Reliability and Transaction Integrity
Java Integration, Real-time Events
9
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template
• High-performance,
low-impact real-time
data integration and
replication
• Timely data for
improved business
insight
• Continuous availability
for 24/7 operation
Oracle Enterprise Data Quality
Data Quality for Customer and Product Data
Cloud
Oracle Enterprise Data Quality
Profile, Explore, Audit
Apps
Parse, Standardize, Cleanse
Database
Big Data
Match, Merge, Enrich
Governance
Case Management
10
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template
• Improves data
accuracy, usability and
‘fitness for purpose
• Unified interface for
ease-of-use, lower
TCO
• Depth of capability in
multiple data domains
reduces project risk
Oracle Data Services Integrator
Data Federation and Virtualization
Cloud
Oracle Data Services Integrator
Federated Data Services
Apps
Data Virtualization
Database
Big Data
Integrates with Oracle SOA
Bidirectional Reads & Writes
Integrates with Oracle Business
Analytics
11
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template
• Easily develop data
services and ad-hoc
queries without
infrastructure changes
• Open and standards
based—SOA, Web
Services, Java,
Service Data Object
Oracle Data Integration
Combining SOA and
Oracle Data Integrator
12
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Data Integration Services
Client Applications: BPEL, OSB, Custom
OGG
Data
Replication
Services
WSDL
ODI
ELT Services
Agent
WSDL
Data Access
Services
Data
Access
MoveTransformed
& Replicate Data
in Real Time
Transform
Bulk Data
JMS
Oracle
Data Warehouse
Other DB
WSDL
WSDL
ODSI Data
Virtualization
Services
OEDQ
Data Quality
Services
Validate
Virtualize Access
to and
Cleanse Data
distributed sources
Application
File
Web
Service
Real-time Data Warehousing with ODI,
OGG and OEDQ
ODI Agent
OEDQ
Transform
Validation
&
Cleansing
Kick off OEDQ
process
Bulk for
load
snapshotofof
Waits
availability
Production
DB into
incremental data
staging
Production Schema
EMP
Transform and load
data into Data
Warehouse
DEPT
EMP
DEPT
Initial Load
Staging Schema
ODI generates and kicks
of OGG process to
replicate incremental
changes
DBMS
Logs
OGG
Trail
Pump
DIM
FACT
Transform
& Load
DIM
DIM
DW Schema
TCP/IP
Extract
DIM
Trail
Replicat
OGG
ODI for Transforming Large Payloads
Oracle BPEL PM or ESB
1
Order DB
2
<yxz>
<zyx>
Reference
3
Reference
8
7
Reference
ODI Invoke
Data Integrator
4
Product
Suppliers
Shared
Metadata
Repository
5
Transform
6
Shared
Container/JVM
Shared SOA Server
1. Large File arrives, detected by BPEL File
Adapter
2. Execution Starts (BPEL/ ESB) – and a step
for transforming a large document
payload occurs
3. Pass XML payload, by reference, to ODI
4. ODI loads payload
5. ODI transforms payload
6. ODI sends payload wherever instructed
7. ODI notifies BPEL/ESB that job is complete
8. Core BPEL/ESB processing completes
15
ODI Loads a BI Warehouse
(with Error Hospital Human Workflow)
1
Oracle BPEL PM or ESB
Event
1.
2
An event occurs which prompts
a data load
2.
Execution starts within BPEL/
ESB process
3.
Instruction to start ODI
4.
ODI extracts & loads data
a)
Some errors w/in batch
b)
Notification to BPEL Human
Workflow engine
c)
Repair & approvals process
d)
Repairs are committed
e)
ODI looks for repaired data
when “Recycle” option is on
5.
ODI transforms the data
6.
ODI confirms job complete
7.
BPEL/ESB process continues
any additional process
execution steps
c
Human
Workflow
7
d
3
Instruction
6
Confirmation
ODI Invoke
Shared
Metadata
Repository
b
Data Integrator
e
Transform
Shared
Container/JVM
Shared SOA Server
a
DB Server
Any DB
5
4
Transform
Records
A
DW Server
Business
Intelligence
Records
B
16
ODI for SOA-based ERP Integration
1.
2.
3.
A business process for
Migration (initial bulk data
load) or Replication
(ongoing synch) is
invoked
a)
b)
1
2
5.
BPEL/ESB begins
processing ERP business
transactions
BPEL/ESB leverage
same Unique IDs for
canonical XML
UI
XML
5
App
Instruction
c
ODI updates Unique
ID for existing objects
ODI confirms job
c)
App
ERP
App 2
Oracle BPEL PM or ESB
XML
ODI creates Unique ID
for new ERP data
objects
4.
6.
UI
BPEL/ESB sends
instruction to ODI
ODI performs E-LT
Messaging and bulk data
may leverage the same unique
object IDs – thereby ensuring
uniform ERP data objects
(eg: PK123 = Pkxyz)
ERP
App 1
Confirmation
4
ODI Invoke
Shared
Canonical
ID XREF
Lookups
Data Integrator
Data
3
Shared
Container/JVM
DDL
a
Data
Transform
b
DDL
All SOA and ETL jobs
keep business data
aligned with Unique IDs
17
ODI High Availability
Performance and Reliability through Weblogic and Coherence
Clients
WebLogic Cluster
ODI Repositories on
Oracle RAC
WLS1
ODI Public Web
Services
ODI Agent
Repository
Explorer
ODI Studio
Designer
WLS2
Proxy /
Load
Balancer
ODI Agent
Operator
WLS3
Topology
ODI Agent
Coherence ODI Scheduler Sync
ODI SDK APIs
ODI Master
Repository
ODI Work
ODI Work
Repository
ODI Work
Repository
Repository
Security
18
XML File processing
<customer id=‘4433’ name=‘Paul’>
<order id=‘1’>
<item id=‘101’ count=‘5’/>
<item id=‘102’ count=‘3’/>
<item id=‘103’ count=‘3’/>
</order>
<order id=‘2’>
<item id=‘201’ count=‘5’/>
<item id=‘202’ count=‘3’/>
</order>
</customer
XML
Schema
customer
order
ODI XML
JDBC
Driver
PK
FK
1
1
2
1
PK
ID
Name
1
4433
Paul
item
PK
FK
id
count
1
1
101
5
2
1
102
3
3
1
103
3
4
2
201
5
5
2
202
3
Complex File processing
Native Format Builder UI
Staging Tables
customer
order
C|4433|Paul
O|1
I|101|5
I|102|3
I|103|3
O|2
I|201|5
I|202|3
I|203|3
nXSD Schema
S_OPTY
ODI
Complex
File JDBC
Driver
PK
FK
1
1
2
1
PK
ID
Name
1
4433
Paul
item
PK
FK
id
count
1
1
101
5
2
1
102
3
3
1
103
3
4
2
201
5
5
2
202
3
Oracle Data Integration
Data Virtualization with
Oracle Services
Integrator
21
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Different Data Access Techniques
•
•
•
Consolidation: Move data from distributed sources to central hub;
consuming applications query hub database
Data Virtualization / Federation: Query and update distributed
sources in-place and provide combined result
Hybrid Approach: Federation and Consolidation are combined based
on best approach for individual use cases
Consumer
Consumer
Consumer
ODSI
Consolidated Hub
ODSI
ODI/OGG
Source A
Source B
Consolidation only
Consolidated Hub
ODI/OGG
Source A
Source B
Data Virtualization only
Source A
Source B
Source C
Hybrid Consolidation +
Data Virtualization
When to Use Which Integration Technology
Data Virtualization
Consolidation
• Data cannot be replicated (security, ownership)
•
Performance is priority
• Consolidation is not efficient for access pattern
•
Data needs to be offloaded from original source
• Real-time access is required and replication is not
efficient
•
Replication is used for redundancy
• Transactional updates
• Need to change model frequently
23
Data Virtualization
Service Bus / Business Process
• Combine & normalize data access services
(Create, Read, Update, Delete)
•
Orchestrate services to business processes
•
Service Routing / Mediation
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Data Virtualization Use Cases
 Data Services Layer
– Provide a clean data abstraction for SOA and application developers
– Shield consumers from changes in data topology
 Augmenting Business Intelligence
– Extend access to sources outside the data warehouse
 Rapid Prototyping of Data Integration
– Quick creation of an interface to integrated data without building a
consolidated store
 Cloud Data Integration
– Combine onsite and offsite data resources in a unified interface
24
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Data Virtualization: Oracle Data Service Integrator
Product Architecture
Java
Web Svc
JDBC
.NET
Fast Track Transport
Metadata
Monitoring/Auditing
ODSI SDK
Admin Console
ODSI Studio
IDE
OSB
Security
Cache
Query Engine
Fast Track Transport
OSB
Database
25
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Web Svc
XML/Flat File
API / Apps
OER
ODSI Core Strengths
• Native SOA Data Integration
– Native XML type system allows processing and passing of hierarchical & SOA data.
– Based on open standards (XML, WS, JDBC/SQL, XQuery)
– Tight integration in Oracle SOA: Fast-path with OSB, Governance with OER
• Normalization and Reuse through Data Services
–
Multi-layer approach to normalizing and transforming source data
• Highly Scalable, Industrial-Strength Performance
–
–
Linear scalability through WLS Clusters
Performance through caching, streaming and query optimization
• Fine-Grained Security and Auditing
–
Row and column-based data security
• Comprehensive Update Capabilities
–
–
26
Automated CRUD operations on data services
Distributed XA transactions through WLS Transaction Manager
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Data Service Design & Modeling
Layered Approach
ODSI Development Environment
28
ODSI Logical Data Service
29
ODSI Parameterized Data Service
30
ODSI Query Plan
31
ODSI Test Execution
32
ODSI Update Mapping
33
ODSI Administration Console
34
Oracle Data Integration
Q&A
Questions & Answers
36
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Join the Data Integration Community
Twitter
twitter.com/ORCLGoldenGate
LinkedIn
“Oracle Data Integration”
Facebook
facebook.com/OracleDataIntegration
Oracle’s Data Integration blog
blogs.oracle.com/dataintegration
Data Integration YouTube channel
youtube.com/oraclegoldengate
Oracle Data Integrator Books
http://www.packtpub.com/oracle-data-integrator-11ggetting-started/book
http://www.packtpub.com/oracle-data-integrator-11gcookbook/book
Oracle.com/goto/dataintegration
37
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Oracle OpenWorld – Key DIS Sessions
Oracle Data Integrator
Monday
Tuesday
Wednesday
CON8513 Oracle Data Integrator
Product Updated and Future
Strategy
1:45 PM Moscone West 2024
HOL10130 Introduction to Oracle
Data Integrator
12:00 PM Marrriott Marquis Nob
Hill AB
CON8515 Customers
Perspectives: Oracle Data
Integrator
1:15 PM Moscone West 2012
HOL10144 Combining Oracle
Data Integrator and Oracle
GoldenGate
3:45PM Marriott Marquis Nob
Hill AB
CON8544 Making the Move to
Oracle Data Integrator
3:30 PM Moscone West 2012
CON8514 Key Patterns
Converge: Data Integration
and Data Virtualization for
SOA
5:15 PM Moscone West 2024
CON8516 Extracting Strategic
Insights from Business Data
wit Oracle Data Integrator
5:15 PM Moscone 2012
38
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Thursday
Graphic Section Divider
39
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
40
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
41
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.