Design Considerations for Large
Scale Deployment of Oracle VM in
Oracle’s Managed Cloud Service
Jose Fernando Niño Higuera
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and should not be
relied upon in making purchasing decisions. The development, release, and timing of any
features or functionality described for Oracle’s products remains at the sole discretion of
Oracle.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
3
Program Agenda
1
What is Oracle Managed Cloud Services (OMCS)
2
OMCS Design Criteria
3
How We Use Oracle VM
4
Lessons Learned and Best Practices
5
Benefits of Oracle VM and OMCS
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
4
Oracle Managed Cloud Services (OMCS) (
)
Three Ways for Customers to use Oracle Software
1. Purchase Software Product and self-host
2. Subscribe to Software as a Service (SaaS)
3. Have OMCS host the customer’s software
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
5
Oracle SaaS, PaaS, IaaS Cloud Offerings
Infrastructure
Platform
Software
as a Service
as a Service
as a Service
Platform
Database
Java
Database Backup
Developer
Documents
Business Intelligence
Mobile
Customers
Marketing
Sales
Service
People
Global Human Resources
Talent Management
Business
Financials
Procurement
Project Portfolio Management
Supply Chain
Value Chain execution
Product Value Chain
Cloud Marketplace
Infrastructure
Compute
Storage
Messaging
Enterprise Performance
Enterprise Planning
Financial Planning
Social
Social Network
Social Marketing
Social Engagement &
Monitoring
Social Data & Insight
Multi-Tenant Shared Machines*
Oracle owns Hardware and Software – Customer pays for usage
* typically
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
6
Oracle Managed Cloud Services (OMCS)
Applications
Technology
Extended
Managed Cloud Service
Managed Cloud Service
Managed Cloud Service
E-Business Suite
PeopleSoft
Siebel
J D Edwards
Hyperion
Business Intelligence
Commerce
Agile
Retail
Governance Risk & Compliance
Fusion Applications
Demand Management
Markdown Optimization
Information Discovery
Project Management
Beehive Collaboration
Transportation Management
User Productivity Kit
Retail Predictive Application
Oracle Database
Fusion Middleware
Web Center
Engineered Systems
Identity Management
Backup
Refresh
Upgrade
Migration
CEMLI Management
Business Transaction Monitoring
Security
PCI & HIPAA Compliance
Disaster Recovery
Non-Production Environment Service
Other Extended Services
Single-Tenant Dedicated Machines
Customer owns Software - Oracle owns Hardware and manages everything
Typically in the Oracle Data Center – but sometimes @customer/partner
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
7
Oracle Managed Cloud Services (OMCS)
Why use OMCS
– Single Provider:
Hardware, Software, Network, Storage, Interoperability
If there’s a problem, it’s Oracle’s problem
– Expertise:
Let Oracle manage Oracle
Large expert pool available around the clock
Direct Access to Product Development Groups
– Leverage:
Design & Optimize once, Repeat often
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
8
Oracle Virtualization Strategy
• At the core of Oracle’s cloud strategy
• Integrated VM lifecycle & cloud management
solution with Oracle Enterprise Manager
• Supports both x86 and SPARC
• Integrated with OpenStack
• Cloud platform for Oracle & Non-Oracle
applications
– Supports Oracle Linux, Oracle Solaris, Microsoft Windows
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
9
About Oracle VM
Oracle VM Templates
Oracle Real Application Clusters(RAC)
Oracle E-Business Suites
Oracle JD Edward EnterpriseOne
…….
Oracle Linux
Oracle Solaris
Oracle Virtual
Networking
Integrated for scale &
ease of deployment
Boosts Performance
by 30%.
X86 and SPARC
Oracle Enterprise
Manager
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
End-to-end
ManagementPhysical to Virtual
To Cloud
10
OMCS Design Criteria
1.
2.
3.
4.
5.
6.
7.
Security
Isolation of Users (Customers)
Stability
Disaster Containment
Large Scale
Performance
Cost
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
11
OMCS Design Criteria: Security
• Protect the Customer’s Data under all circumstances
• Rigorous Security Review Process
Deployment Architecture
Need-to-know access to data
Authentication, Authorization & Audit for all Activities
• Guard against unauthorized Access
Intrusion Detection
• Perimeter Security around the Deployment Cloud
Additional Segregation & Firewalls within
• Security Patches via Routine Maintenance
 Security
Isolation of Users (Customers)
Stability
Disaster Containment
Large Scale
Performance
Cost
Virtualization allows us the luxury of having separate Machines for
each Tier within each Instance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
12
OMCS Design Criteria: Isolation of Users (Customers)
• Strong Separation between Customers is mandatory
One customer must never see another’s data or activities
• After years of testing and Operation
– Oracle VM Virtual Machines are proven just as safe as
physical machines
– Virtual disks with backing store from a storage pool are safe
• Flat Network alone doesn’t isolate enough for OMCS
iptables + ebtables + Perimeter Firewall provide Security
vLAN + Internal Firewall provide Isolation
 Security
 Isolation of Users (Customers)
Stability
Disaster Containment
Large Scale
Performance
Cost
Oracle VM Virtual Machines plus vLANs meet our
Segregation Requirements
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
13
OMCS Design Criteria: Stability
• Preservation of Data must never be in
question
• What matters is the Customer’s App
Infrastructure components have to support this goal
• Avoid Single Points of Failure
Redundancy wherever possible
 Security
 Isolation of Users (Customers)
 Stability
Disaster Containment
Large Scale
Performance
Cost
Choose proven, stable Infrastructure Components with
active/active or active/passive Failover Capabilities
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
14
OMCS Design Criteria: Disaster Containment
• In spite of the best prevention, high-impact
infrastructure breakdowns can happen
 Security
 Isolation of Users (Customers)
 Stability
 Disaster Containment
• Backups, of course
Online snapshots, on-site storage, off-site
• Limit the number of VMs that can be affected
Large Scale
Performance
Cost
• Fully segregated Zones
• We accept certain limitations
e.g. limited migration mobility between zones
• Optional Disaster Recovery at a different Data Center
We partition each data center into smaller
self-contained “Zones”
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
15
OMCS Design Criteria: Large Scale
• Design for 100,000 Virtual Machines
Actual number of deployed VMs is approaching 20,000
• Allow very large VMs
100+ vCPUs and multi-TB memory
Typical today: 4 – 32 vCPU and 16 - 128 GB
Typically 4-8 GB per vCPU for Oracle Applications
• Accommodate Multiple concurrent Operators
Start, stop, resize, clone, etc
 Security
 Isolation of Users (Customers)
 Stability
 Disaster Containment
 Large Scale
Performance
Cost
• Service 1,000+ Customers
Each Customer is a Corporation
Quick turn-around time at scale
Oracle VM 3 with multiple Oracle VM Managers
and lots of Server Pools
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
16
OMCS Design Criteria: Performance
• The Deployment Architecture must not substantially limit
performance provided by the underlying raw hardware
• The Networking Stack in VMs must run at GigE+ speeds
• Use the best Virtualization Method available for each use
case
 Security
 Isolation of Users (Customers)
 Stability
 Disaster Containment
 Large Scale
 Performance
Cost
Select Paravirtualization wherever possible.
Otherwise
Hardware Virtualization with PV Drivers
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
17
OMCS Design Criteria: Cost
Reduce Cost through standard building blocks and repeatable Process
• Standard Hardware
Over-provisioning in the interest of Uniformity is acceptable
• “Certified Configuration” images for all Applications and
Databases
Pre-Design a library of standard building blocks
Invest in Tuning and Testing
Re-use these for every customer
• Repeatable Standard Process Cookbook
Automation wherever possible
• Share Infrastructure where Possible
 Security
 Isolation of Users (Customers)
 Stability
 Disaster Containment
 Large Scale
 Performance
 Cost
Segregate where Necessary
Standard Hardware stays fixed for one model year
Certified Configurations with Periodic Updates
Process Cookbooks
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
18
Oracle VM in OMCS
All Server Deployments are
virtualized by default
15,000 + VMs in operation
All Server Virtualization in
OMCS uses Oracle VM.
Oracle VM 3.2.4 in use since 2012
Evangelize, Certify, Prove Security
Experiment with HVM vs. ParaVirt vs. other
Introduce Windows VM
Performance Optimization Work
Oracle VM 2.2.2 in use in some Legacy Zones
Oracle VM 0.9 (pre-release and joint
Beta) Linux Paravirtualized only
Oracle VM Initial Release
Introduced first Xen
based Virtual Machine
Oracle VM Pre-release beta
2005
2006 – 2007
2007
2008
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
2014
19
Oracle VM Deployment in OMCS
Self-contained Zones
Even a catastrophic zone failure of one zone cannot
affect the other zones
Zone
Oracle VM
Manager
All required networking equipment
Switch/Router, Load Balancer, Firewall, Security
Single switch hop from any to any node in zone
(full 10GigE bandwidth, no shared uplinks)
Redundant Storage
Server Pool
NAS per Server Pool
SAN per Zone
Storage
Four Oracle VM Server Pools
Network
with 12 physical servers each (48 total)
One Oracle VM Manager Instance
Also: Legacy Zones
Giant Zones, being migrated / converted
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
20
Global Deployment
Global
Data Center 1
Data Center 2
Zone 3
Enterprise
Manager
7 Data Centers
OVM Manager
Zone 2
OVM Manager
Zone 1
4 Primary
3 Disaster Recovery
Oracle VM Manager
Multiple Zones
Server Pool
per Data Center
Target 100 zones total
Server Pool
Server Pool
Stor
Storage
Storage
age
Shared Service Zone
Redundant or non-critical
Network
Network
Enterprise Manager
Network
One global Instance
Redundant
Shared
Services
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
21
What do OMCS Customers share?
• Shared
– Data Center Real Estate
– Power, Cooling
– Generic Network
Internet, WAN, LAN
– Storage Pool
– Physical Server Pool
• Dedicated
– Customer Network
WAN and/or VPN
DNS Name Space
LAN Subnets and VLAN
– Customer specific Gear
– Customer Storage
Shares, Projects, LUNs
– Machines
Virtual and Physical
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
22
OMCS Server Hardware
• Standard Building Block
Sun Server X4-2 with 24 cores (48 Threads) and 512 GB memory
CPU oversubscription yields approx 50 - 100 vCPUs
Bonded dual 10gigE NIC
• Specialty Configurations
Sun Server X4-4 for high performance applications
Sun Server X4-2L with SSD for low latency transient storage
128GB memory configuration for certain 32-bit VMs
FibreChannel and Infiniband
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
23
Networking
• Bonded dual 10gigE NIC per physical Server
vLANs in the Hypervisor
• Administrative vLANs
Dom-0 login
Cluster Heartbeat
Live Migration
NFS Network
• Separate vLANs per Customer
Public and Private Middle Tier Access
Privileged and Restricted Database Access
Database Cluster Interconnect
• PVLAN and XVLAN
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
24
Engineered Systems
Engineered Systems are an integral part of OMCS Deployments
Exadata Database Machine
Physical Machines only
Oracle Exalogic Elastic Cloud
With Server Virtualization, IB
Partitioning
Oracle SuperCluster
With Hardware Virtualization
Oracle Big Data Appliance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Exalytics In-Memory
Machine
Physical Machines only
25
Standard and Custom Automation
Automate Provisioning of Server, Network, Storage at once
• Custom Scripting against the Oracle VM Manager and
Oracle Enterprise Manager APIs
• Library of Partial and complete Workflows
• Build abstract composite objects (“Instance”) in one
command
• Infrastructure Provisioning
Subnet, IP, DNS, vLAN, Firewall, LoadBalancer
• Application Provisioning
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
26
Do we use Physical Machines?
No, not normally.
Virtualization is the Default
We even virtualize single VMs using
the whole machine
Yes, sometimes.
Third-party Applications
which are not certified as VMs
Specialty Applications
Appliances,
some Infiniband, FC, DAS
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
27
Lessons Learned
• Don’t hesitate to virtualize machines
Oracle VM is mature, stable, enterprise-proven
• Don’t put all your eggs in one basket
Compartmentalize large domains into smaller zones
• Oracle VM Managers can themselves be hosted in VMs
Just no circular References!
• Virtualization is also a great tool to right-size machines
for License Compliance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
28
Re-Sizing VMs
We always stop / start the VM after vCPU or Memory resizing
Reasons:
1. Most Applications choke on resource reductions
2. Most Applications ignore resource increases
3. Those that can deal with it require re-tuning
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
29
Live Migration and Ksplice
We cannot always impose a reboot (downtime) on our customers
Ksplice
Allows us to patch the kernel in the running VM
Live Migration
Used occasionally for Server (HW) Maintenance
Used occasionally for Capacity Rebalancing
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
30
How We Migrate Machines
Our Data Center safety zones prevent Live Migration across Zones
Occasional Live Migrations within a Zone
The majority of moves are Cold Migrations
a) Shutdown
b) Image move/copy across zone via Router
c) Restart
Special Case: LiveMigration-to-self
Useful to re-initialize certain driver functions
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
31
The VM Tetris Problem
32-bit VMs have to reside in lowest 128GB of physical
Memory
Start 32-bit VMs first, then 64-bit
Repeated start-stop cycles of mixed 32-bit and 64-bit
VMs lead to fragmentation
Eventually, no low memory can be found, and VMs fail
to start or live migrate
Our Solution: Limit 32-bit VMs to small physical
machines (128GB memory)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
32
Core Dumps
Core Dumps can become large and take a long time
Our Environment:
Hypervisor core dumps entire memory to local disk
VMs: Core dump to local disk
NetDump no longer used (too slow)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
33
Performance Considerations
• We found that paravirtualized VMs run at nearly
the native speed of the physical machine
• HVM imposes some performance decrease,
can be greatly reduced through PV drivers
• The Network stack is plenty fast enough for
most applications
treatVMs
VMsas
asfully
fully equivalent
Physical
Machines
WeWe
treat
equivalenttoto
physical
machines
The Advantages of OVM far outweigh some negligible performance impact
The Advantages of Oracle VM far outweigh some negligible performance loss
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
34
Designed & Tested Together
Oracle Develops & Uses The Stack Internally
• Oracle Managed Cloud Services – 15,000+ VMs
• Internal Testing
– 22,700 Oracle x86 servers supporting 182,400 Oracle Virtual Machines
– 26,700,000 test and production hours per week
– Workloads: software/hardware development, corporate infrastructure
• Test Environments
– Oracle x86 Server Hardware
– Oracle Storage
– Oracle Operating Systems (Oracle Solaris and Oracle Linux)
– Oracle VM
– Oracle Database , Oracle Middleware, Oracle Applications
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
35
Stay Connected
Join the Oracle VM and Oracle Cloud Communities
@ORCL_Virtualize
Facebook.com/
OracleVirtualization
Blogs.oracle.com
/virtualization
@OracleCloudZone,
#OracleCloud
Facebook.com/
OracleCloudComputing
Oracle VM Group
Blogs.oracle.com/cloud
YouTube.com/
OracleVirtualization
Download: edelivery.oracle.com/oraclevm
Visit us: oracle.com/virtualization
Learn more: oracle.com/cloud
Try now: cloud.oracle.com
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
36
What OMCS Customers Are Saying:
“We are a diverse business with a multitude of internet and media properties growing at
varying rates, as well as frequent acquisitions and divestures with a global reach. Oracle
Managed Cloud Services provides us with the scalability and flexibility to successfully
manage an expansive and complex organization that supports several finance and
accounting management teams.”
Paul Scribano, Vice President, Finance, Mindspark Interactive Inc., part of IAC Search and Media Inc., IAC
“We have a small IT staff, so we must work very efficiently and optimize our resources.
Oracle Managed Cloud Services is critical to our ability to run and optimize our Oracle EBusiness Suite environment. It ensures extremely high availability, timely patches and
maintenance, industry-leading 24/7 support, and world-class system backup and
recovery.”
Cindy Shieh, Information Systems Manager, Greenball Corp.
“Oracle E-Business Suite running through Oracle Managed Cloud Services provides a
compelling value proposition for Genworth Financial. It allows us to take advantage of
industry-leading enterprise applications and gain the expertise of Oracle managing the
applications on Oracle technology.”
JP Raffenot, Director of IT/Applications, Genworth Financial Inc.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
37
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
38