Design Considerations for Large Scale Deployment of Oracle VM in Oracle’s Managed Cloud Service Jose Fernando Niño Higuera Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 3 Program Agenda 1 What is Oracle Managed Cloud Services (OMCS) 2 OMCS Design Criteria 3 How We Use Oracle VM 4 Lessons Learned and Best Practices 5 Benefits of Oracle VM and OMCS Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 4 Oracle Managed Cloud Services (OMCS) ( ) Three Ways for Customers to use Oracle Software 1. Purchase Software Product and self-host 2. Subscribe to Software as a Service (SaaS) 3. Have OMCS host the customer’s software Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 5 Oracle SaaS, PaaS, IaaS Cloud Offerings Infrastructure Platform Software as a Service as a Service as a Service Platform Database Java Database Backup Developer Documents Business Intelligence Mobile Customers Marketing Sales Service People Global Human Resources Talent Management Business Financials Procurement Project Portfolio Management Supply Chain Value Chain execution Product Value Chain Cloud Marketplace Infrastructure Compute Storage Messaging Enterprise Performance Enterprise Planning Financial Planning Social Social Network Social Marketing Social Engagement & Monitoring Social Data & Insight Multi-Tenant Shared Machines* Oracle owns Hardware and Software – Customer pays for usage * typically Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 6 Oracle Managed Cloud Services (OMCS) Applications Technology Extended Managed Cloud Service Managed Cloud Service Managed Cloud Service E-Business Suite PeopleSoft Siebel J D Edwards Hyperion Business Intelligence Commerce Agile Retail Governance Risk & Compliance Fusion Applications Demand Management Markdown Optimization Information Discovery Project Management Beehive Collaboration Transportation Management User Productivity Kit Retail Predictive Application Oracle Database Fusion Middleware Web Center Engineered Systems Identity Management Backup Refresh Upgrade Migration CEMLI Management Business Transaction Monitoring Security PCI & HIPAA Compliance Disaster Recovery Non-Production Environment Service Other Extended Services Single-Tenant Dedicated Machines Customer owns Software - Oracle owns Hardware and manages everything Typically in the Oracle Data Center – but sometimes @customer/partner Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 7 Oracle Managed Cloud Services (OMCS) Why use OMCS – Single Provider: Hardware, Software, Network, Storage, Interoperability If there’s a problem, it’s Oracle’s problem – Expertise: Let Oracle manage Oracle Large expert pool available around the clock Direct Access to Product Development Groups – Leverage: Design & Optimize once, Repeat often Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 8 Oracle Virtualization Strategy • At the core of Oracle’s cloud strategy • Integrated VM lifecycle & cloud management solution with Oracle Enterprise Manager • Supports both x86 and SPARC • Integrated with OpenStack • Cloud platform for Oracle & Non-Oracle applications – Supports Oracle Linux, Oracle Solaris, Microsoft Windows Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 9 About Oracle VM Oracle VM Templates Oracle Real Application Clusters(RAC) Oracle E-Business Suites Oracle JD Edward EnterpriseOne ……. Oracle Linux Oracle Solaris Oracle Virtual Networking Integrated for scale & ease of deployment Boosts Performance by 30%. X86 and SPARC Oracle Enterprise Manager Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | End-to-end ManagementPhysical to Virtual To Cloud 10 OMCS Design Criteria 1. 2. 3. 4. 5. 6. 7. Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 11 OMCS Design Criteria: Security • Protect the Customer’s Data under all circumstances • Rigorous Security Review Process Deployment Architecture Need-to-know access to data Authentication, Authorization & Audit for all Activities • Guard against unauthorized Access Intrusion Detection • Perimeter Security around the Deployment Cloud Additional Segregation & Firewalls within • Security Patches via Routine Maintenance Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Virtualization allows us the luxury of having separate Machines for each Tier within each Instance Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 12 OMCS Design Criteria: Isolation of Users (Customers) • Strong Separation between Customers is mandatory One customer must never see another’s data or activities • After years of testing and Operation – Oracle VM Virtual Machines are proven just as safe as physical machines – Virtual disks with backing store from a storage pool are safe • Flat Network alone doesn’t isolate enough for OMCS iptables + ebtables + Perimeter Firewall provide Security vLAN + Internal Firewall provide Isolation Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Oracle VM Virtual Machines plus vLANs meet our Segregation Requirements Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 13 OMCS Design Criteria: Stability • Preservation of Data must never be in question • What matters is the Customer’s App Infrastructure components have to support this goal • Avoid Single Points of Failure Redundancy wherever possible Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Choose proven, stable Infrastructure Components with active/active or active/passive Failover Capabilities Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 14 OMCS Design Criteria: Disaster Containment • In spite of the best prevention, high-impact infrastructure breakdowns can happen Security Isolation of Users (Customers) Stability Disaster Containment • Backups, of course Online snapshots, on-site storage, off-site • Limit the number of VMs that can be affected Large Scale Performance Cost • Fully segregated Zones • We accept certain limitations e.g. limited migration mobility between zones • Optional Disaster Recovery at a different Data Center We partition each data center into smaller self-contained “Zones” Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 15 OMCS Design Criteria: Large Scale • Design for 100,000 Virtual Machines Actual number of deployed VMs is approaching 20,000 • Allow very large VMs 100+ vCPUs and multi-TB memory Typical today: 4 – 32 vCPU and 16 - 128 GB Typically 4-8 GB per vCPU for Oracle Applications • Accommodate Multiple concurrent Operators Start, stop, resize, clone, etc Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost • Service 1,000+ Customers Each Customer is a Corporation Quick turn-around time at scale Oracle VM 3 with multiple Oracle VM Managers and lots of Server Pools Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 16 OMCS Design Criteria: Performance • The Deployment Architecture must not substantially limit performance provided by the underlying raw hardware • The Networking Stack in VMs must run at GigE+ speeds • Use the best Virtualization Method available for each use case Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Select Paravirtualization wherever possible. Otherwise Hardware Virtualization with PV Drivers Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 17 OMCS Design Criteria: Cost Reduce Cost through standard building blocks and repeatable Process • Standard Hardware Over-provisioning in the interest of Uniformity is acceptable • “Certified Configuration” images for all Applications and Databases Pre-Design a library of standard building blocks Invest in Tuning and Testing Re-use these for every customer • Repeatable Standard Process Cookbook Automation wherever possible • Share Infrastructure where Possible Security Isolation of Users (Customers) Stability Disaster Containment Large Scale Performance Cost Segregate where Necessary Standard Hardware stays fixed for one model year Certified Configurations with Periodic Updates Process Cookbooks Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 18 Oracle VM in OMCS All Server Deployments are virtualized by default 15,000 + VMs in operation All Server Virtualization in OMCS uses Oracle VM. Oracle VM 3.2.4 in use since 2012 Evangelize, Certify, Prove Security Experiment with HVM vs. ParaVirt vs. other Introduce Windows VM Performance Optimization Work Oracle VM 2.2.2 in use in some Legacy Zones Oracle VM 0.9 (pre-release and joint Beta) Linux Paravirtualized only Oracle VM Initial Release Introduced first Xen based Virtual Machine Oracle VM Pre-release beta 2005 2006 – 2007 2007 2008 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 2014 19 Oracle VM Deployment in OMCS Self-contained Zones Even a catastrophic zone failure of one zone cannot affect the other zones Zone Oracle VM Manager All required networking equipment Switch/Router, Load Balancer, Firewall, Security Single switch hop from any to any node in zone (full 10GigE bandwidth, no shared uplinks) Redundant Storage Server Pool NAS per Server Pool SAN per Zone Storage Four Oracle VM Server Pools Network with 12 physical servers each (48 total) One Oracle VM Manager Instance Also: Legacy Zones Giant Zones, being migrated / converted Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 20 Global Deployment Global Data Center 1 Data Center 2 Zone 3 Enterprise Manager 7 Data Centers OVM Manager Zone 2 OVM Manager Zone 1 4 Primary 3 Disaster Recovery Oracle VM Manager Multiple Zones Server Pool per Data Center Target 100 zones total Server Pool Server Pool Stor Storage Storage age Shared Service Zone Redundant or non-critical Network Network Enterprise Manager Network One global Instance Redundant Shared Services Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 21 What do OMCS Customers share? • Shared – Data Center Real Estate – Power, Cooling – Generic Network Internet, WAN, LAN – Storage Pool – Physical Server Pool • Dedicated – Customer Network WAN and/or VPN DNS Name Space LAN Subnets and VLAN – Customer specific Gear – Customer Storage Shares, Projects, LUNs – Machines Virtual and Physical Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 22 OMCS Server Hardware • Standard Building Block Sun Server X4-2 with 24 cores (48 Threads) and 512 GB memory CPU oversubscription yields approx 50 - 100 vCPUs Bonded dual 10gigE NIC • Specialty Configurations Sun Server X4-4 for high performance applications Sun Server X4-2L with SSD for low latency transient storage 128GB memory configuration for certain 32-bit VMs FibreChannel and Infiniband Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 23 Networking • Bonded dual 10gigE NIC per physical Server vLANs in the Hypervisor • Administrative vLANs Dom-0 login Cluster Heartbeat Live Migration NFS Network • Separate vLANs per Customer Public and Private Middle Tier Access Privileged and Restricted Database Access Database Cluster Interconnect • PVLAN and XVLAN Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 24 Engineered Systems Engineered Systems are an integral part of OMCS Deployments Exadata Database Machine Physical Machines only Oracle Exalogic Elastic Cloud With Server Virtualization, IB Partitioning Oracle SuperCluster With Hardware Virtualization Oracle Big Data Appliance Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Exalytics In-Memory Machine Physical Machines only 25 Standard and Custom Automation Automate Provisioning of Server, Network, Storage at once • Custom Scripting against the Oracle VM Manager and Oracle Enterprise Manager APIs • Library of Partial and complete Workflows • Build abstract composite objects (“Instance”) in one command • Infrastructure Provisioning Subnet, IP, DNS, vLAN, Firewall, LoadBalancer • Application Provisioning Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 26 Do we use Physical Machines? No, not normally. Virtualization is the Default We even virtualize single VMs using the whole machine Yes, sometimes. Third-party Applications which are not certified as VMs Specialty Applications Appliances, some Infiniband, FC, DAS Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 27 Lessons Learned • Don’t hesitate to virtualize machines Oracle VM is mature, stable, enterprise-proven • Don’t put all your eggs in one basket Compartmentalize large domains into smaller zones • Oracle VM Managers can themselves be hosted in VMs Just no circular References! • Virtualization is also a great tool to right-size machines for License Compliance Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 28 Re-Sizing VMs We always stop / start the VM after vCPU or Memory resizing Reasons: 1. Most Applications choke on resource reductions 2. Most Applications ignore resource increases 3. Those that can deal with it require re-tuning Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 29 Live Migration and Ksplice We cannot always impose a reboot (downtime) on our customers Ksplice Allows us to patch the kernel in the running VM Live Migration Used occasionally for Server (HW) Maintenance Used occasionally for Capacity Rebalancing Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 30 How We Migrate Machines Our Data Center safety zones prevent Live Migration across Zones Occasional Live Migrations within a Zone The majority of moves are Cold Migrations a) Shutdown b) Image move/copy across zone via Router c) Restart Special Case: LiveMigration-to-self Useful to re-initialize certain driver functions Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 31 The VM Tetris Problem 32-bit VMs have to reside in lowest 128GB of physical Memory Start 32-bit VMs first, then 64-bit Repeated start-stop cycles of mixed 32-bit and 64-bit VMs lead to fragmentation Eventually, no low memory can be found, and VMs fail to start or live migrate Our Solution: Limit 32-bit VMs to small physical machines (128GB memory) Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 32 Core Dumps Core Dumps can become large and take a long time Our Environment: Hypervisor core dumps entire memory to local disk VMs: Core dump to local disk NetDump no longer used (too slow) Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 33 Performance Considerations • We found that paravirtualized VMs run at nearly the native speed of the physical machine • HVM imposes some performance decrease, can be greatly reduced through PV drivers • The Network stack is plenty fast enough for most applications treatVMs VMsas asfully fully equivalent Physical Machines WeWe treat equivalenttoto physical machines The Advantages of OVM far outweigh some negligible performance impact The Advantages of Oracle VM far outweigh some negligible performance loss Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 34 Designed & Tested Together Oracle Develops & Uses The Stack Internally • Oracle Managed Cloud Services – 15,000+ VMs • Internal Testing – 22,700 Oracle x86 servers supporting 182,400 Oracle Virtual Machines – 26,700,000 test and production hours per week – Workloads: software/hardware development, corporate infrastructure • Test Environments – Oracle x86 Server Hardware – Oracle Storage – Oracle Operating Systems (Oracle Solaris and Oracle Linux) – Oracle VM – Oracle Database , Oracle Middleware, Oracle Applications Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 35 Stay Connected Join the Oracle VM and Oracle Cloud Communities @ORCL_Virtualize Facebook.com/ OracleVirtualization Blogs.oracle.com /virtualization @OracleCloudZone, #OracleCloud Facebook.com/ OracleCloudComputing Oracle VM Group Blogs.oracle.com/cloud YouTube.com/ OracleVirtualization Download: edelivery.oracle.com/oraclevm Visit us: oracle.com/virtualization Learn more: oracle.com/cloud Try now: cloud.oracle.com Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 36 What OMCS Customers Are Saying: “We are a diverse business with a multitude of internet and media properties growing at varying rates, as well as frequent acquisitions and divestures with a global reach. Oracle Managed Cloud Services provides us with the scalability and flexibility to successfully manage an expansive and complex organization that supports several finance and accounting management teams.” Paul Scribano, Vice President, Finance, Mindspark Interactive Inc., part of IAC Search and Media Inc., IAC “We have a small IT staff, so we must work very efficiently and optimize our resources. Oracle Managed Cloud Services is critical to our ability to run and optimize our Oracle EBusiness Suite environment. It ensures extremely high availability, timely patches and maintenance, industry-leading 24/7 support, and world-class system backup and recovery.” Cindy Shieh, Information Systems Manager, Greenball Corp. “Oracle E-Business Suite running through Oracle Managed Cloud Services provides a compelling value proposition for Genworth Financial. It allows us to take advantage of industry-leading enterprise applications and gain the expertise of Oracle managing the applications on Oracle technology.” JP Raffenot, Director of IT/Applications, Genworth Financial Inc. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 37 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 38
© Copyright 2024