1. technology and computing
2. data centers

Proven Infrastructure Guide
EMC VSPEX PRIVATE CLOUD
VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Microsoft Windows Server 2012 R2, EMC VNX Series, and EMC Powered
Backup
EMC VSPEX
Abstract
This document describes the EMC® VSPEX® Proven
Infrastructure solution for private cloud deployments with
VMware vSphere 5.5, EMC VNX®,Series , and EMC Powered
Backup for up to 1,000 virtual machines.
July, 2014
EMC Confidential
Copyright © 2014 EMC Corporation. All rights reserved. Published in the USA.
Published July, 2014
EMC believes the information in this publication is accurate of its publication date.
The information is subject to change without notice.
The information in this publication is provided as is. EMC Corporation makes no
representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or
fitness for a particular purpose. Use, copying, and distribution of any EMC software
described in this publication requires an applicable software license.
EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC
Corporation in the United States and other countries. All other trademarks used
herein are the property of their respective owners.
For the most up-to-date regulatory document for your product line, go to the technical
documentation and advisories section on the EMC Online Support website.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Microsoft Windows Server 2012 R2, EMC VNX Series, and EMC Powered
Backup - Proven Infrastructure Guide
Part Number 12076.3
2
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Contents
Contents
Chapter 1
Executive Summary
13
Introduction ............................................................................................................. 14
Target audience ........................................................................................................ 14
Document purpose ................................................................................................... 14
Business needs ........................................................................................................ 15
Chapter 2
Solution Overview
17
Introduction ............................................................................................................. 18
Virtualization ............................................................................................................ 18
Compute .................................................................................................................. 18
Network .................................................................................................................... 19
Storage..................................................................................................................... 19
EMC VNX Series ................................................................................................... 20
EMC backup and recovery .................................................................................... 26
Chapter 3
Solution Technology Overview
29
Overview .................................................................................................................. 30
Key components ....................................................................................................... 31
Virtualization ............................................................................................................ 32
Overview.............................................................................................................. 32
VMware vSphere 5.5 ............................................................................................ 32
New VMware vSphere 5.5 features ....................................................................... 32
VMware vSphere with Operations Management (vSOM)....................................... 33
VMware vCenter ................................................................................................... 35
VMware vSphere High-Availability ....................................................................... 35
EMC Virtual Storage Integrator for VMware ........................................................... 35
VNX VMware vStorage API for Array Integration support ....................................... 36
Compute .................................................................................................................. 36
Network .................................................................................................................... 39
Overview.............................................................................................................. 39
Storage..................................................................................................................... 41
Overview.............................................................................................................. 41
EMC VNX series.................................................................................................... 41
VNX Snapshots .................................................................................................... 42
VNX SnapSure ..................................................................................................... 42
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
3
Contents
EMC Confidential
VNX Virtual Provisioning ...................................................................................... 43
VNX FAST Cache ................................................................................................... 48
VNX FAST VP ........................................................................................................ 48
vCloud Networking and Security .......................................................................... 48
VNX file shares .................................................................................................... 49
ROBO ................................................................................................................... 49
Backup and recovery ................................................................................................ 49
Overview.............................................................................................................. 49
EMC Avamar deduplication .................................................................................. 50
EMC Data Domain deduplication storage systems ............................................... 50
VMware vSphere data protection ......................................................................... 50
vSphere Replication ............................................................................................. 50
EMC RecoverPoint ................................................................................................ 51
Other technologies ................................................................................................... 51
Overview.............................................................................................................. 51
VMware vCloud Automation Center ...................................................................... 51
VMware vCenter Operations Management Suite .................................................. 52
VMware vCenter Single Sign On ........................................................................... 53
Public-key infrastructure ...................................................................................... 53
EMC Storage Analytics for EMC VNX ..................................................................... 54
PowerPath/VE (for block) ..................................................................................... 54
EMC XtremCache ................................................................................................. 54
Chapter 4
Solution Architecture Overview
57
Overview .................................................................................................................. 58
Solution architecture ................................................................................................ 58
Overview.............................................................................................................. 58
Logical architecture ............................................................................................. 59
Key components .................................................................................................. 60
Hardware resources ............................................................................................. 62
Software resources .............................................................................................. 66
Server configuration guidelines ................................................................................ 67
Overview.............................................................................................................. 67
Ivy Bridge updates ............................................................................................... 67
VMware vSphere memory virtualization for VSPEX ............................................... 70
Memory configuration guidelines ......................................................................... 71
Network configuration guidelines ............................................................................. 71
Overview.............................................................................................................. 71
VLAN.................................................................................................................... 72
4
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Contents
Enable jumbo frames (for iSCSI, FCoE, and NFS) .................................................. 74
Link aggregation (for NFS) .................................................................................... 75
Storage configuration guidelines .............................................................................. 75
Overview.............................................................................................................. 75
VMware vSphere storage virtualization for VSPEX ................................................ 78
VSPEX storage building blocks............................................................................. 78
VSPEX private cloud validated maximums ........................................................... 80
High-availability and failover .................................................................................... 88
Overview.............................................................................................................. 88
Virtualization layer ............................................................................................... 88
Compute layer ..................................................................................................... 88
Network layer....................................................................................................... 89
Storage layer ....................................................................................................... 90
Validation test profile ............................................................................................... 91
Profile characteristics .......................................................................................... 91
Backup and recovery configuration guidelines.......................................................... 91
Sizing guidelines ...................................................................................................... 91
Reference workload .................................................................................................. 92
Overview.............................................................................................................. 92
Defining the reference workload .......................................................................... 92
Applying the reference workload .............................................................................. 93
Overview.............................................................................................................. 93
Example 1: Custom-built application .................................................................. 93
Example 2: Point of sale system .......................................................................... 93
Example 3: Web server ........................................................................................ 94
Example 4: Decision-support database ............................................................... 94
Summary of examples ......................................................................................... 94
Implementing the solution........................................................................................ 95
Overview.............................................................................................................. 95
Resource types .................................................................................................... 95
CPU resources ..................................................................................................... 95
Memory resources ............................................................................................... 95
Network resources ............................................................................................... 96
Storage resources ................................................................................................ 96
Implementation summary .................................................................................... 97
Quick assessment .................................................................................................... 98
Overview.............................................................................................................. 98
CPU requirements ................................................................................................ 98
Memory requirements .......................................................................................... 98
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
5
Contents
EMC Confidential
Storage performance requirements ...................................................................... 99
I/O operations per second ................................................................................... 99
I/O size ................................................................................................................ 99
I/O latency .........................................................................................................100
Storage capacity requirements ..........................................................................100
Determining equivalent reference virtual machines ...........................................100
Fine-tuning hardware resources .........................................................................107
EMC VSPEX Sizing Tool ......................................................................................109
Chapter 5
VSPEX Configuration Guidelines
111
Overview ................................................................................................................112
Pre-deployment tasks .............................................................................................113
Overview............................................................................................................113
Deployment prerequisites ..................................................................................113
Customer configuration data ..................................................................................115
Prepare switches, connect network, and configure switches ...................................115
Overview............................................................................................................115
Prepare network switches ..................................................................................115
Configure infrastructure network ........................................................................115
Configure VLANs ................................................................................................117
Configure jumbo frames (iSCSI and NFS only) ....................................................117
Complete network cabling .................................................................................118
Prepare and configure storage array .......................................................................118
VNX configuration for block protocols ................................................................118
VNX configuration for file protocols ....................................................................121
FAST VP configuration ........................................................................................128
FAST Cache configuration ..................................................................................130
Install and configure vSphere hosts........................................................................133
Overview............................................................................................................133
Install ESXi ........................................................................................................133
Configure ESXi networking .................................................................................133
Install and configure PowerPath/VE (block only) ................................................134
Connect VMware datastores ..............................................................................134
Plan virtual machine memory allocations ...........................................................134
Install and configure SQL server database ..............................................................137
Overview............................................................................................................137
Create a virtual machine for SQL Server .............................................................137
Install Microsoft Windows on the virtual machine ..............................................137
Install SQL Server ..............................................................................................138
6
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Contents
Configure database for VMware vCenter ............................................................138
Configure database for VMware Update Manager...............................................138
Install and configure VMware vCenter server ..........................................................139
Overview............................................................................................................139
Create the vCenter host virtual machine .............................................................140
Install vCenter guest OS .....................................................................................140
Create vCenter ODBC connections .....................................................................140
Install vCenter Server .........................................................................................140
Apply vSphere license keys................................................................................140
Install the EMC VSI plug-in .................................................................................141
Create a virtual machine in vCenter ....................................................................141
Perform partition alignment, and assign File Allocation Unite Size .....................141
Create a template virtual machine .....................................................................141
Deploy virtual machines from the template virtual machine ...............................141
Summary ................................................................................................................141
Chapter 6
Verifying the Solution
143
Overview ................................................................................................................144
Post-install checklist ..............................................................................................145
Deploy and test a single virtual server ....................................................................145
Verify the redundancy of the solution components .................................................145
Block environments ...........................................................................................145
File environments ..............................................................................................146
Chapter 7
System Monitoring
147
Overview ................................................................................................................148
Key areas to monitor ...............................................................................................148
Performance baseline ........................................................................................148
Servers ..............................................................................................................149
Networking ........................................................................................................149
Storage ..............................................................................................................150
VNX resource monitoring guidelines .......................................................................150
Monitoring block storage resources ...................................................................150
Monitoring file storage resources .......................................................................158
Summary ................................................................................................................163
Appendix A Bill of Materials
165
Bill of materials ......................................................................................................166
Appendix B Customer Configuration Data Sheet
175
Customer configuration data sheet .........................................................................176
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
7
Contents
EMC Confidential
Appendix C Server Resource Component Worksheet
179
Server resources component worksheet .................................................................180
Appendix D References
181
References .............................................................................................................182
EMC documentation ..........................................................................................182
Other documentation.........................................................................................182
Appendix E About VSPEX
183
About VSPEX ..........................................................................................................184
8
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Contents
Figures
Figure 1.
Next-Generation VNX with multicore optimization................................ 21
Figure 2.
Active/active processors increase performance, resiliency, and
efficiency ............................................................................................. 22
Figure 3.
New Unisphere Management Suite ...................................................... 23
Figure 4.
Storage processor utilization using Windows deduplication ................ 25
Figure 5.
Disk IOPS using Windows deduplication.............................................. 25
Figure 6.
Disk latency using Windows deduplication .......................................... 26
Figure 7.
EMC backup and recovery solutions .................................................... 27
Figure 8.
Private cloud components ................................................................... 30
Figure 9.
Compute layer flexibility ...................................................................... 37
Figure 10.
Example of highly available network design—for block ........................ 39
Figure 11.
Example of highly available network design—for file ........................... 40
Figure 12.
Storage pool rebalance progress ......................................................... 44
Figure 13.
Thin LUN space utilization ................................................................... 45
Figure 14.
Examining storage pool space utilization............................................. 46
Figure 15.
Defining storage pool utilization thresholds ........................................ 47
Figure 16.
Defining automated notifications (for block) ........................................ 47
Figure 17.
Logical architecture for block storage .................................................. 59
Figure 18.
Logical architecture for file storage ...................................................... 60
Figure 19.
Ivy Bridge processor guidance ............................................................. 68
Figure 20.
Hypervisor memory consumption ........................................................ 70
Figure 21.
Required networks for block storage.................................................... 73
Figure 22.
Required networks for file storage ....................................................... 74
Figure 23.
VMware virtual disk types .................................................................... 78
Figure 24.
Storage layout building block for 13 virtual machines ......................... 79
Figure 25.
Storage layout building block for 125 virtual machines ....................... 79
Figure 26.
Storage layout for 200 virtual machines using VNX5200...................... 81
Figure 27.
Storage layout for 300 virtual machines using VNX5400...................... 82
Figure 28.
Storage layout for 600 virtual machines using VNX 5600 ..................... 84
Figure 29.
Storage layout for 1,000 virtual machines using VNX 5800 .................. 86
Figure 30.
Maximum scale levels and entry points of different arrays ................... 88
Figure 31.
High availability at the virtualization layer ........................................... 88
Figure 32.
Redundant power supplies .................................................................. 89
Figure 33.
Network layer high availability (VNX) – Block storage .......................... 89
Figure 34.
Network layer high availability (VNX) - File storage............................... 90
Figure 35.
VNX series high availability ................................................................. 90
Figure 36.
Resource pool flexibility ...................................................................... 94
Figure 37.
Required resource from the reference virtual machine pool ...............101
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
9
Contents
10
EMC Confidential
Figure 38.
Aggregate resource requirements – stage 1 .......................................103
Figure 39.
Pool configuration – stage 1 ..............................................................103
Figure 40.
Aggregate resource requirements - stage 2 ........................................104
Figure 41.
Pool configuration – stage 2 ..............................................................105
Figure 42.
Aggregate resource requirements for stage 3 .....................................106
Figure 43.
Pool configuration – stage 3 ..............................................................107
Figure 44.
Customizing server resources ............................................................107
Figure 45.
Sample network architecture – Block storage ....................................116
Figure 46.
Sample Ethernet network architecture – File storage .........................117
Figure 47.
Network settings For file dialog box ...................................................123
Figure 48.
Create Interface dialog box ................................................................124
Figure 49.
Create file system dialog box .............................................................127
Figure 50.
Direct Writes Enabled checkbox.........................................................128
Figure 51.
Storage Pool Properties dialog box ....................................................129
Figure 52.
Manage Auto-Tiering dialog box ........................................................129
Figure 53.
Storage System Properties dialog box................................................130
Figure 54.
Create FAST Cache dialog box ............................................................131
Figure 55.
Advanced tab in the Create Storage Pool dialog box ..........................132
Figure 56.
Advanced tab in the Storage Pool Properties dialog box ....................132
Figure 57.
Virtual machine memory settings ......................................................136
Figure 58.
Storage Pool Alerts ............................................................................151
Figure 59.
Storage pools panel ..........................................................................152
Figure 60.
LUN Properties dialog box .................................................................153
Figure 61.
Monitoring and Alerts panel. .............................................................154
Figure 62.
IOPS on the LUNs ..............................................................................155
Figure 63.
IOPS on the drives .............................................................................156
Figure 64.
Latency on the LUNs ..........................................................................156
Figure 65.
SP Utilization .....................................................................................158
Figure 66.
Data Mover statistics .........................................................................159
Figure 67.
Front-end Data Mover network statistics ............................................159
Figure 68.
Storage Pools for File panel ...............................................................160
Figure 69.
File Systems panel.............................................................................160
Figure 70.
File System property panel ................................................................161
Figure 71.
File system performance panel ..........................................................162
Figure 72.
File storage all performance panel .....................................................162
Figure 73.
List of components used in the VSPEX solution
for 200 virtual machines ....................................................................166
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Contents
Tables
Table 1.
VNX customer benefits ........................................................................ 41
Table 2.
Thresholds and settings under VNX OE Block Release 33 .................... 48
Table 3.
Solution hardware ............................................................................... 62
Table 4.
Solution software ................................................................................ 66
Table 5.
Hardware resources for the compute layer ........................................... 69
Table 6.
Hardware resources for network .......................................................... 72
Table 7.
Hardware resources for storage ........................................................... 76
Table 8.
Number of disks required for different number of virtual machines ...... 80
Table 9.
Profile characteristics .......................................................................... 91
Table 10.
Virtual machine characteristics............................................................ 92
Table 11.
Blank worksheet row ........................................................................... 98
Table 12.
Reference virtual machine resources .................................................100
Table 13.
Example worksheet row .....................................................................101
Table 14.
Example applications – stage 1 .........................................................102
Table 15.
Example applications -stage 2 ...........................................................103
Table 16.
Example applications - stage 3 ..........................................................105
Table 17.
Server resource component totals .....................................................108
Table 18.
Deployment process overview ...........................................................112
Table 19.
Tasks for pre-deployment ..................................................................113
Table 20.
Deployment prerequisites checklist ...................................................113
Table 21.
Tasks for switch and network configuration .......................................115
Table 22.
Tasks for VNX configuration ...............................................................118
Table 23.
Storage allocation table for block data ..............................................120
Table 24.
Tasks for storage configuration..........................................................121
Table 25.
Storage allocation table for file ..........................................................124
Table 26.
Tasks for server installation ...............................................................133
Table 27.
Tasks for SQL Server database setup .................................................137
Table 28.
Tasks for vCenter configuration .........................................................139
Table 29.
Tasks for testing the installation ........................................................144
Table 30.
List of components used in the VSPEX solution
for 300 virtual machines ....................................................................168
Table 31.
List of components used in the VSPEX solution
for 600 virtual machines ....................................................................170
Table 32.
List of components used in the VSPEX solution
for 1000 virtual machines..................................................................172
Table 33.
Common server information ..............................................................176
Table 34.
ESXi server information .....................................................................176
Table 35.
Array information...............................................................................177
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
11
Contents
12
EMC Confidential
Table 36.
Network infrastructure information ....................................................177
Table 37.
VLAN information ..............................................................................178
Table 38.
Service accounts ...............................................................................178
Table 39.
Blank worksheet for server resource totals ........................................180
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 1: Executive Summary
Chapter 1
Executive Summary
This chapter presents the following topics:
Introduction .............................................................................................................14
Target audience .......................................................................................................14
Document purpose ...................................................................................................14
Business needs ........................................................................................................15
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
13
Chapter 1: Executive Summary
EMC Confidential
Introduction
EMC® VSPEX® validated and modular architectures are built with proven best-ofbreed technologies to create complete virtualization solutions that enable you to
make an informed decision in the hypervisor, compute, and networking layers. VSPEX
helps to reduce virtualization planning and configuration burdens. When embarking
on server virtualization, virtual desktop deployment, or IT consolidation, VSPEX
accelerates your IT transformation by enabling faster deployments, expanded
choices, greater efficiency, and lower risk.
This document is a comprehensive guide to the technical aspects of this solution.
Server capacity is provided in generic terms for required minimums of CPU, memory,
and network interfaces; the customer is free to select the server and networking
hardware that meet or exceed the stated minimums.
Target audience
The readers of this document must have the necessary training and background to
install and configure VMware vSphere 5.5, EMC VNX® series storage systems, and
associated infrastructure as required by this implementation. External references are
provided where applicable, and readers should be familiar with these documents.
Readers should also be familiar with the infrastructure and database security policies
of the customer’s existing installation.
Individuals selling and sizing a VMware Private Cloud infrastructure must pay
particular attention to the first four chapters of this document. After purchase,
implementers of the solution should focus on the configuration guidelines in Chapter
5, the solution validation in Chapter 6, and the appropriate references and
appendices.
Document purpose
This document includes an initial introduction to the VSPEX architecture, an
explanation of how to modify the architecture for specific engagements, and
instructions on how to effectively deploy and monitor the system.
The VSPEX Private Cloud architecture provides customers with a modern system
capable of hosting many virtual machines at a consistent performance level. This
solution runs on the VMware vSphere virtualization layer backed by highly available
VNX family of storage. The compute and network components, which are defined by
the VSPEX partners, are designed to be redundant and sufficiently powerful to handle
the processing and data needs of the virtual machine environment.
The 200, 300, 600, and 1,000 virtual machine environments discussed are based on
a defined reference workload. Since not every virtual machine has the same
requirements, this document contains methods and guidance to adjust your system
to be cost-effective when deployed. For smaller environments, solutions for up to 125
virtual machines based on the EMC VNXe® series are described in EMC VSPEX Private
Cloud: VMware vSphere 5.5 for up to 125 Virtual Machines.
14
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 1: Executive Summary
A private cloud architecture is a complex system offering. This document facilitates
setup by providing prerequisite software and hardware material lists, step-by-step
sizing guidance and worksheets, and verified deployment steps. After the last
component has been installed, validation tests and monitoring instructions ensure
that your system is running properly. Following the instructions in this document
ensures an efficient and painless journey to the cloud.
Business needs
VSPEX solutions are built with proven best-of-breed technologies to create complete
virtualization solutions that enable you to make an informed decision in the
hypervisor, server, and networking layers.
Business applications are moving into consolidated compute, network, and storage
environments. EMC VSPEX Private Cloud solutions using VMware reduce the
complexity of configuring every component of a traditional deployment model. The
complexity of integration management is reduced while maintaining the application
design and implementation options. Administration is unified, while process
separation can be adequately controlled and monitored. The business needs for the
VSPEX Private Cloud solutions for VMware architectures are listed as follows:

Provide an end-to-end virtualization solution to effectively use the capabilities
of the unified infrastructure components.

Provide a VSPEX Private Cloud solution for VMware for efficiently virtualizing up
to 1,000 virtual machines for varied customer use cases.

Provide a reliable, flexible, and scalable reference design.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
15
Chapter 1: Executive Summary
16
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
EMC Confidential
Chapter 2: Solution Overview
Chapter 2
Solution Overview
This chapter presents the following topics:
Introduction .............................................................................................................18
Virtualization ...........................................................................................................18
Compute ..................................................................................................................18
Network ...................................................................................................................19
Storage ....................................................................................................................19
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
17
Chapter 2: Solution Overview
EMC Confidential
Introduction
The VSPEX Private Cloud for VMware vSphere 5.5 provides complete system
architecture capable of supporting up to 1,000 virtual machines with a redundant
server and network topology and highly available storage. The core components that
make up this solution are virtualization, compute, storage, and networking.
Virtualization
VMware vSphere is the leading virtualization platform in the industry. For years, it has
provided flexibility and cost savings to the end users by enabling the consolidation of
large, inefficient server farms into nimble, reliable cloud infrastructures. The core
VMware vSphere components are the VMware vSphere hypervisor and the VMware
vCenter Server for system management.
The VMware hypervisor runs on a dedicated server and allows multiple operating
systems to run on the system at one time as virtual machines. These hypervisor
systems can be connected to operate in a clustered configuration. These clustered
configurations are then managed as a larger resource pool through VMware vCenter,
and allow for dynamic allocation of CPU, memory, and storage across the cluster.
Features such as VMware vMotion, which allows a virtual machine to move between
different servers with no disruption to the operating system, and Distributed Resource
Scheduler (DRS) which performs vMotions automatically to balance load, make
vSphere a solid business choice.
With vSphere 5.5, a VMware-virtualized environment can host virtual machines with
up to 64 virtual CPUs and 1 TB of virtual random access memory (RAM).
Compute
VSPEX provides the flexibility to design and implement the server components that
you select. The infrastructure must conform to the following attributes:
18

Sufficient cores and memory to support the required number and types of
virtual machines.

Sufficient network connections to enable redundant connectivity to the system
switches.

Excess capacity to withstand a server failure and failover within the
environment.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 2: Solution Overview
Network
VSPEX provides the flexibility to design and implement the customer’s choice of
network components. The infrastructure must conform to the following attributes:

Redundant network links for the hosts, switches, and storage.

Traffic isolation based on industry-accepted best practices.

Support for link aggregation.

IP network switches used to implement this reference architecture must have a
minimum non-blocking backplane capacity which is sufficient for the target
number of virtual machines and their associated workloads. Enterprise-class
switches with advanced features such as Quality of Service are highly
recommended.
Storage
The EMC VNX storage family is the leading shared storage platform in the industry.
VNX provides both file and block access with a broad feature set, which makes it an
ideal choice for any private cloud implementation.
VNX storage includes the following components that are sized for the stated reference
architecture workload:

Host Bus Adapter ports (for block) – Provide host connectivity via fabric to the
array.

Storage processors (SP) – The compute components of the storage array, which
are used for all aspects of data moving into, out of, and between arrays.

Disk drives – Disk spindles and solid state drives (SSDs) that contain the host
or application data and their enclosures.

Data Movers (for file) – Front-end appliances that provide file services to hosts
(optional if CIFS/SMB, NFS services are provided).
The 200, 300, 600, and 1,000 virtual machine VMware Private Cloud solutions
described in this document are based on the VNX5200, VNX5400, VNX5600, and the
VNX5800 storage arrays respectively. The VNX5200 can support a maximum of 125
drives, the VNX5400 can support a maximum of 250 drives, the VNX5600 can host up
to 500 drives, and the VNX5800 can host up to 750 drives.
The EMC VNX series supports a wide range of business class features ideal for the
private cloud environment, including:

Fully Automated Storage Tiering for Virtual Pools (FAST VP™)

FAST Cache

File-level data deduplication/compression

Block deduplication


Thin provisioning
Replication
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
19
Chapter 2: Solution Overview
EMC VNX Series
EMC Confidential

Snapshots/checkpoints

File-Level Retention (FLR)

Quota management

Block compression
Features and Enhancements
The EMC VNX flash-optimized unified storage platform delivers innovation and
enterprise capabilities for file, block, and object storage in a single, scalable, and
easy-to-use solution. Ideal for mixed workloads in physical or virtual environments,
VNX combines powerful and flexible hardware with advanced efficiency,
management, and protection software to meet the demanding needs of today’s
virtualized application environments.
VNX includes many features and enhancements designed and built upon the first
generation’s success. These features and enhancements include:

More capacity through the use of multicore optimization with Multicore
Cache, Multicore RAID, and Multicore FAST Cache (MCx)

Greater efficiency with a flash-optimized hybrid array

Better protection by increasing application availability with active/active
storage processors

Easier administration and deployment by increasing productivity with a new
Unisphere® Management Suite
VSPEX is built with the next generation of VNX to deliver even greater efficiency,
performance, and scale than ever before.
Flash-optimized hybrid array
VNX is a flash-optimized hybrid array that provides automated tiering to deliver the
best performance to your critical data, while intelligently moving less frequently
accessed data to lower-cost disks.
In this hybrid approach, a small percentage of flash drives in the overall system can
provide a high percentage of the overall IOPS. A flash-optimized VNX takes full
advantage of the low latency of flash to deliver cost-saving optimization and high
performance scalability. The EMC Fully Automated Storage Tiering Suite (FAST Cache
and FAST VP) tiers both block and file data across heterogeneous drives and
promotes the most active data to the flash drives, ensuring that customers never
have to make concessions for cost or performance.
Data is used most frequently at the time it is created; therefore, new data is first
stored on flash drives for the best performance. As that data ages and becomes less
active over time, FAST VP moves the data from high-performance to high-capacity
drives automatically, based on customer-defined policies. EMC has enhanced this
functionality with four times better granularity and with new FAST VP solid-state disks
(SSDs) based on enterprise multi-level cell (eMLC) technology to lower the cost per
gigabyte. FAST Cache assists performance by dynamically absorbing unpredicted
spikes in system workloads. All VSPEX use cases benefit from the increased
efficiency.
20
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 2: Solution Overview
VSPEX Proven Infrastructures deliver private cloud, end-user computing, and
virtualized application solutions. With VNX, customers can realize an even greater
return on their investment. VNX also provides out-of-band, block-based deduplication
that can dramatically lower the costs of the flash tier.
VNX Intel MCx Code Path Optimization
The advent of flash technology has been a catalyst in totally changing the
requirements of midrange storage systems. EMC redesigned the midrange storage
platform to efficiently optimize multicore CPUs to provide the highest performing
storage system at the lowest cost in the market.
MCx distributes all VNX data services across all cores—up to 32, as shown in Figure 1.
The VNX series with MCx has dramatically improved the file performance for
transactional applications like databases or virtual machines over network-attached
storage (NAS).
Figure 1.
Next-Generation VNX with multicore optimization
Multicore Cache
The cache is the most valuable asset in the storage subsystem; its efficient use is key
to the overall efficiency of the platform in handling variable and changing workloads.
The cache engine has been modularized to take advantage of all the cores available
in the system.
Multicore RAID
Another important part of the MCx redesign is the handling of I/O to the permanent
back-end storage—hard disk drives (HDDs) and SSDs. Greatly increased performance
improvements in VNX come from the modularization of the back-end data
management processing, which enables MCx to seamlessly scale across all
processors.
VNX performance
Performance enhancements
VNX storage, enabled with the MCx architecture, is optimized for FLASH 1st and
provides unprecedented overall performance, optimizing for transaction performance
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
21
Chapter 2: Solution Overview
EMC Confidential
(cost per IOPS), bandwidth performance (cost per GB/s) with low latency, and
providing optimal capacity efficiency (cost per GB).
VNX provides the following performance improvements:

Up to four times more file transactions when compared with dual controller
arrays

Increased file performance for transactional applications by up to three times,
with a 60 percent better response time

Up to four times more Oracle and Microsoft SQL Server OLTP transactions

Up to six times more virtual machines
Active/active array storage processors
The new VNX architecture provides active/active array storage processors, as shown
in Figure 2, which eliminate application timeouts during path failover since both
paths are actively serving I/O.
Figure 2.
Active/active processors increase performance, resiliency, and efficiency
Load balancing is also improved and applications can achieve an up to two times
improvement in performance. Active/active for block is ideal for applications that
require the highest levels of availability and performance, but do not require tiering or
efficiency services like compression or deduplication.
With this VNX release, VSPEX customers can use virtual Data Movers (VDMs) and VNX
Replicator to perform automated and high-speed file system migrations between
systems. This process migrates all snaps and settings automatically, and enables the
clients to continue operation during the migration.
Note:: The active/active processors are only available for RAID logical unit numbers
(LUNs), not for pool LUNs.
Unisphere Management Suite
The new Unisphere Management Suite extends Unisphere’s easy-to-use, interface to
include VNX Monitoring and Reporting for validating performance and anticipating
22
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 2: Solution Overview
capacity requirements. As shown in Figure 3, the suite also includes Unisphere
Remote for centrally managing up to thousands of VNX and VNXe systems with new
support for XtremCache products.
Figure 3.
New Unisphere Management Suite
Virtualization Management
VMware Virtual Storage Integrator
EMC Virtual Storage Integrator (VSI) is a no-charge VMware vCenter plug-in available
to all VMware users with EMC storage. VSPEX customers can use VSI to simplify
management of virtualized storage. VMware administrators can gain visibility into
their VNX storage using the same familiar vCenter interface to which they are
accustomed.
With VSI, IT administrators can do more work in less time. VSI offers unmatched
access control that enables you to efficiently manage and delegate storage tasks with
confidence. Perform daily management tasks with up 90 percent fewer clicks and up
to 10 times higher productivity.
VMware vStorage APIs for Array Integration
VMware vStorage APIs for Array Integration (VAAI) offloads VMware storage-related
functions from the server to the storage system, enabling more efficient use of server
and network resources for increased performance and consolidation.
VMware vStorage APIs for Storage Awareness
VMware vStorage APIs for Storage Awareness (VASA) is a VMware-defined API that
displays storage information through vCenter. Integration between VASA technology
and VNX makes storage management in a virtualized environment a seamless
experience.
EMC Storage Integrator
EMC Storage Integrator (ESI) is targeted towards the Windows and application
administrator. ESI is easy to use, delivers end-to end monitoring, and is hypervisor
agnostic. Administrators can provision in both virtual and physical environments for a
Windows platform, and troubleshoot by viewing the topology of an application from
the underlying hypervisor to the storage.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
23
Chapter 2: Solution Overview
EMC Confidential
Offloaded Data Transfer
The Offloaded Data Transfer (ODX) feature of Microsoft Windows Server 2012 and
later versions enables data transfers during copy operations to be offloaded to the
storage array, freeing up host cycles. For example, using ODX for a live migration of a
SQL Server virtual machine doubled performance, decreased migration time by 50
percent, reduced CPU on the host sever by 20 percent, and eliminated network traffic.
Block deduplication
Native block deduplication was introduced in Windows Server 2012, and the R2
release contained minor improvements to the offering. It is important to understand
the impact of using OS-based deduplication on overall VSPEX performance and this
becomes critical if array-based deduplication is enabled. Lab testing has created the
following guidance:
24

If deduplication is enabled, either within the array or within the OS, FAST
Cache significantly reduces the overhead impact and minimizes impact on
latency; it is considered a best-practice to enable FAST Cache if deduplication
is enabled within a VSPEX environment.

VNX array based deduplication provided significantly better deduplication
results (approximately 2x improvement in space savings) and proved
beneficial to a wider range of workloads than OS-based deduplication.

Do not enable OS-based and VNX array-based deduplication on the same
LUNs.

Ensure that the allocation unit size matches the I/O size of the workload.
Failure to do so may result in non-optimal deduplication savings.

Windows deduplication will not start if the LUN contains less than 64 GB of
data.

Windows deduplication consumes both host and storage array resources and
requires monitoring to ensure other storage services on the array are not
adversely affected. The following three figures show SP resources
consumption values, IOPS, and latency when implementing Windows
deduplication.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 2: Solution Overview
Figure 4.
Storage processor utilization using Windows deduplication
Figure 5.
Disk IOPS using Windows deduplication
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
25
Chapter 2: Solution Overview
Figure 6.
EMC backup and
recovery
EMC Confidential
Disk latency using Windows deduplication
EMC backup and recovery solutions, EMC Avamar and EMC Data Domain, deliver the
protection confidence needed to accelerate the deployment of VSPEX private clouds.
Optimized for virtual environments, EMC backup and recovery reduces backup times
by 90 percent and increases recovery speeds by 30 times, even offering virtual
machine instant access for worry-free protection. EMC backup appliances add
another layer of assurance with end-to-end verification and self-healing to ensure
successful recoveries.
Our solutions also deliver big saving. With industry-leading deduplication, you can
reduce backup storage by 10 to 30 times, backup management time by 81 percent,
and WAN bandwidth by 99 percent for efficient disaster recovery, delivering a sevenmonth payback period on average. You will be able to scale storage easily and
efficiently as your environment grows.
For smaller VSPEX Private Cloud deployments, we recommend VDP Advanced for your
backup solution. Powered by Avamar technology, VDP Advanced offers the benefits of
Avamar's fast, efficient image-level backup and recovery for complete protection
confidence.
26
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 2: Solution Overview
Figure 7.
EMC backup and recovery solutions
EMC backup and recovery solutions used in this VSPEX solution include EMC Avamar
deduplication software and system, EMC Data Domain deduplication storage system,
and VMware vSphere Data Protection Advanced.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
27
Chapter 2: Solution Overview
28
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
EMC Confidential
Chapter 3: Solution Technology Overview
Chapter 3
Solution Technology Overview
This chapter presents the following topics:
Overview ..................................................................................................................30
Key components ......................................................................................................31
Virtualization ...........................................................................................................32
Compute ..................................................................................................................36
Network ...................................................................................................................39
Storage ....................................................................................................................41
Backup and recovery ................................................................................................ 49
Other technologies ..................................................................................................51
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
29
Chapter 3: Solution Technology Overview
EMC Confidential
Overview
This solution uses the EMC VNX series and VMware vSphere 5.5 to provide storage
and server hardware consolidation in a private cloud. The new virtualized
infrastructure is centrally managed, to provide efficient deployment and management
of a scalable number of virtual machines and associated shared storage.
Figure 8 depicts the solution components.
Figure 8.
Private cloud components
The following sections describe the components in more detail.
30
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
Key components
This section describes the key components of this solution.

Virtualization
The virtualization layer decouples the physical implementation of resources
from the applications that use them. In other words, the applications’ view of
the available resources is no longer directly tied to the hardware. This enables
many key features in the private cloud concept.

Compute
The compute layer provides memory and processing resources for the
virtualization layer software, and for the applications running in the private
cloud. The VSPEX program defines the minimum amount of required compute
layer resources, and enables the partner to implement the solution by using
any server hardware that meets these requirements

Network
The network layer connects the users of the private cloud to the resources in
the cloud, and the storage layer to the compute layer. The VSPEX program
defines the minimum number of required network ports, provides general
guidance on network architecture, and enables the customer to implement the
solution by using any network hardware that meets these requirements.

Storage
The storage layer is critical for the implementation of the private cloud. With
multiple hosts accessing shared data, many of the use cases defined in the
private cloud can be implemented. The EMC VNX storage family used in this
solution provides high-performance data storage while maintaining high
availability.

EMC backup and recovery
The backup and recovery components of the solution provide data protection
when the data in the primary system is deleted, damaged, or unusable.
The Solution architecture section provides details on all the components that make
up the reference architecture.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
31
Chapter 3: Solution Technology Overview
EMC Confidential
Virtualization
Overview
The virtualization layer is a key component of any server virtualization or private cloud
solution. It decouples the application resource requirements from the underlying
physical resources that serve them. This enables greater flexibility in the application
layer by eliminating hardware downtime for maintenance, and allows the system to
physically change without affecting the hosted applications. In a server virtualization
or private cloud use case, it enables multiple independent virtual machines to share
the same physical hardware, rather than being directly implemented on dedicated
hardware.
VMware vSphere
5.5
VMware vSphere 5.5 transforms the physical resources of a computer by virtualizing
the CPU, RAM, hard disk, and network controller. This transformation creates fully
functional virtual machines that run isolated and encapsulated operating systems
and applications like physical computers.
The high-availability features of VMware vSphere 5.5 such as vMotion and Storage
vMotion enable seamless migration of virtual machines and stored files from one
vSphere server to another, or from one data storage area to another, with minimal or
no performance impact. Coupled with vSphere DRS and Storage DRS, virtual
machines have access to the appropriate resources at any point in time through load
balancing of compute and storage resources.
New VMware
vSphere 5.5
features
32
VMware vSphere 5.5 includes an expansive list of new and improved features that
enhance performance, reliability, availability, and recovery of virtualized
environments. Of those features, several have significant impacts upon VSPEX Private
Cloud deployments, which include:

Expanded maximum memory and CPU limits for ESX hosts. Logical and virtual
CPU counts have doubled in this version, as have NUMA node counts and
maximum memory. This means host servers can support larger workloads.

62 TB VMDK file support including RDM. Datastores can hold more data from
more virtual machines, which simplifies storage management and leverages
larger capacity NL-SAS drives

Enhanced VAAI UNMAP support that includes a new esxcli storage vmfs
unmap command with multiple reclamation methods

Enhanced SR-IOV support that simplifies configuration via workflows, and
surfaces more properties into the virtual functions

16 Gb end-to-end support for FC environments

Enhanced LACP functions offering additional hash algorithms and up to 64
link access groups (LAGs)

vSphere Data Protection (VDP), which can now replicate backup data directly
to EMC Avamar

40 Gb Mellanox NIC support
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview

VMware vSphere
with Operations
Management
(vSOM)
VMFS heap improvements, which reduce memory requirements while
allowing access to full 64 TB VMFS address space
VMware vSphere with Operations Management delivers customers with a core
virtualization solution and is fully-supported within VSPEX.
This solution includes vSphere, the industry’s most trusted virtualization platform,
and the critical operational enhancements of performance monitoring and capacity
management (also offered through vCenter Operations Management Suite Standard).
As virtualization is deployed, it is critical to have visibility into an IT environment’s
operating performance to ensure service levels. For example, the ability to monitor
any degradation in health and performance allows customers to identify and
troubleshoot system issues before they affect end users.
vSphere with Operations Management enables advanced virtualization management
capabilities:

Capacity Management helps identify idle and over-provisioned virtual
machiness to reclaim excess capacity and increase virtual machine density
without impacting performance.

Predictive Analytics analyzes vCenter Server performance data, establishes
dynamic thresholds that adapt to the environment, and provides Smart Alerts
about health degradations and performance bottlenecks to drive proactive
action and policy-based automation.

Operations Console displays key performance indicators in easily identifiable
colored badges and provides a comprehensive view into what is driving
current and potential future performance and capacity management issues in
one place.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
33
Chapter 3: Solution Technology Overview
EMC Confidential
Figure 5. vSphere with Operations Management
vSphere with Operations Management enables the most demanding business critical
applications to operate on the most advanced virtualization platform. This
functionality delivers agility, efficiency, and resiliency for customer IT environments.
This strengthens the ability to run business critical applications at high service levels,
with improvements in three main categories:
1.
34
Availability and Performance – Deliver enhanced availability and performance
for business critical applications and next-gen applications, such as Hadoop
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
VMware vCenter
Chapter 3: Solution Technology Overview
2.
Storage – Leverage server-side caching for enhanced performance of
applications
3.
Scalability – Doubling configuration maximums in several key areas to
support the largest workloads possible
VMware vCenter is a centralized management platform for the VMware virtual
infrastructure. This platform provides administrators with a single interface for all
aspects of monitoring, managing, and maintaining the virtual infrastructure, accessed
from multiple devices.
VMware vCenter also manages some advanced features of the VMware virtual
infrastructure such as VMware vSphere High Availability and DRS, along with vMotion
and Update Manager.
VMware vSphere
High-Availability
The VMware vSphere High-Availability feature enables the virtualization layer to
automatically restart virtual machines in various failure conditions.

If the virtual machine operating system has an error, the virtual machine can
automatically restart on the same hardware.

If the physical hardware has an error, the impacted virtual machines can
automatically restart on other servers in the cluster.
Note: To restart virtual machines on different hardware, the servers must have available
resources. The Compute section provides detailed information to enable this function.
With VMware vSphere High-Availability, you can configure policies to determine
which machines automatically restart, and under what conditions to attempt these
operations.
EMC Virtual
Storage Integrator
for VMware
EMC Virtual Storage Integrator (VSI) for VMware vSphere is a plug-in for the vSphere
client that provides a single management interface for EMC storage within the
vSphere environment. Add and remove features to VSI independently; this provides
flexibility for customizing VSI user environments. Features are managed by using the
VSI Feature Manager. VSI provides a unified user experience, which enables new
features to be introduced rapidly in response to customer requirements.
Validation testing uses the following features:

Storage Viewer (SV) — Extend the vSphere client to help discover and identify
EMC VNX storage devices allocated to VMware vSphere hosts and virtual
machines. SV presents the underlying storage details to the virtual data center
administrator, merging the data of several different storage mapping tools into
a few seamless vSphere client views.

Unified Storage Management — Simplified storage administration of the EMC
VNX unified storage platform. It enables VMware administrators to provision
Virtual Machine File System (VMFS) datastores, Raw Device Mapping (RDM)
volumes, or network file system (NFS) seamlessly from within the vSphere
client.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
35
Chapter 3: Solution Technology Overview
EMC Confidential
Refer to the EMC VSI for VMware vSphere product guides on EMC Online Support for
more information.
VNX VMware
vStorage API for
Array Integration
support
Hardware acceleration with VMware vStorage API for Array Integration (VAAI) is a
storage enhancement in vSphere 5.5 that enables vSphere to offload specific storage
operations to compatible storage hardware such as the VNX series platforms. With
the assistance of storage hardware, vSphere performs these operations faster and
consumes less CPU, memory, and storage fabric bandwidth.
Compute
The choice of a server platform for an EMC VSPEX infrastructure is not only based on
the technical requirements of the environment, but on the supportability of the
platform, existing relationships with the server provider, advanced performance,
management features, and many other factors. For this reason, EMC VSPEX solutions
are designed to run on a wide variety of server platforms. Instead of requiring a
specific number of servers with a specific set of requirements, VSPEX documents
minimum requirements for the number of processor cores, and the amount of RAM.
This can be implemented with two or twenty servers, and still be considered the same
VSPEX solution.
In the example shown in Figure 9, the compute layer requirements for a specific
implementation are 25 processor cores, and 200 GB of RAM. One customer might
want to implement this with white-box servers containing 16 processor cores, and 64
GB of RAM, while another customer chooses a higher-end server with 20 processor
cores and 144 GB of RAM.
36
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
Figure 9.
Compute layer flexibility
The first customer needs four of the chosen servers, while the other customer needs
two.
Note: To enable high-availability at the compute layer, each customer needs one
additional server to ensure that the system has enough capability to maintain business
operations when a server fails.
Use the following best practices in the compute layer:

Use several identical, or at least compatible, servers. VSPEX implements
hypervisor level high-availability technologies, which may require similar
instruction sets on the underlying physical hardware. By implementing VSPEX
on identical server units, you can minimize compatibility problems in this area.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
37
Chapter 3: Solution Technology Overview
EMC Confidential

If you implement high-availability at the hypervisor layer, the largest virtual
machine you can create is constrained by the smallest physical server in the
environment.

Implement the available high-availability features in the virtualization layer,
and ensure that the compute layer has sufficient resources to accommodate at
least single server failures. This enables the implementation of minimaldowntime upgrades, and tolerance for single unit failures.
Within the boundaries of these recommendations and best practices, the compute
layer for EMC VSPEX can be flexible to meet your specific needs. Ensure that there are
sufficient processor cores and RAM per core to meet the needs of the target
environment.
38
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
Network
Overview
The infrastructure network requires redundant network links for each vSphere host,
the storage array, the switch interconnect ports, and the switch uplink ports. This
configuration provides both redundancy and additional network bandwidth. This is a
required configuration regardless of whether the network infrastructure for the
solution already exists, or you are deploying it alongside other components of the
solution. Figure 10 and Figure 11 depict an example of this highly available network
topology.
Figure 10. Example of highly available network design—for block
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
39
Chapter 3: Solution Technology Overview
EMC Confidential
Figure 11. Example of highly available network design—for file
This validated solution uses virtual local area networks (VLANs) to segregate network
traffic of various types to improve throughput, manageability, application separation,
high-availability, and security.
For block, EMC unified storage platforms provide network high-availability or
redundancy by two ports per storage processor. If a link is lost on the storage
processor front end port, the link fails over to another port. All network traffic is
distributed across the active links.
For file, EMC unified storage platforms provide network high-availability or
redundancy by using link aggregation. Link aggregation enables multiple active (MAC)
Ethernet connections to appear as a single link with a single MAC address, and
potentially multiple IP addresses. In this solution, Link Aggregation Control Protocol
(LACP) is configured on the VNX, combining multiple Ethernet ports into a single
virtual device. If a link is lost on the Ethernet port, the link fails over to another port.
All network traffic is distributed across the active links.
40
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
Storage
Overview
The storage layer is also a key component of any cloud infrastructure solution that
serves data generated by applications and operating system in the data center
storage processing systems. This increases storage efficiency, management
flexibility, and reduces total cost of ownership. In this VSPEX solution, EMC VNX
series arrays provide features and performance to enable and enhance any
virtualization environment.
EMC VNX series
The EMC VNX family is optimized for virtual applications; and delivers industryleading innovation and enterprise capabilities for file and block storage in a scalable,
easy-to-use solution. This next-generation storage platform combines powerful and
flexible hardware with advanced efficiency, management, and protection software to
meet the demanding needs of today’s enterprises.
Intel Xeon processors power the VNX series for intelligent storage that automatically
and efficiently scales in performance, while ensuring data integrity and security. It is
designed to meet the high performance, high-scalability requirements of midsize and
large enterprises.
Table 1 shows the customer benefits that are provided by VNX series.
Table 1.
VNX customer benefits
Feature
Benefit
Next-generation unified storage, optimized
for virtualized applications
Tight integration with VMware allows for
advanced array features and centralized
management
Capacity optimization features including
compression, deduplication, thin
provisioning, and application-consistent
copies
Reduced storage costs, more efficient
use of resources and easier recovery of
applications
High-availability, designed to deliver five 9s
availability
Higher levels of uptime and reduced
outage risk
Automated tiering with FAST VP and FAST
Cache that can be optimized for the highest
system performance and lowest storage cost
simultaneously
More efficient use of storage resources
without complicated planning and
configuration
Simplified management with EMC Unisphere
for a single management interface for all
NAS, SAN and replication needs
Reduced management overhead and
toolsets required to manage environment
Up to three times improvement in
performance with the latest Intel Xeon
multicore processor technology, optimized
for flash
Reduced latency, increased bandwidth
and IOPS result in more headroom for
demanding workloads
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
41
Chapter 3: Solution Technology Overview
EMC Confidential
Different software suites and packs are also available for the VNX series, which
provide multiple features for enhanced protection and performance:
Software suites

FAST Suite — Automatically optimizes for the highest system performance and
the lowest storage cost simultaneously.

Local Protection Suite — Practices safe data protection and repurposing.

Remote Protection Suite — Protects data against localized failures, outages,
and disasters.

Application Protection Suite — Automates application copies and provides
compliance.

Security and Compliance Suite — Keeps data safe from changes, deletions, and
malicious activity.
Software packs
VNX Snapshots

Total Efficiency Pack — Includes all five software suites.

Total Protection Pack — Includes local, remote, and application protection
suites.
VNX Snapshots is a software feature introduced since VNX OE for Block Release 32,
which creates point-in-time data copies. VNX Snapshots can be used for data
backups, software development and testing, repurposing, data validation and local
rapid restores. VNX Snapshots improves on the existing SnapView Snapshot
functionality by integrating with storage pools.
Note: LUNs created on physical RAID groups, also called RAID LUNs, support only
SnapView Snapshots. This limitation exists because VNX Snapshots require pool space as
part of its technology.
VNX Snapshots support 256 writeable snaps per pool LUN. It supports branching,
also called a Snap of a Snap, as long as the total number of snapshots for any
primary LUN is less than 256, which is a hard limit.
VNX Snapshots use redirect on write (ROW) technology. ROW redirects new writes
destined for the primary LUN to a new location in the storage pool. Such an
implementation is different from copy on first write (COFW) used in SnapView, which
holds the writes to the primary LUN until the original data is copied to the reserved
LUN pool to preserve a snapshot.
This release (Block OE Release 33) also supports consistency groups (CGs). Several
pool LUNs can be combined into a CG and snapped concurrently. When a snapshot of
a CG is initiated, all writes to the member LUNs are held until snapshots have been
created. Typically, CGs are used for LUNs that belong to the same application.
VNX SnapSure
42
VNX SnapSure is an EMC VNX File software feature that enables you to create and
manage checkpoints, which are point-in-time, logical images of a production file
system (PFS). SnapSure uses a copy on first modify principle. A PFS consists of
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
blocks. When a block within the PFS is modified, a copy containing the block’s
original contents is saved to a separate volume called the SavVol. Subsequent
changes made to the same block in the PFS are not copied into the SavVol. The
original blocks from the PFS in the SavVol and the unchanged PFS blocks remaining
in the PFS are read by SnapSure according to a bitmap and block map data-tracking
structure. These blocks combine to provide a complete point-in-time image called a
checkpoint.
A checkpoint reflects the state of a PFS at the time the checkpoint is created.
SnapSure supports two types of checkpoints:

Read-only checkpoint — Read-only file system created from a PFS.

Writeable checkpoint — Read/write file system created from a read-only
checkpoint.
SnapSure can maintain a maximum of 96 read-only checkpoints and 16 writeable
checkpoints per PFS while allowing PFS applications continued access to real time
data.
Note: Each writeable checkpoint associates with a read-only checkpoint, referred to as the
baseline checkpoint. Each baseline checkpoint can have only one associated writeable
checkpoint.
For more detailed information, refer to Using VNX SnapSure.
VNX Virtual
Provisioning
EMC VNX Virtual Provisioning enables organizations to reduce storage costs by
increasing capacity utilization, simplifying storage management, and reducing
application downtime. Virtual Provisioning also helps companies to reduce power
and cooling requirements and reduce capital expenditures.
Virtual Provisioning provides pool-based storage provisioning by implementing pool
LUNs that can be either thin or thick. Thin LUNs provide on-demand storage that
maximizes the utilization of your storage by allocating storage as needed. Thick LUNs
provide both high performance and predictable performance for your applications.
Both types of LUNs benefit from the ease-of-use features of pool-based provisioning.
Pools and pool LUNs are also the building blocks for advanced data services such as
FAST VP, VNX Snapshots, and compression. Pool LUNs also support a variety of
additional features, such as LUN shrink, online expansion, and User Capacity
Threshold setting.
EMC VNX Virtual Provisioning allows you to expand the capacity of a storage pool
from the Unisphere GUI after disks are physically attached to the system. VNX
systems have the ability to rebalance allocated data elements across all member
drives to use new drives after the pool is expanded. The rebalance function starts
automatically and runs in the background after an expand action. Monitor the
progress of a rebalance operation from the General tab of the Pool Properties window
in Unisphere, as shown in Figure 12.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
43
Chapter 3: Solution Technology Overview
EMC Confidential
Figure 12. Storage pool rebalance progress
LUN expansion
Use pool LUN expansion to increase the capacity of existing LUNs. It allows for
provisioning larger capacity as business needs grow.
The VNX family has the capability to expand a pool LUN without disrupting user
access. Pool LUN expansion can be done with a few simple clicks and the expanded
capacity is immediately available. However, you cannot expand a pool LUN if it is part
of a data protection or LUN-migration operation. For example, snapshot LUNs or
migrating LUNs cannot be expanded.
For more detailed information of pool LUN expansion, refer to EMC VNX Virtual
Provisioning — Applied Technology White Paper.
LUN shrink
Use LUN shrink of thin LUN to reduce the capacity of existing LUNs.
VNX can shrink a pool LUN. This capability is only available for LUNs served by
Windows Server 2008 and later. The shrinking process has two steps:
4.
Shrink the file system from Windows Disk Management.
5.
Shrink the pool LUN using a command window and the DISKRAID utility. The
utility is available through the VDS Provider, which is part of the EMC
Solutions Enabler package.
The new LUN size appears as soon as the shrink process is complete. A background
task reclaims the deleted or shrunk space and returns it to the storage pool. Once the
task is completed, any other LUN in that pool can use the reclaimed space.
For more detailed information of thin LUN expansion, refer to EMC VNX Virtual
Provisioning Applied Technology White Paper.
44
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
User alerting through capacity threshold setting
Customers must configure proactive alerts when using a file system or storage pools
based on thin pools. Monitor these resources so that storage is available to be
provisioned when needed and capacity shortages can be avoided.
Figure 13 explains why provisioning with thin pools requires monitoring.
Figure 13. Thin LUN space utilization
Monitor the following values for thin pool utilization:

Total capacity is the total physical capacity available to all LUNs in the pool.

Total allocation is the total physical capacity currently assigned to all pool
LUNs.

Subscribed capacity is the total host reported capacity supported by the pool.

Over-subscribed capacity is the amount of user capacity configured for LUNs
that exceeds the physical capacity in a pool.

Total allocation may never exceed the total capacity, but if it nears that point,
add storage to the pools proactively before reaching a hard limit.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
45
Chapter 3: Solution Technology Overview
EMC Confidential
Figure 14 shows the Storage Pool Properties dialog box in Unisphere, which displays
parameters such as Free Capacity, Percent Full, Total allocation, Total Subscription,
Percent Subscribed and Oversubscribed By Capacity.
Figure 14. Examining storage pool space utilization
When storage pool capacity becomes exhausted, any requests for additional space
allocation on thin provisioned LUNs fail. Applications attempting to write data to
these LUNs usually fail as well, and an outage is the likely result. To avoid this
situation, monitor pool utilization, and alert when thresholds are reached, set the
Percentage Full Threshold to allow enough buffer to make remediation before an
outage situation occurs. Adjust this setting by clicking on the Advanced tab of the
Storage Pool Properties dialog, as seen in Figure 15. This alert is only active if there
are one or more thin LUNs in the pool, because thin LUNs are the only way to
oversubscribe a pool. If the pool only contains thick LUNs, the alert is not active as
there is no risk of running out of space due to oversubscription. You also can specify
the value for Percent Full Threshold, which equals Total Allocation/Total Capacity,
when a pool is created.
46
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
Figure 15. Defining storage pool utilization thresholds
View alerts by using the Alert tab in Unisphere. Figure 16 shows the Unisphere Event
Monitor Wizard, where you can also select the option of receiving alerts through
email, a paging service, or an SNMP trap.
Figure 16. Defining automated notifications (for block)
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
47
Chapter 3: Solution Technology Overview
EMC Confidential
Table 2 displays information about thresholds and their settings under VNX OE Block
33.
Table 2.
Thresholds and settings under VNX OE Block Release 33
Threshold type
Threshold
range
Threshold
default
Alert severity
Side effect
User settable
1% – 84%
70%
Warning
None
Built-in
N/A
85%
Critical
Clears user
settable alert
Allowing total allocation to exceed 90 percent of total capacity puts you at risk of
running out of space and impacting all applications that use thin LUNs in the pool.
VNX FAST Cache
VNX FAST Cache, a part of the VNX FAST Suite, enables flash drives to function as an
expanded cache layer for the array. FAST Cache is an array-wide, non-disruptive
cache, available for both file and block storage. Frequently accessed data is copied to
the FAST Cache in 64 KB increments and subsequent reads and/or writes to the data
chunk are serviced by FAST Cache. This enables immediate promotion of highly active
data to flash drives. This dramatically improves the response time for the active data
and reduces data hot spots that can occur within a LUN. The FAST Cache feature is an
optional component of this solution.
VNX FAST VP
VNX FAST VP, a part of the VNX FAST Suite, can automatically tier data across multiple
types of drives to leverage differences in performance and capacity. FAST VP is
applied at the block storage pool level and automatically adjusts where data is stored
based on how frequently it is accessed. Frequently accessed data is promoted to
higher tiers of storage in 256 MB increments, while infrequently accessed data can be
migrated to a lower tier for cost efficiency. This rebalancing of 256 MB data units, or
slices, is part of a regularly scheduled maintenance operation.
vCloud Networking VMware vShield Edge, App, and Data Security capabilities have been integrated and
enhanced in vCloud Networking and Security, which is part of the VMware vCloud
and Security
Suite. VSPEX Private Cloud solutions with VMware vCloud Networking enables
customers to adopt virtualized networks that eliminate the rigidity and complexity
associated with physical equipment that creates artificial barriers to operating an
optimized network architecture. Physical networking has not kept pace with the
virtualization of the data center and it limits the ability of businesses to rapidly
deploy, move, scale, and protect applications and data according to business needs.
VSPEX with VMware vCloud Networking and Security solves these data center
challenges by virtualizing networks and security to create efficient, agile, extensible
logical constructs that meet the performance and scale requirements of virtualized
data centers. vCloud Networking and Security delivers software-defined networks and
security with a broad range of services in a single solution and includes a virtual
firewall, virtual private network (VPN), load balancing, and VXLAN-extended networks.
Management integration with VMware vCenter Server and VMware vCloud Director
reduces the cost and complexity of data center operations and unlocks the
operational efficiency and agility of private cloud computing.
48
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
VSPEX for Virtualized Applications can also take advantage of vCloud Networking and
Security features. VSPEX allows a business to virtualize Microsoft applications. With
VMware vCloud, these applications can have protection and isolation from risk as
administrators have greater visibility into virtual traffic flows so that they can enforce
policies and implement compliance controls on in-scope systems by implementing
logical grouping and virtual firewalls.
Administrators deploying virtual desktops with VSPEX End User Computing with
VMware vSphere 5.5 and View can also benefit from vCloud Networking and Security
by creating logical security around individual or groups of virtual desktops. This will
ensure that those machine’s users deployed on the VSPEX Proven Infrastructure can
only access the applications and data with authorization, preventing broader access
to the data center. vCloud also enables rapid diagnosis of traffic and potential trouble
spots. Administrators can effectively create software defined networks that scale and
move virtual workloads within their VSPEX Proven Infrastructures without physical
networking or security constraints, all of which can be streamlined via VMware
vCenter and VMware vCloud Director Integration.
VNX file shares
In many environments, it is important to have a common location to store files
accessed by many different individuals. This is implemented as CIFS or NFS file
shares from a file server. The VNX family of storage arrays can provide this service
along with centralized management, client integration, advanced security options,
and efficiency improvement features.
ROBO
Organizations with remote office and branch offices (ROBO) often prefer to locate
data and applications close to the users in order to provide better performance and
lower latency. In these environments, IT departments need to balance the benefits of
local support with the need to maintain central control. Local Systems and storage
should be easy for local personnel to administer, but also support remote
management and flexible aggregation tools that minimize the demands on those
local resources. With VSPEX, you can accelerate the deployment of applications at
remote offices and branch offices. Customers can also leverage Unisphere Remote to
consolidate the monitoring, system alerts, and reporting of hundreds of locations
while maintaining simplicity of operation and unified storage functionality for local
managers.
Backup and recovery
Overview
Backup and recovery, another important component in this VSPEX solution, provides
data protection by backing up data files or volumes on a defined schedule, and then
restores data from backup for recovery after a disaster.
EMC backup and recovery is a smart method of data protection. It consists of best of
class, integrated protection storage and software designed to meet backup and
recovery objectives now and in the future. With EMC market-leading protection
storage, deep data source integration, and feature-rich data management services,
you can deploy an open, modular protection storage architecture that allows you to
scale while lowering cost and complexity.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
49
Chapter 3: Solution Technology Overview
EMC Confidential
EMC Avamar
deduplication
EMC Avamar provides fast, efficient backup and recovery through a complete
software and hardware solution. Equipped with integrated variable-length
deduplication technology, Avamar facilitates fast, daily full backups for virtual
environments, remote offices, enterprise applications, network-attached storage
(NAS) servers, and desktops/laptops. Learn more: http://www.emc.com/avamar
EMC Data Domain
deduplication
storage systems
EMC Data Domain Deduplication storage systems continue to revolutionize disk
backup, archiving, and disaster recovery with high-speed, inline deduplication for
backup and archive workloads. Learn more: http://www.emc.com/datadomain
VMware vSphere
data protection
vSphere Data Protection (VDP) is a proven solution for backing up and restoring
VMware virtual machines. VDP is based on EMC’s award-winning Avamar product and
has many integration points with vSphere 5.5, providing simple discovery of your
virtual machines and efficient policy creation. One of challenges that traditional
systems have with virtual machines is the large amount of data that these files
contain. VDP’s usage of a variable-length deduplication algorithm ensures a
minimum amount of disk space is used and reduces ongoing backup storage growth.
Data is deduplicated across all virtual machines associated with the VDP virtual
appliance.
VDP uses vStorage APIs for Data Protection (VADP), which sends only the changed
blocks of data, resulting in only a fraction of the data being sent over the network.
VDP enables up to eight virtual machines to be backed up concurrently. Because VDP
resides in a dedicated virtual appliance, all the backup processes are offloaded from
the production virtual machines.
VDP can alleviate the burdens of restore requests from administrators by enabling
end users to restore their own files using a web-based tool called vSphere Data
Protection Restore Client. Users can browse their system’s backups in an easy to use
interface that provides search and version control features. The users can restore
individual files or directories without any intervention from IT, freeing up valuable
time and resources, resulting in a better end user experience.
For backup and recovery options, refer to EMC Backup and Recovery Options for
VSPEX Private Clouds Design and Implementation Guide.
vSphere
Replication
vSphere Replication is a feature of the vSphere 5.5 platform that provides business
continuity. vSphere Replication copies a virtual machine defined in your VSPEX
Infrastructures to a second instance of VSPEX or within the clustered servers in a
single VSPEX system. vSphere replication continues to protect the virtual machine on
an ongoing basis and replicates the changes to the copied virtual machine. This
replication ensures that the virtual machine remains protected and is available for
recovery without requiring restoration from backup. Replication application virtual
machines are defined in VSPEX to ensure application-consistent data with a single
click when replication is set up.
Administrators who manage VSPEX for virtualized Microsoft applications can use the
automatic integration of vSphere Replication with Microsoft’s Volume Shadow Copy
Service (VSS) to ensure that applications such as Microsoft Exchange or Microsoft
SQL Server databases are quiescent and consistent when generating replica data. A
very quick call to the virtual machine’s VSS layer flushes the database writers for an
50
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
instant to ensure that the replicated data is static and fully recoverable. This
automated approach simplifies the management and increases the efficiency of your
VSPEX based virtual environment.
EMC RecoverPoint
EMC RecoverPoint is an enterprise-scale solution that protects application data on
heterogeneous SAN-attached servers and storage arrays. RecoverPoint runs on a
dedicated appliance (RPA) and combines industry-leading continuous data protection
technology with a bandwidth-efficient, no-data-loss replication technology. This
technology enables RPA to protect data locally (continuous data protection, CDP),
remotely (continuous remote replication, CRR), or both (CLR), offering the following
advantages:

RecoverPoint CDP replicates data within the same site or to a local bunker site
some distance away, and transfers the data via Fibre Channel (FC).

RecoverPoint CRR uses either FC or an existing IP network to send the data
snapshots to the remote site using techniques that preserve write-order.

In a CLR configuration, RecoverPoint replicates to both a local and a remote
site simultaneously.
RecoverPoint uses lightweight splitting technology on the application server, in the
fabric or in the array, to mirror application writes to the RecoverPoint cluster, and
supports the following write splitter types:

Array-based

Intelligent fabric-based

Host-based
Other technologies
Overview
In addition to the required technical components for EMC VSPEX solutions, other
items may provide additional value depending on the specific use case. These
include, but are not limited to the following technologies.
VMware vCloud
Automation Center
VMware vCloud Automation Center, which is part of the vCloud Suite Enterprise,
orchestrates the provisioning of software-defined data center services as complete
virtual data centers that are ready for consumption. vCloud Automation Center is a
software solution that enables customers to build secure, private clouds by pooling
infrastructure resources from VSPEX into virtual data centers and exposing them to
users through Web-based portals and programmatic interfaces as fully automated,
catalog-based services.
VMware vCloud Automation Center uses pools of resources abstracted from the
underlying physical, virtual, and cloud-based resources to automate the deployment
of virtual resources when and where required. VSPEX with vCloud Automation Center
enables customers to build complete virtual data centers delivering computing,
networking, storage, security, and a complete set of services necessary to make
workloads with minimal overhead.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
51
Chapter 3: Solution Technology Overview
EMC Confidential
Software-defined data center service and the virtual data centers fundamentally
simplify infrastructure provisioning, and enable IT to move at the speed of business.
VMware vCloud Automation Center integrates with existing or new VSPEX Private
Cloud with VMware vSphere 5.5 deployments and supports existing and future
applications by providing elastic standard storage and networking interfaces, such as
Layer-2 connectivity and broadcasting between virtual machines. VMware vCloud
Automation Center uses open standards to preserve deployment flexibility and pave
the way to the hybrid cloud. The key features of VMware vCloud Automation Center
include:





Self-service provisioning
Life-cycle management
Unified cloud management
Multi-VM blueprints
Context aware, policy based governance

Intelligent resource management
All VSPEX Proven Infrastructures can use vCloud Automation Center to orchestrate
deployment of virtual data centers based on single VSPEX or multi-VSPEX
deployments. These infrastructures enable simple and efficient deployment of virtual
machines, applications, and virtual networks.
The VMware vCenter Operations Manager Suite (vCOPS) provides unparalleled
VMware vCenter
visibility into VSPEX virtual environments. vCOPS collects, analyzes data, correlates
Operations
Management Suite abnormalities, identifies the root cause of performance problems, and provides
administrators with the information needed to optimize and tune their VSPEX virtual
infrastructures. vCenter Operations Manager provides an automated approach to
optimizing your VSPEX-powered virtual environment by delivering self-learning
analytic tools that are integrated to provide better performance, capacity usage and
configuration management. vCOPS delivers a comprehensive set of management
capabilities, including:





Performance
Capacity
Adaptability
Configuration and compliance management
Application discovery and monitoring

Cost metering
vCOPS includes five components: VMware vCenter Operations Manager, VMware
vCenter Configuration Manager, VMware vFabric Hyperic, VMware vCenter
Infrastructure Navigator, and VMware vCenter Chargeback Manager.
VMware vCenter Operations Manager is the foundation of the suite and provides the
operational dashboard interface that makes visualizing issues in your VSPEX virtual
environment simple. vFabric Hyperic monitors physical hardware resources, operating
systems, middleware, and applications that you may have deployed on VSPEX.
52
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
vCenter Infrastructure Navigator provides visibility into the application services
running over the virtual-machine infrastructure and their interrelationships for day-today operational management.
vCenter Chargeback Manager enables accurate cost measurement, analysis, and
reporting of virtual machines. It provides visibility into the cost of the virtual
infrastructure that you have defined on VSPEX as being required to support business
services.
VMware vCenter
Single Sign On
With the introduction of VMware vCenter Single Sign-On (SSO) in VMware vSphere
5.5, administrators now have a deeper level of available authentication services for
managing their VSPEX Proven Infrastructures. Authentication by vCenter SSO makes
the VMware cloud infrastructure platform more secure. This function allows the
vSphere software components to communicate with each other through a secure
token exchange mechanism, instead of requiring each component to authenticate a
user separately with a directory service such as Active Directory.
When users log in to the vSphere web client with user names and passwords, the
vCenter SSO server receives their credentials. The credentials are then authenticated
against the back-end identity source(s) and exchanged for a security token, which is
returned to the client to access the solutions within the environment. SSO translates
into time and cost savings which, when factored in against the entire organization,
can result in savings and streamlined workflows.
With vSphere 5.5, users have a unified view of their entire vCenter server environment
because multiple vCenter servers and their inventories are now displayed. This does
not require Linked Mode unless users share roles, permissions, and licenses among
vSphere 5.x vCenter servers.
Administrators can now deploy multiple solutions within an environment with true
single sign-on (SSO) that creates trust between solutions without requiring
authentication every time a user accesses the solution.
VSPEX Private Cloud with VMware vSphere 5.5 is simple, efficient, and flexible.
VMware SSO makes authentication simpler, workers can be more efficient, and
administrators have the flexibility to make SSO servers local or global.
Public-key
infrastructure
The ability to secure data and ensure the identity of devices and users is critical in
today’s enterprise IT environment. This is particularly true in regulated sectors such
as healthcare, financial, and government. VSPEX solutions can offer hardened
computing platforms in many ways, most commonly by implementing a public-key
infrastructure (PKI).
The VSPEX solutions can be engineered with a PKI solution designed to meet the
security criteria of your organization, and can be done via a modular process, where
layers of security are added as needed. The general process involves first
implementing a PKI infrastructure by replacing generic self-certified certificates with
trusted certificates from a third-party certificate authority. Services that support PKI
can then be enabled using the trusted certificates ensuring a high degree of
authentication and encryption where supported.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
53
Chapter 3: Solution Technology Overview
EMC Confidential
Depending on the scope of PKI services needed, it may become necessary to
implement a PKI infrastructure dedicated to those needs. There are many third party
tools that offer these services including end-to-end solutions from RSA that can be
deployed within a VSPEX environment. For additional information, visit the RSA
website.
EMC Storage
Analytics for EMC
VNX
The software combines the features and functionality of VMware vCenter Operations
Manager with deep VNX storage analytics. It delivers custom analytics and
visualizations that provide deep visibility into your EMC infrastructure and enable you
to troubleshoot, identify, and take action on storage performance and capacity
management problems quickly.
“Out-of-the box” custom visualizations enable customers to quickly deploy EMC
infrastructure support within vCenter Operations Manager without the need for
customer integration or Professional Services. This software also delivers actionable
performance analysis to enable customers to quickly identify and resolve
performance and capacity issues for VNX series systems.
PowerPath/VE (for
block)

EMC Storage Analytics for VNX is supported on all VNX systems:Rich storage
analytics: View performance and capacity statistics, including statistics for
FAST Cache and FAST VP.

Topology views: End-to-end topology mapping from virtual machines to the
disk drives helps simplify storage operations management.

SLA maintenance: Quick troubleshooting of performance abnormalities and
remediation assistance helps you maintain service levels.
EMC PowerPath/VE for VMware vSphere 5.5 is a module that provides multi-pathing
extensions for vSphere and works in combination with SAN storage to intelligently
manage FC, iSCSI, and Fiber Channel over Ethernet (FCoE) I/O paths.
PowerPath/VE is installed on the vSphere host and scales to the maximum number of
virtual machines on the host, improving I/O performance. The virtual machines do not
have PowerPath/VE installed nor are they aware that PowerPath/VE is managing I/O
to storage. PowerPath/VE dynamically balances I/O load requests and automatically
detects and recovers from path failures.
EMC XtremCache
EMC XtremCache is a server flash caching solution that reduces latency and increases
throughput to improve application performance by using intelligent caching software
and PCIe flash technology.
Server-side flash caching for maximum speed
XtremCache performs the following functions to improve system performance:
54

Caches the most frequently referenced data on the server-based PCIe card to
put the data closer to the application.

Automatically adapts to changing workloads by determining the most
frequently referenced data and promoting it to the server flash card. This means
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 3: Solution Technology Overview
that the most active data automatically resides on the PCIe card in the server
for faster access.

Offloads the read traffic from the storage array, which allocates greater
processing power to other applications. While one application accelerates with
XtremCache, the array performance for other applications remains the same or
slightly enhanced.
Write-through caching to the array for total protection
XtremCache accelerates reads and protects data by using a write-through cache to
the storage to deliver persistent high-availability, integrity, and disaster recovery.
Application agnostic
XtremCache is transparent to applications; there is no need to rewrite, retest or
recertify to deploy XtremCache in the environment.
Integration with vSphere 5.5
XtremCache enhances both virtualized and physical environments. Integration with
the VSI plug-in to VMware vSphere 5.5 vCenter simplifies the management and
monitoring of XtremCache.
Minimal impact on system resources
Unlike other caching solutions on the market, XtremCache does not require a
significant amount of memory or CPU cycles, as all flash and wear-leveling
management is done on the PCIe card without using server resources. Unlike other
PCIe solutions, there is no significant overhead from using XtremCache on server
resources.
XtremCache creates the most efficient and intelligent I/O path from the application to
the datastore, which results in an infrastructure that is dynamically optimized for
performance, intelligence, and protection for both physical and virtual environments.
XtremCache active/passive clustering support
The configuration of XtremCache clustering scripts ensures that stale data is never
retrieved. The scripts use cluster management events to trigger a mechanism that
purges the cache. The XtremCache-enabled active/passive cluster ensures data
integrity and accelerates application performance.
XtremCache performance considerations
XtremCache performance considerations are:

On a write request, XtremCache first writes to the array, then to the cache, and
then completes the application I/O.

On a read request, XtremCache satisfies the request with cached data, or, when
the data is not present, retrieves the data from the array, writes it to the cache,
and then returns it to the application. The trip to the array can be in the order of
milliseconds; therefore, the array limits how fast the cache can work. As the
number of writes increases, XtremCache performance decreases.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
55
Chapter 3: Solution Technology Overview
56
EMC Confidential

XtremCache is most effective for workloads with a 70 percent or greater
read/write ratio, with small, random I/O (8K is ideal). I/O greater than 128K is
not cached in XtremCache 1.5.
Note:
For more information, refer to the white paper titled Introduction to XtremCache.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Chapter 4
Solution Architecture Overview
This chapter presents the following topics:
Overview ..................................................................................................................58
Solution architecture ............................................................................................... 58
Server configuration guidelines ...............................................................................67
Network configuration guidelines ............................................................................71
Storage configuration guidelines .............................................................................75
High-availability and failover ...................................................................................88
Validation test profile .............................................................................................. 91
Backup and recovery configuration guidelines .........................................................91
Sizing guidelines .....................................................................................................91
Reference workload..................................................................................................92
Applying the reference workload .............................................................................93
Implementing the solution .......................................................................................95
Quick assessment ....................................................................................................98
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
57
Chapter 4: Solution Architecture Overview
EMC Confidential
Overview
This chapter provides a comprehensive guide to the major aspects of this solution.
Server capacity is presented in generic terms for required minimums of CPU, memory,
and network resources; customer can select the server and networking hardware that
meets or exceeds the stated minimums. The specified storage architecture, along
with a system meeting the server and network requirements outlined, has been
validated by EMC to provide high levels of performance while delivering a highly
available architecture for your private cloud deployment.
Each VSPEX Proven Infrastructure balances the storage, network, and compute
resources needed for a set number of virtual machines validated by EMC. Each virtual
machine has its own set of requirements that rarely fit a pre-defined idea of a virtual
machine. In any discussion about virtual infrastructures, it is important to first define
a reference workload. Not all servers perform the same tasks, and it is impractical to
build a reference that takes into account every possible combination of workload
characteristics.
Solution architecture
Overview
The VSPEX private cloud solution for VMware vSphere with EMC VNX validates at four
different points of scale, one configuration with up to 200 virtual machines, one
configuration with up to 300 virtual machines, one configuration with up to 600
virtual machines, and one configuration with up to 1,000 virtual machines. The
defined configurations form the basis of creating a custom solution.
Note: : VSPEX uses the concept of a reference workload to describe and define a virtual
machine. Therefore, one physical or virtual server in an existing environment may not be
equal to one virtual machine in a VSPEX solution. Evaluate your workload in terms of the
reference to arrive at an appropriate point of scale. This document describes the process in
Applying the reference workload
58
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Logical
architecture
Chapter 4: Solution Architecture Overview
The architecture diagrams in this section show the layout of major components in the
solutions. Storage for block and file based systems are shown in the following
diagrams.
Figure 17 characterizes the infrastructure validated with block-based storage, where
an 8 Gb FC/FCoE or 10 Gb-iSCSI SAN carries storage traffic, and 10 GbE carries
management and application traffic.
Figure 17. Logical architecture for block storage
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
59
Chapter 4: Solution Architecture Overview
EMC Confidential
Figure 18 characterizes the infrastructure validated with file-based storage, where 10
GbE carries storage traffic and all other traffic.
Figure 18. Logical architecture for file storage
Key components
This architecture includes the following key components:
VMware vSphere 5.5—Provides a common virtualization layer to host a server
environment. The specifics of the validated environment are listed in Table 3.
vSphere 5.5 provides highly available infrastructure through such features as:

vMotion—Provides live migration of virtual machines within a virtual
infrastructure cluster, with no virtual machine downtime or service disruption.

Storage vMotion—Provides live migration of virtual machine disk files within
and across storage arrays with no virtual machine downtime or service
disruption.

vSphere High-Availability (HA)—Detects and provides rapid recovery for a failed
virtual machine in a cluster.

Distributed Resource Scheduler (DRS)—Provides load balancing of computing
capacity in a cluster.

Storage Distributed Resource Scheduler (SDRS)—Provides load balancing
across multiple datastores based on space usage and I/O latency.
VMware vCenter Server—Provides a scalable and extensible platform that forms the
foundation for virtualization management for the VMware vSphere cluster. vCenter
manages all vSphere hosts and their virtual machines.
SQL Server—VMware vCenter Server requires a database service to store
configuration and monitoring details. This solution uses a Microsoft SQL 2008 R2
server.
60
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
DNS Server—Use DNS services for the various solution components to perform name
resolution. This solution uses the Microsoft DNS Service running on Windows 2012
R2 server.
Active Directory Server—Various solution components require Active Directory
services to function properly. The Microsoft AD Service runs on a Windows Server
2012 R2 server.
Shared Infrastructure—Add DNS and authentication/authorization services, such as
AD Service, with existing infrastructure or set up as part of the new virtual
infrastructure.
IP Network—A standard Ethernet network carries all network traffic with redundant
cabling and switching. A shared IP network carries user and management traffic.
Storage network
The storage network is an isolated network that provides hosts with access to the
storage array. VSPEX offers different options for block-based and file-based storage.
Storage network for block:
This solution provides three options for block-based storage networks.

Fibre Channel (FC)—A set of standards that define protocols for performing high
speed serial data transfer. FC provides a standard data transport frame among
servers and shared storage devices.

Fibre Channel over Ethernet (FCoE)—A newer storage networking protocol that
supports FC natively over Ethernet, by encapsulating FC frames into Ethernet
frames. This allows the encapsulated FC frames to run alongside traditional
Internet Protocol (IP) traffic.

10 Gb Ethernet (iSCSI)—Enables the transport of SCSI blocks over a TCP/IP
network. iSCSI works by encapsulating SCSI commands into TCP packets and
sending the packets over the IP network.
Storage network for file:
With file-based storage, a private, non-routable 10 GbE subnet carries the storage
traffic.
VNX storage array
The VSPEX private cloud configuration begins with the VNX family storage arrays,
including:

EMC VNX5200 array—Provides storage to vSphere hosts for up to 200 virtual
machines.

EMC VNX5400 array—Provides storage to vSphere hosts for up to 300 virtual
machines.

EMC VNX5600 array—Provides storage to vSphere hosts for up to 600 virtual
machines.

EMC VNX5800 array—Provides storage to vSphere hosts for up to 1,000 virtual
machines.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
61
Chapter 4: Solution Architecture Overview
EMC Confidential
VNX family storage arrays include the following components:
Hardware
resources

Storage processors (SPs)—Support block data with UltraFlex I/O technology
that supports Fibre Channel, iSCSI, and FCoE protocols The SPs provide access
for all external hosts, and for the file side of the VNX array.

Disk Processor Enclosure (DPE)—Is 3U in size, and houses the SPs and the first
tray of disks. The VNX5200, VNX5400, VNX5600, and VNX5800 use this
component.

X-Blades (or Data Movers)—Access data from the back end and provide host
access using the same UltraFlex I/O technology that supports the NFS, CIFS,
MPFS, and pNFS protocols. The X-Blades in each array are scalable and provide
redundancy to ensure that no single point of failure exists.

The Data Mover Enclosure (DME)—Is 2U in size and houses the Data Movers
(X-Blades). All VNX for file models use a DME.

Standby power supply (SPS)—Is 1U in size and provides enough power to each
SP to ensure that any data in-flight de-stages to the array’s vault area in the
event of a power failure. This ensures that no writes are lost. On restart of the
array, the pending writes are reconciled and made persistent.

Control Station—Is 1U in size and provides management functions to the
X-Blades. The Control Station is responsible for X-Blade failover. An optional
secondary Control Station ensures redundancy on the VNX array.

Disk Array Enclosures (DAE)—House the drives used in the array.
Table 3 lists the hardware used in this solution.
Table 3.
Solution hardware
Component
VMware
vSphere
servers
Configuration
CPU
1 vCPU per virtual machine
4 vCPUs per physical core
For 200 virtual machines:
 200 vCPUs
 Minimum of 50 physical CPUs
For 300 virtual machines:
 300 vCPUs
 Minimum of 75 physical CPUs
For 600 virtual machines:
 600 vCPUs
 Minimum of 150 physical CPUs
For 1,000 virtual machines:
 1,000 vCPUs
 Minimum of 250 physical CPUs
62
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Component
Configuration
Memory
2 GB RAM per virtual machine
2 GB RAM reservation per VMware vSphere host
For 200 virtual machines:
 Minimum of 400 GB RAM
 Add 2 GB for each physical server
For 300 virtual machines:
 Minimum of 600 GB RAM
 Add 2 GB for each physical server
For 600 virtual machines:
 Minimum of 1200 GB RAM
 Add 2 GB for each physical server
For 1,000 virtual machines:
 Minimum of 2,000 GB RAM
 Add 2 GB for each physical server
Network
Block
2 x 10 GbE NICs per server
2 HBA per server
File
4 x 10 GbE NICs per server
Note: Add at least one additional server to the infrastructure beyond the
minimum requirements to implement VMware vSphere High-Availability (HA)
functionality and to meet the listed minimums.
Network
infrastructure
Minimum
switching
capacity
Block
2 physical switches
2 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 ports per VMware vSphere server, for storage
network
2 ports per SP, for storage data
File
2 physical switches
4 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 x 10 GbE ports per Data Mover for data
EMC Backup
Avamar
Refer to EMC Backup and Recovery Options for
Data Domain
Refer to EMC Backup and Recovery Options for
VSPEX Private Clouds Design and
Implementation Guide.
VSPEX Private Clouds Design and
Implementation Guide.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
63
Chapter 4: Solution Architecture Overview
EMC Confidential
Component
EMC VNX
series
storage array
Configuration
Block
Common:
 1 x 1 GbE interface per control station for
management
 1 x 1 GbE interface per SP for management
 2 front end ports per SP
 system disks for VNX OE
For 200 virtual machines:
 EMC VNX5200
 75 x 600 GB 15k rpm 3.5-inch SAS drives
 4 x 200 GB flash drives
 3 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
For 300 virtual machines:
 EMC VNX5400
 110 x 600 GB 15k rpm 3.5-inch SAS drives
 6 x 200 GB flash drives
 4 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
For 600 virtual machines:
 EMC VNX5600
 220 x 600 GB 15k rpm 3.5-inch SAS drives
 10 x 200 GB flash drives
 8 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
For 1,000 virtual machines:
 EMC VNX5800
 360 x 600 GB 15k rpm 3.5-inch SAS drives
 16 x 200 GB flash drives
 12 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
64
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Component
Configuration
File
Common:
 2 x 10 GbE interfaces per Data Mover
 1 x 1 GbE interface per Control Station for
management
 1 x 1 GbE interface per SP for management
 system disks for VNX OE
For 200 virtual machines:
 EMC VNX5200
 2 Data Movers (active / standby)
 75 x 600 GB 15k rpm 3.5-inch SAS drives
 4 x 200 GB flash drives
 3 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
For 300 virtual machines:
 EMC VNX5400
 2 Data Movers (active / standby)
 110 x 600 GB 15k rpm 3.5-inch SAS drives
 6 x 200 GB flash drives
 4 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as hot spare
For 600 virtual machines:
 EMC VNX5600
 2 Data Movers (active / standby)
 220 x 600 GB 15k rpm 3.5-inch SAS drives
 10 x 200 GB flash drives
 8 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
For 1,000 virtual machines:
 EMC VNX5800
 3 Data Movers (2 x active /1 x standby)
 360 x 600 GB 15k rpm 3.5-inch SAS drives
 16 x 200 GB flash drives
 12 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
Note:: In VNX5800, it is recommended to run no more
than 600 virtual machines on a single active Data
Mover. Configure two active Data Movers (2 x active/1 x
standby) when scaling to 600 or larger in that case.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
65
Chapter 4: Solution Architecture Overview
EMC Confidential
Component
Shared
infrastructure
Configuration
In most cases, a customer environment already has infrastructure services
such as Active Directory, DNS, and other services configured. The setup of
these services is beyond the scope of this document.
If implemented without existing infrastructure, the new minimum
requirements are:
 2 physical servers
 16 GB RAM per server
 4 processor cores per server
 2 x 1 GbE ports per server
Note:: These services can be migrated into VSPEX post-deployment;
however, they must exist before VSPEX can be deployed.
Note:: The solution recommends using a 10 GbE network or an equivalent 1GbE network
infrastructure as long as the underlying requirements around bandwidth and redundancy are
fulfilled.
Software resources Table 4 lists the software used in this solution.
Table 4.
Solution software
Software
Configuration
VMware vSphere 5.5
vSphere server
Enterprise Edition
vCenter Server
Standard Edition
Operating system for vCenter Server
Windows Server 2008 R2 SP1 Standard Edition
Note: Any operating system that is supported
for vCenter can be used.
Microsoft SQL Server
Version 2008 R2 Standard Edition
Note: Any supported database for vCenter can
be used.
EMC VNX
VNX OE for file
8.0
VNX OE for block
05.33
EMC VSI for VMware vSphere: Unified
Storage Management
Check for latest version
EMC VSI for VMware vSphere: Storage
Viewer
EMC PowerPath/VE
Check for latest version
EMC backup
Avamar
66
Refer to EMC Backup and Recovery Options for
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Software
Configuration
Data Domain OS
VSPEX Private Clouds Design and Implementation
Guide.
Virtual machines (used for validation – not required for deployment)
Base operating system
Microsoft Window Server 2012 R2 Data Center
Edition
Server configuration guidelines
Overview
When designing and ordering the compute/server layer of the VSPEX solution
described below, several factors may impact the final purchase. From a virtualization
perspective, if a system workload is well understood, features such as Memory
Ballooning and Transparent Page Sharing can reduce the aggregate memory
requirement.
If the virtual machine pool does not have a high level of peak or concurrent usage,
reduce the number of vCPUs. Conversely, if the applications being deployed are
highly computational in nature, increase the number of CPUs and memory purchased.
Current VSPEX sizing guidelines specify a virtual CPU core to physical CPU core ratio
of 4:1. This ratio was based upon an average sampling of CPU technologies available
at the time of testing. As CPU technologies advance, OEM server vendors that are
VSPEX partners may suggest differing (normally higher) ratios. Please follow the
updated guidance supplied by your OEM server vendor.
Ivy Bridge updates
Testing on the release of Intel’s Ivy Bridge series processors shows significant
increases in VM density from the server resource perspective. If your server
deployment comprises Ivy Bridge processors, we recommend increasing the
vCPU/pCPU ratio from 4:1 to 8:1. This essentially halves the number of server cores
required to host the RVMs.
Figure 19 demonstrates results from tested configurations:
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
67
Chapter 4: Solution Architecture Overview
Figure 19. Ivy Bridge processor guidance
68
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
EMC Confidential
Chapter 4: Solution Architecture Overview
Table 5 lists the hardware resources used for the compute layer.
Table 5.
Hardware resources for the compute layer
Component
VMware
vSphere
servers
Configuration
CPU
1 vCPU per virtual machine
4 vCPUs per physical core
For 200 virtual machines:
 200 vCPUs
 Minimum of 50 physical CPUs
For 300 virtual machines:
 300 vCPUs
 Minimum of 75 physical CPUs
For 600 virtual machines:
 600 vCPUs
 Minimum of 150 physical CPUs
For 1,000 virtual machines:
 1,000 vCPUs
 Minimum of 250 physical CPUs
Memory
2 GB RAM per virtual machine
2 GB RAM reservation per VMware vSphere host
For 200 virtual machines:
 Minimum of 400 GB RAM
 Add 2 GB for each physical server
For 300 virtual machines:
 Minimum of 600 GB RAM
 Add 2 GB for each physical server
For 600 virtual machines:
 Minimum of 1200 GB RAM
 Add 2 GB for each physical server
For 1,000 virtual machines:
 Minimum of 2000 GB RAM
 Add 2GB for each physical server
Network
Block
2 x 10 GbE NICs per server
2 HBA per server
File
4 x 10 GbE NICs per server
Note: Add at least one additional server to the infrastructure beyond the
minimum requirements to implement VMware vSphere High Availability (HA)
functionality and to meet the listed minimums.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
69
Chapter 4: Solution Architecture Overview
EMC Confidential
Note: The solution recommends using a 10 GbE network or an equivalent 1GbE network
infrastructure as long as the underlying requirements around bandwidth and redundancy are
fulfilled.
VMware vSphere
memory
virtualization for
VSPEX
VMware vSphere 5.5 has a number of advanced features that help maximize
performance and overall resource utilization. The most important of these are in the
area of memory management. This section describes some of these features, and the
items to consider when using these features in the environment.
In general, virtual machines on a single hypervisor consume memory as a pool of
resources, as shown in Figure 20.
Figure 20. Hypervisor memory consumption
Understanding the technologies in this section enhances this basic concept.
Memory compression
Memory over-commitment occurs when more memory is allocated to virtual machines
than is physically present in a VMware vSphere host. Using sophisticated techniques,
such as ballooning and transparent page sharing, vSphere 5.5 can handle memory
over-commitment without any performance degradation. However, if memory usage
exceeds server capacity, vSphere might resort to swapping out portions of the
memory of a virtual machine.
70
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Non-Uniform Memory Access (NUMA)
vSphere 5.5 uses a NUMA load-balancer to assign a home node to a virtual machine.
Because the home node allocates virtual machine memory, memory access is local
and provides the best performance possible. Applications that do not directly support
NUMA also benefit from this feature.
Transparent page sharing
Virtual machines running similar operating systems and applications typically have
similar sets of memory content. Page sharing enables the hypervisor to reclaim any
redundant copies of memory pages and keep only one copy, which frees up the total
host memory consumption. If most of your application virtual machines run the same
operating system and application binaries, total memory usage can be reduced to
increase consolidation ratios.
Memory ballooning
By using a balloon driver loaded in the guest operating system, the hypervisor can
reclaim host physical memory if memory resources are under contention, with little or
no impact to the performance of the application.
Memory
configuration
guidelines
This section provides guidelines for allocating memory to virtual machines. The
guidelines outlined here take into account vSphere memory overhead and the virtual
machine memory settings.
vSphere memory overhead
Some associated overhead is required for the virtualization of memory resources. The
memory space overhead has two components:

The fixed system overhead for the VMkernel

Additional overhead for each virtual machine
Memory overhead depends on the number of virtual CPUs and configured memory for
the guest operating system.
Allocating memory to virtual machines
Many factors determine the proper sizing for virtual machine memory in VSPEX
architectures. With the number of application services and use cases available,
determining a suitable configuration for an environment requires creating a baseline
configuration, testing, and making adjustments for optimal results.
Network configuration guidelines
Overview
This section provides guidelines for setting up a redundant, highly available network
configuration. The guidelines outlined consider jumbo frames, VLANs, and LACP on
EMC unified storage. For detailed network resource requirements, refer to Table 6.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
71
Chapter 4: Solution Architecture Overview
Table 6.
EMC Confidential
Hardware resources for network
Component
Network
infrastructure
Configuration
Minimum
switching
capacity
Block
2 physical switches
2 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 ports per VMware vSphere server, for storage
network
2 ports per SP, for storage data
File
2 physical switches
4 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 x 10 GbE ports per Data Mover for data
Note: The solution may use 1 GbE network infrastructure as long as the underlying
requirements around bandwidth and redundancy are fulfilled.
VLAN
Isolate network traffic so that the traffic between hosts and storage, hosts and
clients, and management traffic all move over isolated networks. In some cases
physical isolation may be required for regulatory or policy compliance reasons; but in
many cases logical isolation with VLANs is sufficient.
This solution uses a minimum of three VLANs for:
72

Client access

Storage(for iSCSI and NFS)

Management
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Figure 21 depicts the VLANs and the network connectivity requirements for a blockbased VNX array.
Figure 21. Required networks for block storage
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
73
Chapter 4: Solution Architecture Overview
EMC Confidential
Figure 22 depicts the VLANs for file and the network connectivity requirements for a
file-based VNX array.
Figure 22. Required networks for file storage
Note: Figure 22 demonstrates the network connectivity requirements for a VNX array using
10 GbE connections. Create a similar topology for 1 GbE network connections.
The client access network is for users of the system, or clients, to communicate with
the infrastructure. The storage network provides communication between the
compute layer and the storage layer. Administrators use the management network as
a dedicated way to access the management connections on the storage array,
network switches, and hosts.
Note: Some best practices call for additional network isolation for cluster traffic,
virtualization layer communication, and other features. Implement these additional
networks if necessary.
Enable jumbo
frames (for iSCSI,
FCoE, and NFS)
74
This solution recommends setting MTU at 9,000 (jumbo frames) for efficient storage
and migration traffic. Most switch vendors also suggest also enabling baby jumbo
frames (setting MTU at 2158) to prevent frame fragmentation. Refer to the switch
vendor guidelines to enable jumbo frames on switch ports for storage and host ports
on the switches.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Link aggregation
(for NFS)
Chapter 4: Solution Architecture Overview
Link aggregation resembles an Ethernet channel, but uses LACP IEEE 802.3ad
standard. The IEEE 802.3ad standard supports link aggregations with two or more
ports. All ports in the aggregation must have the same speed and be full duplex. In
this solution, LACP is configured on VNX, combining multiple Ethernet ports into a
single virtual device. If a link is lost in the Ethernet port, the link fails over to another
port. All network traffic is distributed across the active links.
Storage configuration guidelines
Overview
This section provides guidelines for setting up the storage layer of the solution to
provide high-availability and the expected level of performance.
VMware vSphere 5.5 allows more than one method of storage when hosting virtual
machines. The tested solutions described below use different block protocols
(FC/FCoE/iSCSI) and NFS (for file), and the storage layout described adheres to all
current best practices. A customer or architect with the necessary training and
background can make modifications based on their understanding of the system
usage and load if required. However, the building blocks described in this document
ensure acceptable performance. The VSPEX storage building blocks section
documents specific recommendations for customization.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
75
Chapter 4: Solution Architecture Overview
EMC Confidential
Table 7 lists the hardware resources used for storage.
Table 7.
Hardware resources for storage
Component
EMC VNX
series
storage array
Configuration
Block
Common:
 1 x 1 GbE interface per Control Station for management
 1 x 1 GbE interface per SP for management
 2 front end ports per SP
 system disks for VNX OE
For 200 virtual machines:
 EMC VNX5200
 75 x 600 GB 15k rpm 3.5-inch SAS drives
 4 x 200 GB flash drives
 3 x 600 GB 15k rpm 3.5-inch SAS drives as hot spares
 1 x 200 GB flash drive as a hot spare
For 300 virtual machines:
 EMC VNX5400
 110x 600 GB 15k rpm 3.5-inch SAS drives
 6 x 200 GB flash drives
 4 x 600 GB 15k rpm 3.5-inch SAS drives as hot spares
 1 x 200 GB flash drive as a hot spare
For 600 virtual machines:
 EMC VNX5600
 220 x 600 GB 15k rpm 3.5-inch SAS drives
 10 x 200 GB flash drives
 8 x 600 GB 15k rpm 3.5-inch SAS drives as hot spares
 1 x 200 GB flash drive as a hot spare
For 1,000 virtual machines:
 EMC VNX5800
 360 x 600 GB 15k rpm 3.5-inch SAS drives
 16 x 200 GB flash drives
 12 x 600 GB 15k rpm 3.5-inch SAS drives as hot spares
 1 x 200 GB flash drive as a hot spare
76
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Component
Configuration
File
Common:
 2 x 10 GbE interfaces per Data Mover
 1 x 1 GbE interface per Control Station for management
 1 x 1 GbE interface per SP for management
 system disks for VNX OE
For 200 virtual machines:
 EMC VNX5200
 2 Data Movers (active / standby)
 75 x 600 GB 15k rpm 3.5-inch SAS drives
 4 x 200 GB flash drives
 3 x 600 GB 15k rpm 3.5-inch SAS drives as hot spares
 1 x 200 GB flash drive as hot spare
For 300 virtual machines:
 EMC VNX5400
 2 Data Movers (active / standby)
 110 x 600 GB 15k rpm 3.5-inch SAS drives
 6 x 200 GB flash drives
 4 x 600 GB 15k rpm 3.5-inch SAS drives as hot spares
 1 x 200 GB flash drive as hot spare
For 600 virtual machines:
 EMC VNX5600
 2 Data Movers (active / standby)
 220 x 600 GB 15k rpm 3.5-inch SAS drives
 10 x 200 GB flash drives
 8 x 600 GB 15k rpm 3.5-inch SAS drives as hot spares
 1 x 200 GB flash drive as a hot spare
For 1,000 virtual machines:
 EMC VNX5800
 3 Data Movers (2 x active /1 x standby)
 360 x 600 GB 15k rpm 3.5-inch SAS drives
 16 x 200 GB flash drives
 12 x 600 GB 15k rpm 3.5-inch SAS drives as hot spares
 1 x 200 GB flash drive as a hot spare
Note: For the VNX5800, it is recommended to run no more
than 600 virtual machines on a single active Data Mover.
Configure two active Data Movers (2 x active/1 x standby)
when scaling to 600 or larger in that case.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
77
Chapter 4: Solution Architecture Overview
VMware vSphere
storage
virtualization for
VSPEX
EMC Confidential
VMware ESXi provides host-level storage virtualization, virtualizes the physical
storage, and presents the virtualized storage to the virtual machines.
A virtual machine stores its operating system and all the other files related to the
virtual machine activities in a virtual disk. The virtual disk itself consists of one or
more files. VMware uses a virtual SCSI controller to present virtual disks to a guest
operating system running inside the virtual machines.
Virtual disks reside on a datastore. Depending on the protocol used, a datastore can
be either a VMware VMFS datastore, or an NFS datastore. An additional option, RDM,
allows the virtual infrastructure to connect a physical device directly to a virtual
machine.
Figure 23. VMware virtual disk types
VMFS
VMFS is a cluster file system that provides storage virtualization optimized for virtual
machines. Deploy over any SCSI-based local or network storage.
Raw Device Mapping (RDM)
VMware also provides RDM, which allows a virtual machine to directly access a
volume on the physical storage. Only use RDM with FC or iSCSI.
NFS
VMware supports using NFS from an external NAS storage system or device as a
virtual machine datastore.
VSPEX storage
building blocks
78
Sizing the storage system to meet virtual server IOPS is a complicated process. When
I/O reaches the storage array, several components such as the Data Mover (for filebased storage), SPs, back-end dynamic random access memory (DRAM) cache, FAST
VP or FAST Cache (if used), and disks serve that I/O. Customers must consider various
factors when planning and scaling their storage system to balance capacity,
performance, and cost for their applications.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
VSPEX uses a building block approach to reduce complexity. A building block is a set
of disk spindles that can support a certain number of virtual servers in the VSPEX
architecture. Each building block combines several disk spindles to create a storage
pool that supports the needs of the private cloud environment. Each building block
storage pool, regardless of the size, contains two flash drives with FAST VP storage
tiering to enhance metadata operations and performance.
VSPEX solutions have been engineered to provide a variety of sizing configurations
which afford flexibility when designing the solution. Customers can start out by
deploying smaller configurations and scale up as their needs grow. At the same time,
customers can avoid over-purchasing by choosing a configuration that closely meets
their needs. To accomplish this, VSPEX solutions can be deployed using one or both
of the scale-points below to obtain the ideal configuration all while guaranteeing a
given performance level.
Building block for 13 virtual servers
The first building block can contain up to 13 virtual servers, with two flash drives and
five SAS drives in a storage pool, as shown in Figure 24.
Figure 24. Storage layout building block for 13 virtual machines
This is the smallest building block qualified for the VSPEX architecture. This building
block can be expanded by adding five SAS drives and allowing the pool to restripe to
add support for 13 more virtual servers. For details about pool expansion and
restriping, refer to EMC VNX Virtual Provisioning Applied Technology White Paper.
Building block for 125 virtual servers
The second building block can contain up to 125 virtual servers. It contains two flash
drives and 45 SAS drives, as shown in Figure 25. The following sections outline an
approach to grow from 13 virtual machines to 125 virtual machines in a pool.
However, after reaching 125 virtual machines in a pool, do not expand to more virtual
machines. Create a new pool and start the scaling sequence again.
Figure 25. Storage layout building block for 125 virtual machines
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
79
Chapter 4: Solution Architecture Overview
EMC Confidential
Implement this building block with all of the resources in the pool initially, or expand
the pool over time as the environment grows. Table 8 lists the flash and SAS drive
requirements in a pool for different numbers of virtual servers.
Table 8.
Number of disks required for different number of virtual machines
Virtual servers
flash drives
SAS drives
13
2
5
26
2
10
39
2
15
52
2
20
65
2
25
78
2
30
91
2
35
104
2
40
117
2
45
125
2
45*
Note: Due to increased efficiency with larger stripes, the building block with 45 SAS drives
can support up to 125 virtual servers.
To grow the environment beyond 125 virtual servers, create another storage pool
using the building block method described here.
VSPEX private
cloud validated
maximums
VSPEX private cloud configurations are validated on the VNX5200, VNX5400,
VNX5600, and VNX5800 platforms. Each platform has different capabilities in terms
of processors, memory, and disks. For each array, there is a recommended maximum
VSPEX private cloud configuration. In addition to the VSPEX private cloud building
blocks, each storage array must contain the drives used for the VNX Operating
Environment (OE) and hot spare disks for the environment.
Note:
Allocate at least one hot spare for every 30 disks of a given type and size.
VNX5200
The VNX5200 is validated for up to 200 virtual servers. Figure 26 shows a typical
configuration.
80
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Figure 26. Storage layout for 200 virtual machines using VNX5200
This configuration uses the following storage layout:

Seventy-five 600 GB SAS drives are allocated to two block-based storage pools:
one RAID-5 (4+1) pool with 45 SAS disks for 125 virtual machines and one
RAID-5 (4+1) pool with 30 SAS disks for 75 virtual machines.
Note: To meet the load recommendations, all drives in the storage pool
must be 15k rpm and the same size. Storage layout algorithms may
produce sub-optimal results with drives of different sizes.

Four 200 GB flash drives are configured for Fast VP, two for each pool
configured as RAID 1/0.

Three 600 GB SAS drives are configured as hot spares.

One 200 GB flash drive is configured as a hot spare.

Enable FAST VP to automatically tier data to leverage differences in
performance and capacity.

FAST VP:

Works at the block storage pool level and automatically adjusts where data
is stored based on how frequently it is accessed.

Promotes frequently accessed data to higher tiers of storage in 256-MB
increments and migrates infrequently accessed data to a lower tier for cost
efficiency. This rebalancing of 256 MB data units, or slices, is part of a
regularly scheduled maintenance operation.

For block, allocate at least two LUNs to the vSphere cluster from a single
storage pool to serve as datastores for the virtual servers.

For file, allocate at least two NFS shares to the vSphere cluster from a single
storage pool to serve as datastores for the virtual servers.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
81
Chapter 4: Solution Architecture Overview

EMC Confidential
Optionally configure flash drives as FAST Cache (up to 1 TB) in the array. LUNs
or storage pools where virtual machines reside that have a higher than average
I/O requirement can benefit from the FAST Cache feature. These drives are an
optional part of the solution, and additional licenses may be required to use
the FAST Suite.
Using this configuration, the VNX5200 can support 200 virtual servers as defined in
Reference workload.
VNX5400
The VNX5400 is validated using for up to 300 virtual servers. There are multiple ways
to achieve this configuration with the building blocks. Figure 27 shows one potential
configuration.
Figure 27. Storage layout for 300 virtual machines using VNX5400
This configuration uses the following storage layout:

One hundred and ten 600 GB SAS drives are allocated to three block-based
storage pools: two pools with 45 SAS disks for 125 virtual machines each and
one pool with 20 SAS disks for 50 virtual machines.
Note:
The pool uses system drives for additional storage.
Note: If required, substitute larger drives for more capacity. To meet the
load recommendations, all drives in the storage pool must be 15k rpm and
the same size. Storage layout algorithms may produce sub-optimal results
with drives of different sizes.
82
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview

Six 200 GB flash drives are configured for Fast VP, two for each pool.

Four 600 GB SAS drives are configured as hot spares.

One 200 GB flash drive is configured as a hot spare.

Enable FAST VP to automatically tier data to leverage differences in
performance and capacity. FAST VP:

Works at the block storage pool level and automatically adjusts where data
is stored based on how frequently it is accessed.

Promotes frequently accessed data to higher tiers of storage in 256-MB
increments and migrates infrequently accessed data to a lower tier for cost
efficiency. This rebalancing of 256 MB data units, or slices, is part of a
regularly scheduled maintenance operation.

For block, allocate at least two LUNs to the vSphere cluster from a single
storage pool to serve as datastores for the virtual servers.

For file, allocate at least two NFS shares to the vSphere cluster from a single
storage pool to serve as datastores for the virtual servers.

Optionally configure flash drives as FAST Cache (up to 1 TB) in the array. LUNs
or storage pools where virtual machines reside that have a higher than average
I/O requirement can benefit from the FAST Cache feature. These drives are an
optional part of the solution, and additional licenses may be required to use
the FAST Suite.
Using this configuration, the VNX5400 can support 300 virtual servers as defined in
Reference workload.
VNX5600
The VNX5600 is validated for up to 600 virtual servers. There are multiple ways to
achieve this configuration with building blocks. Figure 28 shows one potential
configuration.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
83
Chapter 4: Solution Architecture Overview
EMC Confidential
Figure 28. Storage layout for 600 virtual machines using VNX 5600
There are several other ways to achieve this scale using the building blocks above.
This is simply one example.
This configuration uses the following storage layout:

Note:
84
Two hundred and twenty 600 GB SAS disks are allocated to five block-based
storage pools: four pools with 45 SAS disks for 125 virtual machines each and
one pool with 40 SAS disks for 100 virtual machines.
This pool does not use system drives and is not used for additional storage.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Note: If required, substitute larger drives for more capacity. To meet the load
recommendations, the drives all need to be 15k rpm and the same size. If using different
sizes, storage layout algorithms may give sub-optimal results.

Ten 200 GB flash drives are configured for Fast VP, two for each pool.

Eight 600 GB SAS drives are configured as hot spares.

One 200 GB flash drive is configured as a hot spare.

Enable FAST VP to automatically tier data to leverage differences in
performance and capacity. FAST VP:

Works at the block storage pool level and automatically adjusts where data
is stored based on access frequency.

Promotes frequently-accessed data to higher tiers of storage in 256 MB
increments, and migrates infrequently-accessed data to a lower tier for
cost-efficiency. This rebalancing of 256 MB data units, or slices, is part of a
regularly scheduled maintenance operation.

For block, allocate at least two LUNs to the vSphere cluster from a single
storage pool to serve as datastores for the virtual servers.

For file, allocate at least two NFS shares to the vSphere cluster from a single
storage pool to serve as datastores for the virtual servers.

Optionally configure flash drives as FAST Cache (up to 2 TB) in the array. These
drives are a required part of the solution, and additional licenses may be
required to use the FAST Suite.
Using this configuration, the VNX5600 can support 600 virtual servers as defined in
Reference workload.
VNX5800
VNX5800 can scale to 1,000 virtual servers. There are multiple ways to achieve this
configuration with building blocks. Figure 29 shows one way to achieve that level of
scale.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
85
Chapter 4: Solution Architecture Overview
Figure 29. Storage layout for 1,000 virtual machines using VNX 5800
86
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
EMC Confidential
Chapter 4: Solution Architecture Overview
This configuration uses the following storage layout:

Three hundred and sixty 600 GB SAS drives are allocated to eight block-based
storage pools: each with 45 SAS disks for 125 virtual machines
Note: The pool does not use system drives, and is not used for additional
storage.
Note: If required, substitute larger drives for more capacity. To meet the
load recommendations, all drives in the storage pool need to be 15k rpm
and the same size. Storage layout algorithms may produce sub-optimal
results with drives of different sizes.

Sixteen 200 GB flash drives are configured for FAST VP, two for each pool.

Twelve 600 GB SAS drives are configured as hot spares.

One 200 GB flash drive is configured as a hot spare.

Enable FAST VP to automatically tier data to leverage differences in
performance and capacity. FAST VP:

Works at the block storage pool level and automatically adjusts where data
is stored based on access frequency.

Promotes frequently accessed data to higher tiers of storage in 256-MB
increments and migrates infrequently-accessed data to a lower tier for costefficiency. This rebalancing of 256 MB data units, or slices, is part of a
regularly scheduled maintenance operation.
 For block, allocate at least two LUNs to the vSphere cluster from a single storage
pool to serve as datastores for the virtual servers.
 For file, allocate at least two NFS shares to the vSphere cluster from a single
storage pool to serve as datastores for the virtual servers.

Optionally configure flash drives as FAST Cache (up to 3 TB) in the array. These
drives are not a required part of the solution, and additional licenses may be
required in order to use the FAST Suite.
Using this configuration, VNX5800 can support 1,000 virtual servers as defined in
Reference workload.
Conclusion
The scale levels listed in Figure 30 highlight the entry points and supported maximum
values for the arrays in the VSPEX private cloud environment. The entry points
represent optimal model demarcations in terms of the number of virtual machines
within the environment. This helps you to determine which VNX array to choose
based upon your requirements. You can choose to configure any of the listed arrays
with a smaller number of virtual machines than the maximum values supported using
the building block approach described earlier.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
87
Chapter 4: Solution Architecture Overview
EMC Confidential
Figure 30. Maximum scale levels and entry points of different arrays
High-availability and failover
Overview
This VSPEX solution provides a high availability virtualized server, network, and
storage infrastructure. When implemented in accordance with this guide, business
operations survive from single-unit failures with little or no impact.
Virtualization layer Configure high availability in the virtualization layer, and enable the hypervisor to
automatically restart failed virtual machines. Figure 31 illustrates the hypervisor layer
responding to a failure in the compute layer.
Figure 31. High availability at the virtualization layer
By implementing high availability at the virtualization layer, even in a hardware
failure, the infrastructure attempts to keep as many services running as possible.
Compute layer
88
While the choice of servers to implement in the compute layer is flexible, use
enterprise class servers designed for the data center. This type of server has
redundant power supplies, as shown in Figure 32. Connect these servers to separate
power distribution units (PDUs) in accordance with your server vendor’s best
practices.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Figure 32. Redundant power supplies
To configure high-availability in the virtualization layer, configure the compute layer
with enough resources to meet the needs of the environment, even with a server
failure, as demonstrated in Figure 31.
Network layer
The advanced networking features of the VNX family provide protection against
network connection failures at the array. Each vSphere host has multiple connections
to user and storage Ethernet networks to guard against link failures, as shown in
Figure 33 and Figure 34. Spread these connections across multiple Ethernet switches
to guard against component failure in the network.
Figure 33. Network layer high availability (VNX) – Block storage
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
89
Chapter 4: Solution Architecture Overview
EMC Confidential
Figure 34. Network layer high availability (VNX) - File storage
Ensure there is no single point of failure to allow the compute layer to access storage
and communicate with users even if a component fails.
Storage layer
The VNX family is designed is for five 9s availability by using redundant components
throughout the array. All the array components are capable of continued operation in
case of hardware failure. The RAID disk configuration on the array provides protection
against data loss caused by individual disk failures, and the available hot spare
drives can replace a failing disk, as shown in Figure 35.
Figure 35. VNX series high availability
90
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
EMC storage arrays are highly available by default. When configured according to the
directions in their installation guides, no single unit failures result in data loss or
unavailability.
Validation test profile
Profile
characteristics
The VSPEX solution was validated with the environment profile described in Table 9.
Table 9.
Profile characteristics
Profile characteristic
Value
Number of virtual machines
200/300/600/1,000
Virtual machine OS
Windows Server 2012 Data Center
Edition
Processors per virtual machine
1
Number of virtual processors per physical CPU core
4
RAM per virtual machine
2 GB
Average storage available for each virtual machine
100 GB
Average IOPS per virtual machine
25 IOPS
Number of LUNs or NFS shares to store virtual
machine disks
6/10/16
Number of virtual machines per LUN or NFS share
62 or 63 per LUN or NFS share
Disk and RAID type for LUNs or NFS shares
RAID 5, 600 GB, 15k rpm, 3.5-inch
SAS disks
Note: This solution was tested and validated with Windows Server 2012 R2 as the
operating system for vSphere virtual machines, but Windows Server 2008. Windows Server
2008 R2, and Windows Server 2012 are also supported. The vSphere 5.5 configuration is the
same for all supported versions of Windows Server.
Backup and recovery configuration guidelines
For details regarding backup and recovery configuration for this VSPEX Private Cloud
solution, refer to EMC Backup and Recovery Options for VSPEX Private Clouds Design
and Implementation Guide.
Sizing guidelines
The following sections provide definitions of the reference workload used to size and
implement the VSPEX architectures. The sections include instructions on how to
correlate those reference workloads to customer workloads, and how that may
change the end delivery from the server and network perspective.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
91
Chapter 4: Solution Architecture Overview
EMC Confidential
Modify the storage definition by adding drives for greater capacity and performance,
and by adding features such as FAST Cache and FAST VP. The disk layouts provide
support for the appropriate number of virtual machines at the defined performance
level and for typical operations such as snapshots. Decreasing the number of
recommended drives or stepping down an array type can result in lower IOPS per
virtual machine, and a reduced user experience caused by higher response times.
Reference workload
Overview
When you move an existing server to a virtual infrastructure, you can gain efficiency
by right-sizing the virtual hardware resources assigned to that system.
Each VSPEX Proven Infrastructure balances the storage, network, and compute
resources needed for a set number of virtual machines, as validated by EMC. In
practice, each virtual machine has its own requirements that rarely fit a pre-defined
idea of a virtual machine. In any discussion about virtual infrastructures, first define a
reference workload. Not all servers perform the same tasks, and it is impractical to
build a reference that considers every possible combination of workload
characteristics.
To simplify the discussion, this section presents a representative customer reference
Defining the
reference workload workload. By comparing your actual customer usage to this reference workload, you
can determine which reference architecture to choose.
For the VSPEX solutions, the reference workload is a single virtual machine. Table 10
lists the characteristics of this virtual machine.
Table 10.
Virtual machine characteristics
Characteristic
Value
Virtual machine operating system
Microsoft Windows Server 2012 R2
Data Center Edition
Virtual processors per virtual machine
1
RAM per virtual machine
2 GB
Available storage capacity per virtual machine
100 GB
IOPS per virtual machine
25
I/O pattern
Random
I/O read/write ratio
2:1
This specification for a virtual machine does not represent any specific application.
Rather, it represents a single common point of reference by which to measure other
virtual machines.
92
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Applying the reference workload
Overview
When you consider an existing server for movement into a virtual infrastructure, you
have the opportunity to gain efficiency by right-sizing the virtual hardware resources
assigned to that system.
The solution creates a pool of resources that are sufficient to host a target number of
reference virtual machines with the characteristics shown in Table 10. Virtual
machines may not exactly match the specifications. In that case, define a single
specific customer virtual machine as the equivalent of some number of reference
virtual machines together, and assume these virtual machines are in use in the pool.
Continue to provision virtual machines from the resource pool until no resources
remain.
Example 1:
Custom-built
application
A small custom-built application server must move into this virtual infrastructure. The
physical hardware that supports the application is not fully used. A careful analysis of
the existing application reveals that the application can use one processor and needs
3 GB of memory to run normally. The I/O workload ranges between 4 IOPS at idle time
to a peak of 15 IOPS when busy. The entire application consumes about 30 GB on
local hard drive storage.
Based on these numbers, the resource pool needs the following resources:

CPU of one reference virtual machine

Memory of two reference virtual machines

Storage of one reference virtual machine

I/Os of one reference virtual machine
In this example, an appropriate virtual machine uses the resources for two of the
reference virtual machines. If implemented on a VNX5400 storage system, which can
support up to 300 virtual machines, resources for 298 reference virtual machines
remain.
Example 2: Point
of sale system
The database server for a customer’s point-of-sale system must move into this virtual
infrastructure. It is currently running on a physical system with four CPUs and 16 GB
of memory. It uses 200 GB of storage and generates 200 IOPS during an average busy
cycle.
The requirements to virtualize this application are:

CPUs of four reference virtual machines

Memory of eight reference virtual machine

Storage of two reference virtual machines

I/Os of eight reference virtual machines
In this case, the one appropriate virtual machine uses the resources of eight
reference virtual machines. If implemented on a VNX5400 storage system, which can
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
93
Chapter 4: Solution Architecture Overview
EMC Confidential
support up to 300 virtual machines, resources for 292 reference virtual machines
remain.
Example 3: Web
server
The customer’s web server must move into this virtual infrastructure. It is currently
running on a physical system with two CPUs and 8 GB of memory. It uses 25 GB of
storage and generates 50 IOPS during an average busy cycle.
The requirements to virtualize this application are:

CPUs of two reference virtual machines

Memory of four reference virtual machines

Storage of one reference virtual machine

I/Os of two reference virtual machines
In this case, the one appropriate virtual machine uses the resources of four reference
virtual machines. If implemented on a VNX5400 storage system which can support up
to 300 virtual machines, resources for 296 reference virtual machines remain.
Example 4:
Decision-support
database
The database server for a customer’s decision support system must move into this
virtual infrastructure. It is currently running on a physical system with 10 CPUs and 64
GB of memory. It uses 5 TB of storage and generates 700 IOPS during an average
busy cycle.
The requirements to virtualize this application are:

CPUs of 10 reference virtual machines

Memory of 32 reference virtual machines

Storage of 52 reference virtual machines

I/Os of 28 reference virtual machines
In this case, one virtual machine uses the resources of 52 reference virtual machines.
If implemented on a VNX5400 storage system, which can support up to 300 virtual
machines, resources for 248 reference virtual machines remain.
Summary of
examples
These four examples illustrate the flexibility of the resource pool model. In all four
cases, the workloads reduce the amount of available resources in the pool. All four
examples can be implemented on the same virtual infrastructure with an initial
capacity for 300 reference virtual machines, and resources for 234 reference virtual
machines remain in the resource pool as shown in Figure 36.
Figure 36. Resource pool flexibility
94
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
In more advanced cases, there may be tradeoffs between memory and I/O or other
relationships where increasing the amount of one resource decreases the need for
another. In these cases, the interactions between resource allocations become highly
complex, and are beyond the scope of the document. Examine the change in resource
balance and determine the new level of requirements. Add these virtual machines to
the infrastructure with the method described in the examples.
Implementing the solution
Overview
The solution described in this document requires a set of hardware to be available for
the CPU, memory, network, and storage needs of the system. These are general
requirements that are independent of any particular implementation except that the
requirements grow linearly with the target level of scale. This section describes some
considerations for implementing the requirements.
Resource types
The solution defines the hardware requirements in terms of four basic types of
resources:

CPU resources

Memory resources

Network resources

Storage resources
This section describes the resource types, their use in the solution, and key
implementation considerations in a customer environment.
CPU resources
The solution defines the number of CPU cores required, but not a specific type or
configuration. New deployments should use recent revisions of common processor
technologies. It is assumed that these perform as well as, or better than, the systems
used to validate the solution.
In any running system, monitor the utilization of resources and adapt as needed. The
reference virtual machine and required hardware resources in the solution assume
that there are four virtual CPUs for each physical processor core (4:1 ratio). Usually,
this provides an appropriate level of resources for the hosted virtual machines;
however, this ratio may not be appropriate in all use cases. Monitor the CPU
utilization at the hypervisor layer to determine if more resources are required.
Memory resources
Each virtual server in the solution must have 2 GB of memory. Because of budget
constraints, in a virtual environment, it is common to provision virtual machines with
more memory than is installed on the physical hypervisor server. The memory overcommitment technique takes advantage of the fact that each virtual machine does
not use all allocated memory. To oversubscribe the memory usage to some degree
makes business sense. The administrator has the responsibility to proactively
monitor the oversubscription rate so that it does not shift the bottleneck away from
the server and become a burden to the storage subsystem.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
95
Chapter 4: Solution Architecture Overview
EMC Confidential
If VMware ESXi runs out of memory for the guest operating systems, paging takes
place and results in extra I/O activity going to the vswap files. If the storage
subsystem is sized correctly, occasional spikes due to vswap activity may not cause
performance issues as transient bursts of load can be absorbed. However, if the
memory oversubscription rate is so high that the storage subsystem is severely
impacted by a continuing overload of vswap activity, add more disks for increased
performance. The administrator must decide if it is more cost effective to add more
physical memory to the server, or to increase the amount of storage. With memory
modules being a commodity, it is likely less expensive to choose the former option.
This solution is validated with statically assigned memory and no over-commitment
of memory resources. If a real-world environment uses memory over-commit, monitor
the system memory utilization and associated page file I/O activity consistently to
ensure that a memory shortfall does not cause unexpected results.
Network resources
The solution outlines the minimum needs of the system. If additional bandwidth is
needed, add capability to both the storage array and the hypervisor host to meet the
requirements. The options for network connectivity on the server depend on the type
of server. The storage arrays have a number of included network ports, and can add
ports using EMC UltraFlex I/O modules.
For reference purposes in the validated environment, each virtual machine generates
25 IOPS per second with an average size of 8 KB. This means that each virtual
machine generates at least 200 KB/s of traffic on the storage network. For an
environment rated for 100 virtual machines, this is calculated as a minimum of
approximately 20 MB/sec. This is well within the bounds of modern networks.
However, this does not consider other operations. For example, additional bandwidth
is needed for:

User network traffic

Virtual machine migration

Administrative and management operations
The requirements for each network depend on how it will be used. It is not practical to
provide precise numbers in this context. However, the network described in the
reference architecture for each solution must be sufficient to handle average
workloads for the previously described use cases.
Regardless of the network traffic requirements, always have at least two physical
network connections shared for a logical network so that a single link failure does not
affect the availability of the system. Design the network so that the aggregate
bandwidth in the event of a failure is sufficient to accommodate the full workload.
Storage resources
96
The storage building blocks described in this solution contain layouts for the disks
used in the validation of the system. Each layout balances the available storage
capacity with the performance capability of the drives. Consider a few factors when
examining storage sizing. Specifically, the array has a collection of disks assigned to
a storage pool. From that storage pool, provision datastores to the VMware vSphere
cluster. Each layer has a specific configuration defined for the solution and
documented in the deployment section of this guide in Chapter 5.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
It is acceptable to:
 Replace drives with larger capacity drives of the same type and performance
characteristics, or with higher performance drives of the same type and
capacity.
 Change the placement of drives in the drive shelves to comply with updated or
new drive shelf arrangements.
 Increase the scale using the building blocks with a larger number of drives up to
the limit defined in the VSPEX private cloud validated maximums section.
Observe the following best practices:

Use the latest best practices guidance from EMC regarding drive placement
within the shelf. Refer to Applied Best Practices Guide: EMC VNX Unified Best
Practice for Performance.

When expanding the capability of a storage pool using the building blocks
described in this document use the same type and size of drive in the pool.
Create a new pool for different to use different drive types and sizes. This
prevents uneven performance across the pool.

Configure at least one hot spare for every type and size of drive on the system.

Configure at least one hot spare for every 30 drives of a given type.
In other cases where there is a need to deviate from the proposed number and type of
drives specified, or from the specified pool and datastore layouts, ensure that the
target layout delivers the same or greater resources to the system and conforms to
EMC published best practices.
Implementation
summary
The requirements in the reference architecture are what EMC considers the minimum
set of resources to handle the workloads required based on the stated definition of a
reference virtual server. In any customer implementation, the load of a system varies
over time as users interact with the system. Add resources to a system if the virtual
machines differ significantly from the reference definition, and vary in the same
resource group.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
97
Chapter 4: Solution Architecture Overview
EMC Confidential
Quick assessment
Overview
An assessment of the customer environment helps to ensure that you implement the
correct VSPEX solution. This section provides an easy-to-use worksheet to simplify
the sizing calculations and help assess the customer environment.
First, summarize the applications planned for migration into the VSPEX private cloud.
For each application, determine the number of virtual CPUs, the amount of memory,
the required storage performance, the required storage capacity, and the number of
reference virtual machines required from the resource pool. Applying the reference
workload provides examples of this process.
Fill out a row in the worksheet for each application, as listed in Table 11.
Table 11.
Blank worksheet row
CPU
(virtual
CPUs)
Application
Example
application
Memory
(GB)
IOPS
Resource
requirements
Capacity
(GB)
Equivalent
reference
virtual
machines
NA
Equivalent
reference
virtual
machines
Fill out the resource requirements for the application. The row requires inputs on four
different resources: CPU, memory, IOPS, and capacity.
CPU requirements
Optimizing CPU utilization is a significant goal for almost any virtualization project. A
simple view of the virtualization operation suggests a one-to-one mapping between
physical CPU cores and virtual CPU cores regardless of the physical CPU utilization. In
reality, consider whether the target application can effectively use all CPUs
presented. Use a performance-monitoring tool, such as esxtop, on vSphere hosts to
examine the CPU Utilization counter for each CPU. If they are equivalent, implement
that number of virtual CPUs when moving into the virtual infrastructure. However, if
some CPUs are used and some are not, consider decreasing the number of virtual
CPUs required.
In any operation involving performance monitoring, collect data samples for a period
of time that includes all operational use cases of the system. Use either the maximum
or 95th percentile value of the resource requirements for planning purposes.
Memory
requirements
98
Server memory plays a key role in ensuring application functionality and
performance. Therefore, each server process has different targets for the acceptable
amount of available memory. When moving an application into a virtual environment,
consider the current memory available to the system and monitor the free memory by
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
using a performance-monitoring tool, such as VMware esxtop, to determine memory
efficiency.
In any operation involving performance monitoring, collect data samples for a period
of time that includes all operational use cases of the system. Use either the maximum
or 95th percentile value of the resource requirements for planning purposes.
Storage
performance
requirements
I/O operations per
second
The storage performance requirements for an application are usually the least
understood aspect of performance. Three components become important when
discussing the I/O performance of a system:

The number of requests coming in, or IOPS

The size of the request, or I/O size. For example, a request for 4 KB of data is
easier and faster to process than a request for 4 MB of data.

The average I/O response time, or I/O latency
The reference virtual machine calls for 25 IOPS. To monitor this on an existing system,
use a performance-monitoring tool such as VMware esxtop, which provides several
counters that can help. The most common are:
For block:

Physical Disk \Commands/sec

Physical Disk \Reads/sec

Physical Disk \Writes/sec

Physical Disk \ Average Guest MilliSec/Command
For file:

Physical Disk NFS Volume \Commands/sec

Physical Disk NFS Volume \Reads/sec

Physical Disk NFS Volume \Writes/sec

Physical Disk NFS Volume \Average Guest MilliSec/Command
The reference virtual machine assumes a 2:1 read: write ratio. Use these counters to
determine the total number of IOPS, and the approximate ratio of reads to writes for
the customer application.
I/O size
The I/O size is important because smaller I/O requests are faster and easier to
process than large I/O requests. The reference virtual machine assumes an average
I/O request size of 8 KB, which is appropriate for a large range of applications. Most
applications use I/O sizes that are even, powers of 2 –4 KB, 8 KB, 16 KB, 32 KB, and
so on are common. The performance counter does a simple average; it is common to
see 11 KB or 15 KB instead of the common I/O sizes.
The reference virtual machine assumes an 8 KB I/O size. If the average customer I/O
size is less than 8 KB, use the observed IOPS number. However, if the average I/O
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
99
Chapter 4: Solution Architecture Overview
EMC Confidential
size is significantly higher, apply a scaling factor to account for the large I/O size. A
safe estimate is to divide the I/O size by 8 KB and use that factor. For example, if the
application uses mostly 32 KB I/O requests, use a factor of four (32 KB/8 KB = 4). If
that application generates 100 IOPS at 32 KB, the factor indicates you should plan for
400 IOPS, since the reference virtual machine assumed 8 KB I/O sizes.
I/O latency
The average I/O response time, or I/O latency, is a measurement of how quickly the
storage system processes I/O requests. The VSPEX solutions meet a target average
I/O latency of 20 ms. The recommendations in this document allow the system to
continue to meet that target; however, monitor the system and re-evaluate the
resource pool utilization if needed.
To monitor I/O latency, use the “Physical Disk \ Average Guest MilliSec/Command”
counter (block storage) or “Physical Disk NFS Volume \ Average Guest
MilliSec/Command” counter (file storage) in esxtop. If the I/O latency is continuously
over the target, re-evaluate the virtual machines in the environment to ensure these
machines do not use more resources than intended.
Storage capacity
requirements
The storage capacity requirement for a running application is usually the easiest
resource to quantify. Determine the disk space used, and add an appropriate factor to
accommodate growth. For example, virtualizing a server that currently uses 40 GB of a
200 GB internal drive with anticipated growth of approximately 20 percent over the
next year requires 48 GB. In addition, reserve space for regular maintenance patches
and swapping files. Some file systems, such as Microsoft NTFS, degrade in
performance if they become too full.
Determining
equivalent
reference virtual
machines
With all of the resources defined, determine an appropriate value for the equivalent
reference virtual machines line by using the relationships in Table 12. Round all
values up to the closest whole number.
Table 12.
Reference virtual machine resources
Relationship between
requirements and
equivalent reference virtual
machines
Resource
Value for reference virtual
machine
CPU
1
Equivalent reference virtual
machines = resource
requirements
Memory
2
Equivalent reference virtual
machines = (Resource
Requirements)/2
IOPS
25
Equivalent reference virtual
machines = (resource
requirements)/25
Capacity
100
Equivalent reference virtual
machines = (resource
requirements)/100
100 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
For example, the point of sale system database used in Example 2: Point of sale
system requires four CPUs, 16 GB of memory, 200 IOPS, and 200 GB of storage. This
translates to four reference virtual machines of CPU, eight reference virtual machines
of memory, eight reference virtual machines of IOPS, and two reference virtual
machines of capacity. Table 13 demonstrates how that machine fits into the
worksheet row.
Table 13.
Example worksheet row
CPU
Application
Example
application
(virtual
CPUs)
Memory
(GB)
IOPS
Capacity
(GB)
Equivalent
reference
virtual
machines
Resource
requirements
4
16
200
200
N/A
Equivalent
reference
virtual
machines
4
8
8
2
8
Use the highest value in the row to fill in Equivalent reference virtual machines
column. As shown in Figure 37, the example requires eight reference virtual
machines.
Figure 37. Required resource from the reference virtual machine pool
Implementation example - Stage 1
A customer wants to build a virtual infrastructure to support one custom-built
application, one point of sale system, and one web server. The customer computes
the sum of the Equivalent reference virtual machines column on the right side of the
worksheet as listed in Table 14 to calculate the total number of reference virtual
machines required. The table shows the result of the calculation, along with the
value, rounded up to the nearest whole number.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 101
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 4: Solution Architecture Overview
Table 14.
EMC Confidential
Example applications – stage 1
Application
Server resources
Storage resources
CPU
Memory
(GB)
IOPS
Capacity
(GB)
Reference
virtual
machines
(virtual
CPUs)
Example
application
#1: Custom
built
application
Resource
requirements
1
3
15
30
NA
Equivalent
reference virtual
machines
1
2
1
1
2
Example
application
#2: Point of
sale system
Resource
requirements
4
16
200
200
NA
Equivalent
reference virtual
machines
4
8
8
2
8
Example
application
#3: Web
server
Resource
requirements
2
8
50
25
NA
Equivalent
reference virtual
machines
2
4
2
1
4
Total equivalent reference virtual machines
14
This example requires 14 reference virtual machines. According to the sizing
guidelines, one storage pool with 10 SAS drives and two or more flash drives
provides sufficient resources for the current needs and room for growth. You can use
a VNX5400, which supports up to 300 reference virtual machines.
102 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Figure 38. Aggregate resource requirements – stage 1
Figure 38 shows that twelve reference virtual machines are available after
implementing VNX5400 with 10 SAS drives and two flash drives.
Figure 39. Pool configuration – stage 1
Figure 39 shows the pool configuration in this example.
Implementation example – stage 2
Next, this customer must add a decision support database to this virtual
infrastructure. Using the same strategy, the number of reference virtual machines
required can be calculated, as shown in Table 15.
Table 15.
Example applications -stage 2
Application
Server resources
Storage resources
CPU
Memory
(GB)
IOPS
Capacity
(GB)
Reference
virtual
machines
(virtual
CPUs)
Example
application
#1: Custom
built
application
Resource
requirements
1
3
15
30
N/A
Equivalent
reference virtual
machines
1
2
1
1
2
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 103
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 4: Solution Architecture Overview
EMC Confidential
Server resources
Storage resources
Example
application
#2: Point of
sale system
Resource
requirements
4
16
200
200
N/A
Equivalent
reference virtual
machines
4
8
8
2
8
Example
application
#3: Web
server
Resource
requirements
2
8
50
25
N/A
Equivalent
reference virtual
machines
2
4
4
1
4
Example
application
#4: Decision
support
database
Resource
Requirements
10
64
700
5,120
N/A
Equivalent
reference virtual
machines
10
32
28
52
52
Total equivalent reference virtual machines
66
This example requires 66 reference virtual machines. According to the sizing
guidelines, one storage pool with 30 SAS drives and two flash drives or more
provides sufficient resources for the current needs and room for growth. You can
implement this storage layout with a VNX5400, for up to 300 reference virtual
machines.
Figure 40 shows that 12 reference virtual machines are available after implementing
VNX5400 with 30 SAS drives and two flash drives.
Figure 40. Aggregate resource requirements - stage 2
104 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Figure 41 shows the pool configuration in this example.
Figure 41. Pool configuration – stage 2
Implementation example – stage 3
With business growth, the customer must implement a much larger virtual
environment to support one custom built application, one point of sale system, two
web servers, and three decision Support databases. Using the same strategy,
calculate the number of equivalent reference virtual machines, as shown in Table 16.
Table 16.
Example applications - stage 3
Application
Server resources
Storage resources
CPU
Memory
(GB)
IOPS
Capacity
(GB)
Reference
virtual
machines
(virtual
CPUs)
Example
application
#1: Custom
built
application
Resource
requirements
1
3
15
30
N/A
Equivalent
reference virtual
machines
1
2
1
1
2
Example
application
#2: Point of
sale system
Resource
requirements
4
16
200
200
N/A
Equivalent
reference virtual
machines
4
8
8
2
8
Example
application
#3: Web
server #1
Resource
requirements
2
8
50
25
N/A
Equivalent
reference virtual
machines
2
4
4
1
4
Example
application
#4: Decision
support
database #1
Resource
requirements
10
64
700
5,120
N/A
Equivalent
reference virtual
machines
10
32
28
52
52
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 105
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 4: Solution Architecture Overview
EMC Confidential
Server resources
Storage resources
Example
application
#5: Web
server #2
Resource
requirements
2
8
50
25
N/A
Equivalent
reference virtual
machines
2
4
4
1
4
Example
application
#6: Decision
support
database #2
Resource
requirements
10
64
700
5,120
N/A
Equivalent
reference virtual
machines
10
32
28
52
52
Example
application
#7: Decision
support
database #3
Resource
requirements
10
64
700
5,120
N/A
Equivalent
reference virtual
machines
10
32
28
52
52
Total equivalent reference virtual machines
174
This example requires174 reference virtual machines. According to our sizing, two
pools with 70 SAS drives and 4 or more flash drives provides sufficient resources for
the current needs and room for growth. You can implement this storage layout with
VNX5400 for up to 300 reference virtual machines.
Figure 42 shows 16 reference virtual machines are available after implementing
VNX5400 with 70 SAS drives and 4 flash drives.
Figure 42. Aggregate resource requirements for stage 3
106 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
Figure 43. Pool configuration – stage 3
Figure 43 shows the pool configuration in this example.
Fine-tuning
hardware
resources
Usually, the process described determines the recommended hardware size for
servers and storage. However, in some cases there is a desire to further customize
the hardware resources available to the system. A complete description of system
architecture is beyond the scope of this document; however, additional
customization can be done at this point.
Storage resources
In some applications, application data must be separated from other workloads. The
storage layouts in the VSPEX architectures put all of the virtual machines in a single
resource pool. To achieve workload separation, purchase additional disk drives for
the application workload and add them to a dedicated pool.
With the method outlined in Determining equivalent reference virtual machines, it is
easy to build a virtual infrastructure scaling from 13 reference virtual machines to
1000 reference virtual machines with the building blocks described in VSPEX storage
building blocks, while keeping in mind the recommended limits of each storage array
documented in the VSPEX private cloud validated maximums.
Server resources
For some workloads the relationship between server needs and storage needs does
not match what is outlined in the reference virtual machine. Size the server and
storage layers separately in this scenario.
Figure 44. Customizing server resources
To do this, first total the resource requirements for the server components as shown
in Table 17. In the Server component totals line at the bottom of the worksheet, add
up the server resource requirements from the applications in the table.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 107
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 4: Solution Architecture Overview
EMC Confidential
Note: When customizing resources in this way, confirm that storage sizing is still
appropriate. The Storage component totals line at the bottom of Table 17 describes the
required amount of storage.
Table 17.
Server resource component totals
Application
Server resources
Storage resources
CPU
Memory
(GB)
IOPS
Capacity
(GB)
reference
virtual
machines
(virtual
CPUs)
Example
application
#1: Custom
built
application
Resource
requirements
1
3
15
30
N/A
Equivalent
reference virtual
machines
1
2
1
1
2
Example
application
#2: Point of
sale system
Resource
requirements
4
16
200
200
N/A
Equivalent
reference virtual
machines
4
8
8
2
8
Example
application
#3: Web
server #1
Resource
requirements
2
8
50
25
N/A
Equivalent
reference virtual
machines
2
4
4
1
4
Example
application
#4: Decision
support
database #1
Resource
requirements
10
64
700
5,120
N/A
Equivalent
reference virtual
machines
10
32
28
52
52
Example
application
#5: Web
server #2
Resource
requirements
2
8
50
25
N/A
Equivalent
reference virtual
machines
2
4
4
1
4
Example
application
#6: Decision
support
database #2
Resource
requirements
10
64
700
5,120
N/A
Equivalent
reference virtual
machines
10
32
28
52
52
Example
application
Resource
requirements
10
64
700
5,120
N/A
108 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 4: Solution Architecture Overview
#7: Decision
support
database #3
Equivalent
reference virtual
machines
Server resources
Storage resources
10
28
32
52
Total equivalent reference virtual machines
52
174
Server customization
Server component totals
39
227
NA
Storage customization
Storage component totals
2415
15640
NA
Storage component Equivalent reference virtual machines
97
157
NA
Total equivalent reference virtual machines - Storage
157
Note: Calculate the sum of the resource requirements row for each application, not the
equivalent reference virtual machines, to get the server and storage component totals.
In this example, the target architecture required 39 virtual CPUs and 227 GB of
memory. If four virtual machines per physical processor core are used, and memory
over-provisioning is not necessary, the architecture requires 10 physical processor
cores and 227 GB of memory. With these numbers, the solution can be effectively
implemented with fewer server and storage resources.
Note: Keep high-availability requirements in mind when customizing the resource pool
hardware.
Appendix C provides a blank Server Resource Component Worksheet.
EMC VSPEX Sizing
Tool
To simplify the sizing of this solution EMC has produced the VSPEX Sizing Tool. This
tool uses the same sizing process described in the section above, and also
incorporates sizing for other VSPEX solutions.
The VSPEX Sizing Tool enables you to input your resource requirements from the
customer’s answers in the qualification worksheet. After you complete the inputs to
the VSPEX Sizing Tool, the tool generates a series of recommendations, which allows
you to validate your sizing assumptions while providing platform configuration
information that meets those requirements. This tool can be accessed at the
following location: EMC VSPEX Sizing Tool
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 109
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 4: Solution Architecture Overview
.
110 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Chapter 5
VSPEX Configuration Guidelines
This chapter presents the following topics:
Overview ................................................................................................................112
Pre-deployment tasks ............................................................................................113
Customer configuration data ..................................................................................115
Prepare switches, connect network, and configure switches .................................115
Prepare and configure storage array ......................................................................118
Install and configure vSphere hosts .......................................................................133
Install and configure SQL server database .............................................................137
Install and configure VMware vCenter server .........................................................139
Summary................................................................................................................141
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 111
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
EMC Confidential
Overview
The deployment process consists of the stages listed in Table 18. After deployment,
integrate the VSPEX infrastructure with the existing customer network and server
infrastructure.
Table 18 lists the main stages in the solution deployment process. The table also
includes references to chapters that contain relevant procedures.
Table 18.
Deployment process overview
Stage
Description
reference
1
Verify prerequisites
Pre-deployment tasks
2
Obtain the deployment tools
Deployment prerequisites
3
Gather customer configuration
data
Customer configuration data
4
Rack and cable the
components
Refer to the vendor documentation.
5
Configure the switches and
networks, connect to the
customer network
Prepare switches, connect network, and
configure switches
6
Install and configure the VNX
Prepare and configure storage array
7
Configure virtual machine
datastores
Prepare and configure storage array
8
Install and configure the
servers
Install and configure vSphere hosts
9
Set up SQL Server (used by
VMware vCenter™)
Install and configure SQL server database
10
Install and configure vCenter
and virtual machine
networking
Configure database for VMware vCenter
112 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Pre-deployment tasks
Overview
Deployment
prerequisites
The pre-deployment tasks include procedures not directly related to environment
installation and configuration, and provide needed results at the time of installation.
Examples of pre-deployment tasks are collection of hostnames, IP addresses, VLAN
IDs, license keys, and installation media. Perform these tasks before the customer
visit to decrease the time required onsite.
Table 19.
Tasks for pre-deployment
Task
Description
reference
Gather
documents
Gather the related documents listed in
Appendix D. These documents provide
detail on setup procedures and
deployment best practices for the various
components of the solution.
References: EMC
documentation
Gather tools
Gather the required and optional tools for
the deployment. Use Table 20 to confirm
that all equipment, software, and
appropriate licenses are available before
starting the deployment process.
Table 20 Deployment
prerequisites checklist
Gather data
Collect the customer-specific
configuration data for networking,
naming, and required accounts. Enter this
information into the Customer
configuration data sheet for reference
during the deployment process.
Appendix B
Table 20 lists the hardware, software, and licenses required to configure the solution.
For additional information, refer to Table 3 and Table 4.
Table 20.
Deployment prerequisites checklist
Requirement
Description
reference
Hardware
Physical servers to host virtual servers:
Sufficient physical server capacity to
host 200, 300, 600 or 1,000 virtual
servers.
Table 3: Solution
hardware
VMware vSphere servers to host virtual
infrastructure servers
Note: The existing infrastructure
requirement may already meet this
requirement.
Switch port capacity and capabilities as
required by the virtual server
infrastructure.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 113
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
Requirement
EMC Confidential
Description
reference
EMC VNX5200 (200 virtual machines) or
EMC VNX5400 (300 virtual machines) or
EMC VNX5600 (600 virtual machines) or
EMC VNX5800 (1000 virtual machines):
Multiprotocol storage array with the
required disk layout.
Software
VMware ESXi™ installation media.
VMware vCenter Server installation
media.
EMC VSI for VMware vSphere: Unified
Storage Management.
EMC Online Support
EMC VSI for VMware vSphere: Storage
Viewer.
Microsoft Windows Server 2008 R2
installation media (suggested OS for
VMware vCenter).
Microsoft SQL Server 2008 R2 or newer
installation media.
Note: This requirement may be covered
in the existing infrastructure.
EMC vStorage API for Array Integration
Plug-in.
EMC Online Support
Microsoft Windows Server 2012 R2 Data
center installation media (suggested OS
for virtual machine guest OS) or
Windows Server 2008 R2 installation
media.
Licenses
VMware vCenter license key.
VMware ESXi license keys.
Microsoft Windows Server 2008 R2
Standard (or higher) license keys.
Microsoft Windows Server 2012 R2 Data
center license keys.
Note: An existing Microsoft Key
Management Server (KMS) may cover
this requirement.
Microsoft SQL Server license key.
Note: The existing infrastructure may
already meet this requirement.
114 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Customer configuration data
Assemble information such as IP addresses and hostnames as part of the planning
process to reduce time onsite.
Appendix B provides a table to maintain a record of relevant customer information.
Add, record, and modify information as needed during the deployment progress.
Additionally, complete the VNX File and Unified Worksheet, available on EMC Online
Support, to record the most comprehensive array-specific information.
Prepare switches, connect network, and configure switches
Overview
This section lists the network infrastructure requirements needed to support this
architecture. Table 21 provides a summary of the tasks for switch and network
configuration, and references for further information.
Table 21.
Tasks for switch and network configuration
Task
Description
reference
Configure
infrastructure
network
Configure storage array and ESXi host
infrastructure networking as specified
in Prepare and configure storage array
and Install and configure vSphere
hosts.
Prepare and configure storage
array and Install and configure
vSphere hosts.
Configure
VLANs
Configure private and public VLANs as
required.
Your vendor’s switch
configuration guide
Complete
network
cabling
Connect the switch interconnect ports.
Connect the VNX ports.
Connect the ESXi server ports.
Prepare network
switches
For validated levels of performance and high-availability, this solution requires the
switching capacity listed in Table 3. Do not use new hardware if existing infrastructure
meets the requirements.
Configure
infrastructure
network
The infrastructure network requires redundant network links for each ESXi host, the
storage array, the switch interconnect ports, and the switch uplink ports, to provide
both redundancy and additional network bandwidth. This is a required configuration
regardless of whether the network infrastructure or the solution already exists, or you
are deploying it alongside other components of the solution.
Figure 45 and Figure 46 show a sample redundant infrastructure for this solution. The
diagrams illustrate the use of redundant switches and links to ensure that there are
no single points of failure.
In Figure 45, converged switches provide customers with different protocol options
(FC, FCoE, or iSCSI) for storage networks. While existing FC switches are acceptable
for the FC protocol option, use 10 Gb Ethernet network switches for iSCSI.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 115
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
Figure 45. Sample network architecture – Block storage
116 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Figure 46 shows a sample redundant Ethernet infrastructure for file storage. The
diagram illustrates the use of redundant switches and links to ensure that no single
points of failure exist in the network connectivity.
Figure 46. Sample Ethernet network architecture – File storage
Configure VLANs
Configure jumbo
frames (iSCSI and
NFS only)
Ensure there are adequate switch ports for the storage array and ESXi hosts. Use a
minimum of two VLANs for:

Virtual machine networking and ESXi management. (These are customer- facing
networks. Separate them if required.)

Storage networking (iSCSI and NFS only) and vMotion.
Use jumbo frames for iSCSI and NFS protocols. Set the MTU to 9,000 for the switch
ports for the iSCSI or NFS storage network. Consult your switch configuration guide for
instructions.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 117
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
Complete network
cabling
EMC Confidential
Ensure the following:

All servers, storage arrays, switch interconnects, and switch uplinks plug into
separate switching infrastructures and have redundant connections.

There is a complete connection to the existing customer network.
Note: Ensure that unforeseen interactions do not cause service issues when you connect
the new equipment to the customer network.
Prepare and configure storage array
Implementation instructions and best practices may vary because of the storage
network protocol selected for the solution. There are three steps in each case:
1.
Configure the VNX.
2.
Provision storage to the hosts.
3.
Configure FAST VP.
4.
Optionally configure FAST Cache.
The following sections explain the options for each step separately, depending on
whether one of the block protocols (FC, FCoE, iSCSI), or the file protocol (NFS) is
selected:

For FC, FCoE, or iSCSI, refer to the instructions marked for block protocols.

For NFS, refer to the instructions marked for file protocols.
VNX configuration This section describes how to configure the VNX storage array for host access with
for block protocols block protocols such FC, FCoE, and iSCSI. In this solution, the VNX provides data
storage for VMware hosts.
Table 22.
Tasks for VNX configuration
Task
Description
Prepare the VNX
Physically install the VNX hardware
with the procedures in the product
documentation.
Set up the initial
VNX configuration
Configure the IP addresses and
other key parameters on the VNX.
Provision storage
for VMware hosts
Create the storage areas required
for the solution.
reference
 VNX5200 Unified
Installation Guide
 VNX5400 Unified
Installation Guide
 VNX5600 Unified
Installation Guide
 VNX5800 Unified
Installation Guide
 Unisphere System
Getting Started Guide
 Your vendor’s switch
configuration guide
118 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Prepare the VNX
The VNX5200, VNX5400, VNX5600 or VNX5800 Unified Installation Guide provides
instructions to assemble, rack, cable, and power up the VNX. There are no specific
setup steps for this solution.
Set up the initial VNX configuration
After completing the initial VNX setup, configure key information about the existing
environment so that the storage array can communicate. Configure the following
common items in accordance with your IT data center policies and existing
infrastructure information:

DNS

NTP

Storage network interfaces
For data connection using the FC or FCoE protocols:
Ensure that one or more servers are connected to the VNX storage system, either
directly or through qualified FC or FCoE switches. Refer to the EMC Host Connectivity
Guide for VMware ESX Server for more detailed instructions.
For data connection using iSCSI protocol:
Connect one or more servers to the VNX storage system, either directly or through
qualified IP switches. Refer to EMC Host Connectivity Guide for VMware ESX Server for
more detailed instructions.
Additionally, configure the following items in accordance with your IT data center
policies and existing infrastructure information:
1.
Set up a storage network IP address.
Logically isolate the other networks in the solution as described, in Chapter 3.
This ensures that other network traffic does not impact traffic between hosts
and storage.
2.
Enable jumbo frames on the VNX iSCSI ports.
Use jumbo frames for iSCSI networks to permit greater network bandwidth.
Apply the MTU size specified below across all network interfaces in the
environment:
a.
In Unisphere, select Settings > Network > Settings for Block.
b.
Select the appropriate iSCSI network interface.
c.
Click Properties.
d.
Set the MTU size to 9,000.
e.
Click OK to apply the changes.
The reference documents listed in Table 22 provide more information on how to
configure the VNX platform. Storage configuration guidelines provide more
information on the disk layout.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 119
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
EMC Confidential
Provision storage for VMware hosts
This section describes provisioning block storage for VMware hosts. To provision file
storage, refer to VNX configuration for file protocols.
Complete the following steps in Unisphere to configure LUNs on the VNX array to
store virtual servers:
1.
Create the number of storage pools required for the environment based on
the sizing information in Error! Reference source not found. This example
uses the array recommended maximums described in Chapter 4.
a.
Log in to Unisphere.
b.
Select the array for in this solution.
c.
Select Storage > Storage Configuration > Storage Pools.
d.
Click the Pools tab.
f.
Click Create.
Note:
Table 23.
The pool does not use system drives for additional storage.
Storage allocation table for block data
Number of
pools
Number of 15 K
SAS drives per
pool
Number of
flash drives
per pool
Number of
LUNs per pool
LUN size (TB)
200 virtual
machines
1
45
2
2
7
1
30
2
2
4
Total
2
75
4
4
2 x 7 TB LUNs
2 x 4 TB LUNs
300 virtual
machines
2
45
2
2
7
1
20
2
2
3
Total
3
110
6
6
4 x 7 TB LUNs
2 x 3 TB LUNs
600 virtual
machines
4
45
2
2
7
1
40
2
2
6
Total
5
220
10
10
8 x 7 TB LUNs
2 x 6 TB LUNs
1000 virtual
machines
8
45
2
2
7
Total
8
360
16
16
16 x 7 TB LUNs
Configuration
Note:: Each virtual machine occupies 102 GB in this solution, with 100 GB for the OS and user space,
and a 2 GB swap file.
120 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Create the hot spare disks at this point. Refer to the appropriate installation
guide for additional information.
Figure 26 depicts the target storage layout for 200 virtual machines.
Figure 27 depicts the target storage layout for 300 virtual machines.
Figure 28 depicts the target storage layout for 600 virtual machines.
Figure 29 depicts the target storage layout for 1,000 virtual machines.
3.
4.
VNX configuration
for file protocols
Use the pool created in Step 1 to provision thin LUNs:
a.
Click Storage > LUNs.
b.
Click Create.
c.
Select the pool created in Step 1. Always create two thin LUNs in one
physical storage pool. User Capacity depends on the specific number of
virtual machines. Refer to Table 23 for more information.
Create storage group and add LUNs and VMware servers:
a.
Click Hosts > Storage Groups.
b.
Click Create and enter a name for it.
c.
Select the created storage group.
d.
Click LUNs. In the Available LUNs panel, select all the LUNs created in the
previous steps. The Selected LUNs dialog appears.
e.
Configure and add the VMware hosts to the storage pool.
This section describes file storage provisioning for VMware.
Table 24.
Tasks for storage configuration
Task
Description
Reference
Prepare the VNX
Physically install the VNX hardware
with the procedures in the product
documentation.
 VNX5200 Unified
Set up the initial
VNX configuration
Create a network
interface
Configure the IP address
information and other key
parameters on the VNX.
Configure the IP address and
network interface information for
the NFS server.
Create a storage
pool for file
Create the pool structure and LUNs
to contain the file system.
Create file systems
Establish the file system that will be
shared with the NFS protocol and
export it to the VMware hosts.
Installation Guide
 VNX5400 Unified
Installation Guide
 VNX5600 Unified
Installation Guide
 VNX5800 Unified
Installation Guide
 Unisphere System Getting
Started Guide
 Your vendor’s switch
configuration guide
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 121
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
EMC Confidential
Prepare the VNX
TheVNX5200, VNX5400, VNX5600, or VNX5800 Unified Installation Guide provides
instructions on assemble, rack, cable, and power up the VNX. There are no specific
setup steps for this solution.
Set up the initial VNX configuration
After the initial VNX setup, configure key information about the existing environment
to allow the storage array to communicate with other devices in the environment.
Ensure one or more servers connect to the VNX storage system, either directly or
through qualified IP switches. Configure the following items in accordance with your
IT data center policies and existing infrastructure information:

DNS

NTP

Storage network interfaces

Storage network IP address

CIFS services and Active Directory Domain membership
Refer to EMC Host Connectivity Guide for Windows for more detailed instructions.
Enable jumbo frames on the VNX storage network interfaces
Use jumbo frames for storage networks to permit greater network bandwidth. Apply
the MTU size specified below across all network interfaces in the environment:
1.
In Unisphere, click Settings > Network > Settings for File.
2.
Select the appropriate network interface from the Interfaces tab.
3.
Click Properties.
4.
Set the MTU size to 9,000.
5.
Click OK to apply the changes.
The reference documents listed in Table 22 provide more information on how to
configure the VNX platform. Storage configuration guidelines provide more
information on the disk layout.
Create a network interface
A network interface maps to a NFS export. File shares provide access through this
interface.
Complete the following steps to create a network interface:
6.
Log in to the VNX.
7.
From the dashboard of the VNX, click Settings > Network > Settings For File.
8.
On the Interfaces tab, click Create.
122 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Figure 47.
Network settings For file dialog box
In the Create Network Interface wizard, complete the following steps:
9.
Select the Data Mover which will provide the file share.
10. Select the device name where the network interface will reside.
Note: Run the following command as nasadmin from the Control Station to
ensure the selected device has a link connected.
> server_sysconfig <datamovername> -pci
This command lists the link status (UP or DOWN) for all devices on the specified
Data Mover.
3.
Type an IP address for the interface.
4.
Type a Name for the interface.
5.
Type the netmask for the interface.
6.
The Broadcast Address field populates automatically after you provide the IP
address and netmask.
7.
Enter the MTU size for the interface to 9,000.
Note:
Ensure that all devices on the (switch, servers) have the same MTU size.
8.
If required, specify the VLAN ID.
9.
Click OK.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 123
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
Figure 48.
EMC Confidential
Create Interface dialog box
Create storage pool for file
Complete the following steps in Unisphere to configure LUNs on the VNX array to
store virtual servers:
1.
Create the number of storage pools required for the environment based on
the sizing information in Chapter 4. This example uses the array
recommended maximums as described in Chapter 4.
a.
Log in to Unisphere.
b.
Select the array for this solution.
c.
Click Storage > Storage Configuration > Storage Pools > Pools.
d.
Click Create.
Note:
The pool does not use system drives for additional storage.
Table 25. Storage allocation table for file
Number of
pools
Number of
15K SAS
drives per
pool
Number
of Flash
drives
per pool
Number of
LUNs per
pool
Number
of FS per
storage
pool for
file
LUN size
(GB)
FS size
(TB)
200 virtual
machines
1
45
2
20
2
800
5
1
30
2
20
2
600
4
Total
2
75
4
40
4
20 X 800GB
LUNs
20 X 600GB
LUNs
2 x 5TB
FS
2 x 4 TB
FS
Configuration
124 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
300 virtual
machines
2
45
2
20
2
800
7
1
20
2
20
2
400
3
Total
3
110
6
60
6
40 X 800GB
LUNs
20 X 400GB
LUNs
4 x 7 TB
FS
2 x 3 TB
FS
600 virtual
machines
4
45
2
20
2
800
7
1
40
2
20
2
700
6
Total
5
220
10
100
10
80 X 800GB
LUNs
20 X 700GB
LUNs
8 x 7 TB
FS
2 x 6 TB
FS
1000 virtual
machines
8
45
2
20
2
800
7
Total
8
360
16
160
16
160 X
800GB
LUNs
16 x 7 TB
FS
Create the hot spare disks. Refer to the EMC VNX5400 Unified Installation
Guide for additional information.
Figure 26depicts the target storage layout for 200 virtual machines.
Figure 27 depicts the target storage layout for 300 virtual machines.
Figure 28 depicts the target storage layout for 600 virtual machines.
Figure 29 depicts the target storage layout for 1,000 virtual machines.
2.
Use the pool created in step 1, and provision LUNs:
a.
Select Storage > LUNs.
b.
Click Create.
c.
Select the pool created in step1. . Under LUN Properties, uncheck the
Thin checkbox. For User Capacity, refer to Table 25 for detail on the size
of LUNs. The Number of LUNs to create depends on the disk number in
the pool. Refer to Table 25 for detail on the number of LUNs needed in
each pool.
Note: For FAST VP implementations, assign no more than 95% of the
available storage pool capacity for File.
3.
Connect the Provisioned LUNs to the Data Mover for file access:
a.
Click Hosts > Storage Groups.
b.
Select filestorage.
c.
Click Connect LUNs.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 125
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
d.
4.
EMC Confidential
In the Available LUNs panel, expand SP A and SP B and select all the
LUNs created in the previous steps. The Selected LUNs panel appears.
Click OK.
Rescan storage systems to detect newly-available storage.
a.
Click Storage tab.
b.
Under File Storage pane, click Rescan Storage Systems.
c.
Click OK to proceed in the window that opens.
Use a new Storage Pool for File to create multiple file systems.
Create file systems
A file system exports an NFS file share. Create a file system before creating the NFS
file share.
VNX requires a storage pool and a network interface to create a file system.
If no storage pools or interfaces exist, follow the steps in Create a network interface
and Create storage pool for file to create a storage pool and a network interface.
Create two thin file systems from each Storage Pool for File. Refer to Table 25 for
details on the number of file systems. Complete the following steps to create file
systems on the VNX for NFS File shares:
5.
Log in to Unisphere.
6.
Select Storage > Storage Configuration > File Systems.
7.
Click Create. The File System Creation Wizard appears.
8.
Specify the file system details:
a.
Select Storage Pool.
b.
Type a File System Name.
c.
Select a Storage Pool to contain the file system from.
d.
Select the Storage Capacity of the file system. Refer to Table 25 for detail
storage capacity.
e.
Select Thin Enabled.
f.
Multiply the number of terabytes specified for the file system in table 25
by 1048575 to get the file size in megabytes. Enter this figure in the
Maximum Capacity (MB) field.
g.
Select Data Mover (R/W) to own the file system.
Note: The selected Data Mover must have an interface defined on it.
h.
Click OK.
126 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Figure 49.
Create file system dialog box
The newly created file system appears on the File Systems tab.
9.
Click Mounts.
10. Select the created file system and then click Properties.
11. Select Set Advanced Options.
12. Select Direct Writes Enabled.
13. Select CIFS Sync Writes Enabled.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 127
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
EMC Confidential
Figure 50. Direct Writes Enabled checkbox
14. Click OK.
15. Export the file systems using NFS, and give root access to ESXi servers.
a. Click Storage > Shared Folders > NFS.
b. Click Create.
16. In the dialog, add the IP addresses of all ESXi servers in Read/Write Hosts and
Root Hosts.
FAST VP
configuration
This procedure applies to both file and block storage implementations. Complete the
following steps to configure FAST VP. Assign two flash drives in each block-based
storage pool:
17. Navigate to the block storage pool created in the previous step using
Unisphere. Select the storage pool to configure FAST VP.
18. Click Properties for a specific storage pool to open the Storage Pool Properties
dialog. Figure 51 shows the tiering information for a specific FAST pool.
Note: The Tier Status area shows FAST relocation information specific to the
selected pool.
19. Select the scheduled relocation at the pool level from the Auto-Tiering list.
Select either Scheduled (recommended) or Manual.
128 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
20.
In the Tier Details area, you can see the exact distribution of your data.
Figure 51. Storage Pool Properties dialog box
You can also connect to the relocation Schedule for the array using the button
in the top right corner to access the Manage Auto-Tiering dialog box as shown
in Figure 52.
Figure 52. Manage Auto-Tiering dialog box
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 129
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
EMC Confidential
Use the status dialog box to control the Data Relocation Rate. The default rate
is set to Medium so as not to significantly affect host I/O.
Note: FAST VP is an automated tool that provides the ability to create a relocation
schedule. Schedule the relocations during off-hours to minimize any potential
performance impact.
FAST Cache
configuration
Optionally, configure FAST Cache. To configure FAST Cache on the storage pools for
this solution, complete the following steps:
Note: The flash drives listed in the sizing section of Chapter 4 are intended for use with
FAST VP and configured in the section above. FAST Cache is an optional component of this
solution which can provide improved performance as outlined in Chapter 3.
2.
Configure flash drives as FAST Cache:
a.
Click Properties from the dashboard or Manage Cache in the left-hand
pane of the Unisphere interface to access the Storage System Properties
dialog box, as shown in Figure 53.
b.
Click the FAST Cache tab to view FAST Cache information.
Figure 53. Storage System Properties dialog box
c.
Click Create to open the Create FAST Cache dialog box, as shown in
Figure 54.
130 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
The RAID Type field displays as RAID 1 after creating the FAST Cache. You
can choose the number of flash drives from this screen. The bottom of
the screen shows the flash drives used to create FAST Cache. Select
Manual to choose the drives manually.
d.
Refer to Storage configuration guidelines to determine the number of
flash drives needed in this solution.
Note: If a sufficient number of flash drives are not available, FLARE
displays an error message and does not create FAST Cache.
Figure 54.
Create FAST Cache dialog box
21. Enable FAST Cache on the storage pool.
If a LUN is created in a storage pool, you can only configure FAST Cache for
that LUN at the storage pool level. All the LUNs created in the storage pool
have FAST Cache enabled or disabled. Configure LUNs from the Advanced tab
in the Create Storage Pool dialog box, as shown in Figure 55. After installing
FAST Cache on the VNX series, it is enabled by default at storage pool
creation.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 131
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
EMC Confidential
Figure 55. Advanced tab in the Create Storage Pool dialog box
If the storage pool is created, use the Advanced tab in the Storage Pool
Properties dialog box to configure FAST Cache as shown in Figure 56.
Figure 56. Advanced tab in the Storage Pool Properties dialog box
Note: The VNX FAST Cache feature does not cause an instantaneous
performance improvement. The system must collect data about access
patterns and promote frequently used information into the cache. This
process can take a few hours during which the performance of the array
steadily improves.
132 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Install and configure vSphere hosts
Overview
This section provides the requirements for the installation and configuration of the
ESXi hosts and infrastructure servers required to support the architecture. Table 26
describes the tasks that must be completed.
Table 26.
Install ESXi
Tasks for server installation
Task
Description
Reference
Install ESXi
Install the ESXi hypervisor on the
physical servers that are
deployed for the solution.
vSphere Installation and Setup
Guide
Configure ESXi
networking
Configure ESXi networking
including NIC trunking, VMkernel
ports, and virtual machine port
groups and jumbo frames.
vSphere Networking
Install and
configure
PowerPath/VE
(block storage
only)
Install and configure
PowerPath/VE to manage
multipathing for VNX LUNs.
PowerPath VE for VMware vSphere
Installation and Administration
Guide.
Connect
VMware
datastores
Connect the VMware datastores
to the ESXi hosts deployed for
the solution.
vSphere Storage Guide
Plan virtual
machine
memory
allocations
Ensure that VMware memory
management technologies are
configured properly for the
environment.
vSphere Installation and Setup
Guide
When starting the servers being used for ESXi, confirm or enable the hardwareassisted CPU virtualization and the hardware-assisted MMU virtualization setting in
the BIOS for each server. If the servers have a RAID controller, configure mirroring on
the local disks.
Boot the ESXi install media and install the hypervisor on each of the servers. ESXi
requires hostnames, IP addresses, and a root password for installation. Appendix B
provides appropriate values.
In addition, install the HBA drivers or configure iSCSI initiators on each ESXi host. For
details refer to EMC Host Connectivity Guide for VMware ESX Server.
Configure ESXi
networking
During the installation of VMware ESXi, a standard virtual switch (vSwitch) is created.
By default, ESXi chooses only one physical NIC as a virtual switch uplink. To maintain
redundancy and bandwidth requirements, add an additional NIC either by using the
ESXi console or by connecting to the ESXi host from the vSphere Client.
Each VMware ESXi server must have multiple interface cards for each virtual network
to ensure redundancy and provide network load balancing and network adapter
failover.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 133
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
EMC Confidential
VMware ESXi networking configuration, including load balancing and failover options,
are described in vSphere Networking. Choose the appropriate load balancing option
based on what is supported by the network infrastructure.
Create VMkernel ports as required, based on the infrastructure configuration:

VMkernel port for storage network (iSCSI and NFS protocols)

VMkernel port for VMware vMotion

Virtual server port groups (used by the virtual servers to communicate on the
network)
vSphere Networking describes the procedure for configuring these settings. Refer to
Appendix D for more information.
Jumbo frames (iSCSI and NFS only)
Enable jumbo frames for the NIC if using NIC for iSCSI and NFS data. Set the MTU to
9,000. Consult your NIC vendor’s configuration guide for instructions.
Install and
configure
PowerPath/VE
(block only)
To improve and enhance the performance and capabilities of VNX storage array,
install PowerPath/VE on the VMware vSphere host. For detailed installation steps,
refer to the PowerPath VE for VMware vSphere Installation and Administration Guide.
Connect VMware
datastores
Connect the datastores configured in the Install and configure vSphere hosts section
to the appropriate ESXi servers. These include the datastores configured for:

Virtual server storage

Infrastructure virtual machine storage (if required)

SQL Server storage (if required)
The vSphere Storage Guide provides instructions on how to connect the VMware
datastores to the ESXi host. Refer to Appendix D for more information.
Plan virtual
machine memory
allocations
Server capacity in the solution is required for two purposes:

To support the new virtualized server infrastructure

To support the required infrastructure services such as
authentication/authorization, DNS, and databases
For information on minimum infrastructure requirements, refer to Table 3. If existing
infrastructure services meet the requirements, the hardware listed for infrastructure
services is not required.
Memory configuration
When configuring server memory, properly size and configure the solution. This
section provides an overview on memory allocation for the virtual servers, and factors
in vSphere overhead and the virtual machine configuration.
134 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
ESXi memory management
Memory virtualization techniques allow the vSphere hypervisor to abstract physical
host resources such as memory to provide resource isolation across multiple virtual
machines, and avoid resource exhaustion. In cases where advanced processors, such
as Intel processors with EPT support, are deployed, this abstraction takes place
within the CPU. Otherwise, this process occurs within the hypervisor itself.
vSphere employs the following memory management techniques:

Allocation of memory resources greater than those physically available to the
virtual machine is known as memory over-commitment.

Identical memory pages that are shared across virtual machines are merged
with a feature known as transparent page sharing. Duplicate pages return to the
host free memory pool for reuse.

Memory compression - ESXi stores pages, which would otherwise be swapped
out to disk through host swapping, in a compressed cache located in the main
memory.

Memory ballooning relieves host resource exhaustion. This process requests
free pages to be allocated from the virtual machine to the host for reuse.

Hypervisor swapping causes the host to force arbitrary virtual machine pages
out to disk.
Additional information can be obtained from the Understanding Memory Resource
Management in VMware vSphere 5.0 White Paper.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 135
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
EMC Confidential
Virtual machine memory concepts
Figure 57 shows the memory settings parameters in the virtual machine.
Figure 57. Virtual machine memory settings
The memory settings are:

Configured memory—Physical memory allocated to the virtual machine at the
time of creation.

Reserved memory—Memory that is guaranteed to the virtual machine.

Touched memory—Memory that is active or in use by the virtual machine.

Swappable—Memory de-allocated from the virtual machine if the host is under
memory pressure from other virtual machine s with ballooning, compression, or
swapping.
The recommended best practices are:

Do not disable the default memory reclamation techniques. These lightweight
processes enable flexibility with minimal impact to workloads.

Intelligently size memory allocation for virtual machines. Over-allocation
wastes resources, while under-allocation causes performance impacts that can
affect other virtual machine sharing resources.
Over-committing can lead to resource exhaustion if the hypervisor cannot
procure memory resources. In severe cases when hypervisor swapping is
encountered, virtual machine performance might be adversely affected.
Having performance baselines for your virtual machine workloads assists in
this process.
Additional information on tools such as esxtop is in the Interpreting esxtop Statistics
document.
136 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Install and configure SQL server database
Overview
Table 27 describes how to set up and configure a Microsoft SQL Server database for
the solution. At the end of this chapter, you will have SQL Server installed on a virtual
machine, with the databases required by VMware vCenter configured for use.
Table 27.
Create a virtual
machine for SQL
Server
Tasks for SQL Server database setup
Task
Description
reference
Create a
virtual
machine for
SQL Server
Create a virtual machine to host
SQL Server. Verify that the virtual
server meets the hardware and
software requirements.
http://msdn.microsoft.com
Install
Microsoft
Windows on
the virtual
machine
Install Microsoft Windows Server
2008 R2 on the virtual machine
created to host SQL Server.
http://technet.microsoft.com
Install SQL
Server
Install SQL Server on the virtual
machine designated for that
purpose.
http://technet.microsoft.com
Configure
database for
VMware
vCenter.
Create the database required for
the vCenter server on the
appropriate datastore.
Preparing vCenter Server Databases
Configure
database for
VMware
Update
Manager
Create the database required for
Update Manager on the
appropriate datastore.
Preparing the Update Manager
Database
Create the virtual machine with enough computing resources on one of the ESXi
servers designated for infrastructure virtual machines. Use the datastore designated
for the shared infrastructure.
Note: The customer environment may already contain SQL Server for this role. In that
case, refer to the Configure database for VMware vCenter section.
Install Microsoft
Windows on the
virtual machine
The SQL Server service must run on Microsoft Windows. Install the required Windows
version on the virtual machine, and select the appropriate network, time, and
authentication settings.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 137
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
Install SQL Server
EMC Confidential
Install SQL Server on the virtual machine with the SQL Server installation media.
One of the installable components in the SQL Server installer is the SQL Server
Management Studio (SSMS). Install this component on the SQL server directly, and
on an administrator console.
In many implementations, you may want to store data files in locations other than the
default path.
To change the default path for storing data files:
3.
Right-click the server object in SSMS and select Database Properties. The
Properties window appears.
4.
Change the default data and log directories for new databases created on
the server.
Note: For high-availability, install SQL Server on a Microsoft Failover Cluster, or on a
virtual machine protected by VMware VMHA clustering. Do not combine these technologies.
Configure
database for
VMware vCenter
To use VMware vCenter in this solution, create a database for the service. The
requirements and steps to configure the vCenter Server database correctly are
covered in the section, Preparing vCenter Server Databases. Refer to the list of
documents in Appendix D for more information.
Note: Do not use the Microsoft SQL Server Express–based database option for this
solution.
Create individual login accounts for each service accessing a SQL Server database.
Configure
database for
VMware Update
Manager
To use VMware Update Manager in this solution, create a database for the service.
The requirements and steps to configure the Update Manager database are covered
in Configure database for VMware Update Manager . Create individual login accounts
for each service accessing a database on SQL Server. Consult your database
administrator for your organization’s policy.
138 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
Install and configure VMware vCenter server
Overview
This section provides information on how to configure the VMware vCenter. Complete
the tasks in Table 28.
Table 28.
Tasks for vCenter configuration
Task
Description
Reference
Create the vCenter host
virtual machine
Create a virtual machine to be
used for VMware vCenter
Server.
vSphere Virtual Machine
Administration
Install vCenter guest
operating system
Install Windows Server 2008
R2 Standard Edition on the
vCenter host virtual machine.
Update the virtual
machine
Install VMware Tools, enable
hardware acceleration, and
allow remote console access.
vSphere Virtual Machine
Administration
Create vCenter ODBC
connections
Create the 64-bit vCenter and
32-bit vCenter Update
Manager ODBC connections.
vSphere Installation and Setup
Installing and Administering
VMware vSphere Update
Manager
Install vCenter Server
Install vCenter Server
software.
vSphere Installation and Setup
Install vCenter Update
Manager
Install vCenter Update
Manager software.
Installing and Administering
VMware vSphere Update
Manager
Create a virtual data
center
Create a virtual data center.
vCenter Server and Host
Management
Apply vSphere license
keys
Type the vSphere license
keys in the vCenter licensing
menu.
vSphere Installation and Setup
Add ESXi hosts
Connect vCenter to ESXi
hosts.
vCenter Server and Host
Management
Configure vSphere
clustering
Create a vSphere cluster and
move the ESXi hosts into it.
vSphere Resource
Management
Perform array ESXi host
discovery
Perform ESXi host discovery
from the Unisphere console.
Using EMC VNX Storage with
VMware vSphere–TechBook
Install the vCenter
Update Manager plug-in
Install the vCenter Update
Manager plug-in on the
administration console.
Installing and Administering
VMware vSphere Update
Manager
Install the EMC VNX UEM
CLI
Install the EMC VNX UEM
command line interface on
the administration console.
EMC VSI for VMware vSphere:
Unified Storage Management—
Product Guide
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 139
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
EMC Confidential
Task
Description
Reference
Install the EMC VSI
plug-in
Install the EMC Virtual
Storage Integrator plug-in on
the administration console.
EMC VSI for VMware vSphere:
Unified Storage Management—
Create a virtual machine
in vCenter
Create a virtual machine
using vCenter
vSphere Virtual Machine
Administration
Perform partition
alignment, and assign
File Allocation Unite Size
Using Diskpart.exe to Perform
Partition Alignment, Assign
Drive Letters, and Assign File
Allocation Unit Size of virtual
machine’s disk drive
http://technet.microsoft.com/
Create a template virtual
machine
Create a template virtual
machine from the existing
virtual machine.
vSphere Virtual Machine
Administration
Product Guide
Create a customization
specification at this time.
Deploy virtual machines
from the template virtual
machine
Create the vCenter
host virtual
machine
Deploy the virtual machines
from the template virtual
machine.
vSphere Virtual Machine
Administration
To deploy the VMware vCenter Server as a virtual machine on an ESXi server installed
as part of this solution, connect directly to an infrastructure ESXi server using the
vSphere Client.
Create a virtual machine on the ESXi server with the customer guest OS configuration,
using the infrastructure server datastore presented from the storage array.
The memory and processor requirements for the vCenter Server depend on the
number of ESXi hosts and virtual machines managed. The requirements are in the
vSphere Installation and Setup Guide.
Install vCenter
guest OS
Install the guest OS on the vCenter host virtual machine. VMware recommends using
Windows Server 2008 R2 Standard Edition.
Create vCenter
ODBC connections
Before installing vCenter Server and vCenter Update Manager, create the ODBC
connections required for database communication. These ODBC connections use SQL
Server authentication for database authentication. Appendix C provides a place to
record SQL Server login information.
Install vCenter
Server
Install vCenter Server by using the VMware VIMSetup installation media. Use the
customer-provided username, organization, and vCenter license key when installing
vCenter.
Apply vSphere
license keys
To perform license maintenance, log in to vCenter Server and select the
Administration > Licensing menu from the vSphere Client. Use the vCenter License
140 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 5: VSPEX Configuration Guidelines
console to enter the license keys for the ESXi hosts. The keys can then be applied to
the ESXi hosts as they are imported into vCenter.
Install the EMC VSI Integrate the VNX storage system with VMware vCenter by using EMC VSI for VMware
vSphere: Unified Storage Management. Administrators can use the plug-in to manage
plug-in
VNX storage tasks from the vCenter hypervisor.
After installing the plug-in on the vSphere console, administrators can use vCenter to:

Create NFS datastores on VNX and mount them on ESXi servers.

Create LUNs on VNX and map them to ESXi servers.

Extend NFS datastores/LUNs.

Create Fast or full clones of virtual machines for NFS file storage.
Create a virtual machine in vCenter to use as a virtual machine template. After you
Create a virtual
machine in vCenter install the virtual machine, then install the software, and change the Windows and
application settings.
Refer to vSphere Virtual Machine Administration on the VMware website to create a
virtual machine.
Perform partition
alignment, and
assign File
Allocation Unite
Size
Perform disk partition alignment on virtual machines with operation systems prior to
Windows Server 2008. Align the disk drive with an offset of 1,024 KB, and format the
disk drive with a file allocation unit (cluster) size of 8 KB.
Create a template
virtual machine
Convert a virtual machine into a template. Create a customization specification when
creating a template..
Refer to the article Disk Partition Alignment Best Practices for SQL Server to perform
partition alignment, assign drive letters, and assign file allocation unit size using
diskpart.exe
Refer to vSphere Virtual Machine Administration to create the template and
specification.
Refer to vSphere Virtual Machine Administration to deploy the virtual machines with
Deploy virtual
machines from the the virtual machine template and the customization specification.
template virtual
machine
Summary
This chapter presents the required steps to deploy and configure the various aspects
of the VSPEX solution, which includes both the physical and logical components. At
this point, the VSPEX solution is fully functional
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 141
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 5: VSPEX Configuration Guidelines
142 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
EMC Confidential
Chapter 6: Verifying the Solution
Chapter 6
Verifying the Solution
This chapter presents the following topics:
Overview ................................................................................................................144
Post-install checklist .............................................................................................145
Deploy and test a single virtual server ...................................................................145
Verify the redundancy of the solution components ................................................145
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 143
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 6: Verifying the Solution
EMC Confidential
Overview
This chapter provides a list of items to review after configuring the solution. The goal
of this chapter is to verify the configuration and functionality of specific aspects of
the solution, and ensure that the configuration meets core availability requirements.
Complete the tasks listed in Table 29.
Table 29.
Tasks for testing the installation
Task
Description
Reference
Post install
checklist
Verify that sufficient virtual ports
exist on each vSphere host virtual
switch.
vSphere Networking
Verify that each vSphere host has
access to the required datastores
and VLANs.
 vSphere Storage Guide
Verify that the vMotion interfaces
are configured correctly on all
vSphere hosts.
vSphere Networking
Deploy and
test a single
virtual server
Deploy a single virtual machine
using the vSphere interface.
 vCenter Server and Host
Management
Verify
redundancy of
the solution
components
Restart each storage processor in
turn, and ensure that LUN
connectivity is maintained.
Steps shown below
Disable each of the redundant
switches in turn and verify that the
vSphere host, virtual machine, and
storage array connectivity remains
intact.
Vendor documentation
On a vSphere host that contains at
least one virtual machine, enable
maintenance mode and verify that
the virtual machine can
successfully migrate to an alternate
host.
vCenter Server and Host
Management
 vSphere Networking
 vSphere Virtual Machine
Management
144 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 6: Verifying the Solution
Post-install checklist
The following configuration items are critical to the functionality of the solution.
On each vSphere server, verify the following items prior to deployment into
production:

The vSwitch that hosts the client VLANs is configured with sufficient ports to
accommodate the maximum number of virtual machines that it may host.

All required virtual machine port groups are configured, and each server has
access to the required VMware datastores.

An interface is configured correctly for vMotion using the material in the
vSphere Networking guide.
Deploy and test a single virtual server
Deploy a virtual machine to verify that the solution functions as expected. Verify that
the virtual machine is joined to the applicable domain, has access to the expected
networks, and that it is possible to login to it.
Verify the redundancy of the solution components
To ensure that the various components of the solution maintain availability
requirements, test specific scenarios related to maintenance or hardware failures.
Block
environments
Complete the following steps to restart each VNX storage processor in turn and verify
that connectivity to VMware datastores is maintained throughout each restart:
22. Log in to the Control Station with administrator credentials.
23. Navigate to /nas/sbin.
24. Restart SP A by using the ./navicli -h spa rebootsp command.
25. During the restart cycle, check for presence of datastores on ESXi hosts.
26. When cycle completes, restart SP B by using ./navicli –h spb rebootsp.
27. Enable maintenance mode and verify that you can successfully migrate a
virtual machine to an alternate host.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 145
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 6: Verifying the Solution
File environments
EMC Confidential
Perform a failover of each VNX Data Mover in turn and verify that connectivity to NFS
datastores is maintained. For simplicity, use the following approach for each Data
Mover.
Note:
Optionally, restart the Data Mover through the Unisphere interface.
28. From the Control Station prompt, run the server_cpu <movername> -reboot
command, where <movername> is the name of the Data Mover
29. To verify that network redundancy features function as expected, disable each
of the redundant switching infrastructures in turn. While each of the switching
infrastructures is disabled, verify that all the components of the solution
maintain connectivity to each other and to any existing client infrastructures.
30. Enable maintenance mode and verify that you can successfully migrate a
virtual machine to an alternate host.
146 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 7: System Monitoring
Chapter 7
System Monitoring
This chapter presents the following topics:
Overview ................................................................................................................148
Key areas to monitor ..............................................................................................148
VNX resource monitoring guidelines ......................................................................150
Summary................................................................................................................163
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 147
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 7: System Monitoring
EMC Confidential
Overview
System monitoring of the VSPEX environment is no different from monitoring any core
IT system; it is a relevant and core component of administration. The monitoring
levels involved in a highly virtualized infrastructure, such as a VSPEX environment,
are somewhat more complex than a purely physical infrastructure, as the interaction
and interrelationships between various components can be subtle and nuanced.
However, those experienced in administering virtualized environments should be
familiar with the key concepts and focus areas. The key differentiators are monitoring
at scale and the ability to monitor end-to-end systems and workflows.
Several business needs require proactive, consistent monitoring of the environment:

Stable, predictable performance

Sizing and capacity needs

Availability and accessibility

Elasticity – the dynamic addition, subtraction, and modification of workloads

Data protection
If self-service provisioning is enabled in the environment, the ability to monitor the
system is more critical because clients can generate virtual machines and workloads
dynamically. This can adversely affect the entire system.
This chapter provides the basic knowledge necessary to monitor the key components
of a VSPEX Proven Infrastructure environment. Additional resources are included at
the end of this chapter.
Key areas to monitor
VSPEX Proven Infrastructures provide end-to-end solutions and system monitoring of
three discrete, but highly interrelated areas:

Servers, both virtual machines and clusters

Networking

Storage
This chapter focuses primarily on monitoring key components of the storage
infrastructure, the VNX array, but briefly describes other components.
Performance
baseline
When a workload is added to a VSPEX deployment, server, storage, and networking
resources are consumed. As additional workloads are added, modified, or removed,
resource availability and more importantly, capabilities change, which impact all
other workloads running on the platform. Customers should fully understand their
workload characteristics on all key components prior to deploying them on a VSPEX
platform; this is a requirement to correctly size resource utilization against the
defined reference virtual machine.
148 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 7: System Monitoring
Deploy the first workload, and then measure the end-to-end resource consumption
with a platform performance. This removes the guesswork from sizing activities and
ensures initial assumptions were valid. As additional workloads deploy, rerun
benchmarks to determine cumulative load and impact on existing virtual machines
and their application workloads. Adjust resource allocation accordingly to ensure that
any oversubscription is not negatively impacting overall system performance. Run
these baselines consistently to ensure the platform as a whole, and the virtual
machines themselves, operate as expected. The following components comprise a
core performance baseline.
Servers
The key resources to monitor from a server perspective include:

Processors

Memory

Disk (local, NAS, and SAN)

Networking
Monitor these areas from both a physical host level (the hypervisor host level) and
from a virtual level (from within the guest virtual machine). Depending on your
operating system, there are tools available to monitor and capture this data. For
example, if your VSPEX deployment uses ESXi servers as the hypervisor, you can
use ESXtop to monitor and log these metrics. Windows Server 2012 guests can use
the perfmon utility. Follow your vendor’s guidance to determine performance
thresholds for specific deployment scenarios, which can vary greatly depending on
the application.
Detailed information about these tools is available from:
http://technet.microsoft.com/en-us/library/cc749115.aspx
http://download3.vmware.com/vmworld/2006/adc0199.pdf
Keep in mind that each VSPEX Proven Infrastructure provides a guaranteed level of
performance based on the number of reference virtual machines deployed and their
defined workload.
Networking
Ensure that there is adequate bandwidth for networking communications. This
includes monitoring network loads at the server and virtual machine level, the fabric
(switch) level, and if network file or block protocols such as NFS/CIFS/SMB are
implemented, at the storage level. From the server and virtual machine level, the
monitoring tools mentioned previously provide sufficient metrics to analyze flows into
and out of the servers and guests. Key items to track include aggregate throughput or
bandwidth, latencies, and IOPS size. Capture additional data from network card or
HBA utilities.
From the fabric perspective, tools that monitor switching infrastructure vary by
vendor. Key items to monitor include port utilization, aggregate fabric utilization,
processor utilization, queue depths and inter switch link (ISL) utilization. Networking
storage protocols are discussed in the following section.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 149
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 7: System Monitoring
Storage
EMC Confidential
Monitoring the storage aspect of a VSPEX implementation is crucial to maintaining
the overall health and performance of the system. Fortunately, the tools provided with
the VNX family of storage arrays provide an easy yet powerful manner in which to gain
insight into how the underlying storage components are operating. For both block
and file protocols, there are several key areas to focus on, including:

Capacity

IOPS latency

SP utilization
For CIFS/SMB/NFS protocols, the following additional components should be
monitored:

Data Mover, CPU, and memory usage

File system latency

Network interfaces throughput in, throughput out
Addition considerations (primarily from a tuning perspective) include:

I/O size

Workload characteristics

Cache utilization
These factors are outside the scope of this document; however storage tuning is an
essential component of performance optimization. EMC offers the following
additional guidance on the subject through EMC Online Support:
EMC VNX Unified Best Practices for Performance Applied Best Practices Guide
Using EMC VNX Storage with VMware vSphere
VNX resource monitoring guidelines
Monitor the VNX with the Unisphere GUI, which is accessible by opening an HTTPS
session to the Control Station IP address. The VNX family is a unified storage platform
that provides both block storage and file storage access through a single entity.
Monitoring is divided into two parts:
Monitoring block
storage resources

Monitoring block storage resources

Monitoring file storage resources
This section explains how to use Unisphere to monitor block storage resource usage
that includes capacity, IOPS, and latency.
Capacity
In Unisphere, two panels display capacity information. These panels provide a quick
assessment of the overall free space available within the configured LUNs and
underlying storage pools. For block, sufficient free storage should remain in the
150 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 7: System Monitoring
configured pools to allow for anticipated growth and activities such as snapshot
creation. Configure threshold alerts to warn storage administrators when capacity use
rises above 80 percent. In that case, auto-expansion may need to be adjusted or
additional space allocated to the pool. If LUN utilization is high, reclaim space or
allocate additional space.
To set capacity threshold alerts for a specific pool, complete the following steps:
31. Select that pool and click Properties > Advanced tab.
32. In the Storage Pool Alerts area, choose a number for Percent Full Threshold of
this pool, as shown in Figure 58.
Figure 58. Storage Pool Alerts
To drill-down into capacity for block, complete the following steps:
33. In Unisphere, select the VNX system to examine.
34. Select Storage > Storage Configurations > Storage Pools. This opens the
Storage Pools panel.
35. Examine the columns titled Free Capacity and % Consumed, as shown in
Figure 59.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 151
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 7: System Monitoring
EMC Confidential
Figure 59. Storage pools panel
Monitor capacity at the storage pool level, and at the LUN level.
36. Click Storage and select LUNs. This opens the LUN panel.
37. Select a LUN to examine and click Properties, which displays detailed LUN
information, as shown in Figure 60.
38. Verify the LUN Capacity details in the dialog box. User Capacity is the total
physical capacity available to all thin LUNs in the pool. Consumed Capacity is
the total physical capacity currently assigned to all thin LUNs.
152 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 7: System Monitoring
Figure 60. LUN Properties dialog box
Examine capacity alerts, along with all other system events, by opening the Alerts
panel, and the SP Event Logs panel, both of which are accessed under the Monitoring
and Alerts panel, as shown in Figure 61.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 153
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 7: System Monitoring
EMC Confidential
Figure 61. Monitoring and Alerts panel.
IOPS
The effects of an I/O workload serviced by an improperly configured storage system,
or one whose resources are exhausted, can be felt system-wide. Monitoring the IOPS
that the storage array services includes looking at metrics from the host ports in the
SPs, along with requests serviced by the back-end disks. The VSPEX solutions are
carefully sized to deliver a certain performance level for a particular workload level.
Ensure that IOPS are not exceeding design parameters.
Statistical reporting for IOPS (along with other key metrics) can be examined by
opening the Statistics for Block panel by selecting VNX >System > Monitoring and
Alerts > Statistics for Block. Monitor the statistics online or offline using the Unisphere
Analyzer, which requires a license.
Another metric to examine is Total Bandwidth (MB/s). An 8 Gbps front-end SP port
can process 800 MB per second. The average bandwidth must not exceed 80 percent
of the link bandwidth under normal operating conditions.
IOPS delivered to the LUNs are often more than those delivered by the hosts. This is
particularly true with thin LUNs, as there is additional metadata associated with
managing the I/O streams. Unisphere Analyzer shows the IOPS on each LUN, as
shown in Figure 62.
154 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 7: System Monitoring
Figure 62. IOPS on the LUNs
Certain RAID levels also impart write-penalties that create additional back-end IOPS.
Examine the IOPS delivered to (and serviced from) the underlying physical disks,
which can also be viewed in the Unisphere Analyzer in Figure 63.
The guidelines for drive performance are

180 IOPS for 15k RPM SAS drives

120 IOPS for 10k RPM SAS drives

80 IOPS for NL SAS drives
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 155
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 7: System Monitoring
EMC Confidential
Figure 63. IOPS on the drives
Latency
Latency is the by-product of delays processing I/O requests. This context focuses on
monitoring storage latency, specifically block-level I/O. Using similar procedures from
a previous section, view the latency from the LUN level, as shown in Figure 64.
Figure 64. Latency on the LUNs
156 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 7: System Monitoring
Latency can be introduced anywhere along the I/O stream, from the application layer,
through the transport, and out to the final storage devices. Determining precise
causes of excessive latency requires a methodological approach.
Excessive latency in an FC network is uncommon. Unless there is a defective
component such as an HBA or cable, delays introduced in the network fabric layer are
normally a result of misconfigured switching fabrics. An overburdened storage array
typically causes latency within an FC environment. Focus primarily on the LUNs and
the underlying disk pools ability to service I/O requests. Requests that cannot be
serviced are queued, which introduces latency.
The same paradigm applies to Ethernet-based protocols such as iSCSI and FCoE.
However, additional factors come into play because these storage protocols use
Ethernet as the underlying transport. Isolate the network traffic (either physical or
logical) for storage, and preferably some implementation of Quality of Service (QoS)
in a shared/converged fabric. If network problems are not introducing excessive
latency, examine the storage array. In addition to overburdened disks, excessive SP
utilization can also introduce latency. SP utilization levels greater than 80 percent
indicate a potential problem. Background processes such as replication,
deduplication, and snapshots all compete for SP resources. Monitor these processes
to ensure they do not cause SP resource exhaustion. Possible mitigation techniques
include staggering background jobs, setting replication limits, and adding more
physical resources or rebalancing the I/O workloads. Growth may also mandate
moving to more powerful hardware.
For SP metrics, examine data under the SP tab of the Unisphere Analyzer, as shown in
Figure 65 Review metrics such as Utilization %, Queue Length, and Response Time
(ms). High values for any of these metrics indicate the storage array is under duress
and likely requires mitigation.
EMC best practices recommend a threshold of 70% utilization, response time of 20
ms, and queue length of 10.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 157
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 7: System Monitoring
EMC Confidential
Figure 65. SP Utilization
Monitoring file
storage resources
File-based protocols such as NFS and CIFS/SMB involve additional management
processes beyond those for block storage. Data Movers, hardware components that
provide an interface between NFS and CIFS/SMB users, and the SPs, provide these
management services for VNX Unified systems. Data Movers process file protocol
requests on the client side, and convert the requests to the appropriate SCSI block
semantics on the array side. The additional components and protocols introduce
additional monitoring requirements such as Data Mover network link utilization,
memory utilization, and Data Mover processor utilization.
To examine Data Mover metrics in the Statistics for File panel, select VNX > System >
Monitoring and Alerts > Statistics for File. By clicking the Data Mover link, summary
metrics are displayed, as shown in Figure 66. Usage levels in excess of 80 percent
indicate potential performance concerns and likely require mitigation through Data
Mover reconfiguration, additional physical resources, or both.
158 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 7: System Monitoring
Figure 66. Data Mover statistics
Select Network Device from the Statistics panel to observe front-end network
statistics. The Network Device Statistics window appears, as shown in Figure 67 If
throughput figures exceed 80 percent of the link bandwidth to the client, configure
additional links to relieve the network saturation.
Figure 67. Front-end Data Mover network statistics
Capacity
Similar to block storage monitoring, Unisphere has a statistics panel for file storage.
Select Storage > Storage Configurations > Storage Pools for File to check file storage
space utilization at pool level as shown in Figure 68.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 159
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 7: System Monitoring
EMC Confidential
Figure 68. Storage Pools for File panel
Monitor capacity at the pool and file system level.
39. Click Storage > File Systems. The File Systems window appears, as shown in
Figure 69.
Figure 69. File Systems panel
40. Select a file system to examine and click Properties, which displays detailed
file system information, as shown in Figure 70.
41. Examine the File Storage area for Used and Free Capacity.
160 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 7: System Monitoring
Figure 70. File System property panel
IOPS
In addition to block storage IOPS, Unisphere also provides the ability to monitor file
system IOPS. Select System > Monitoring and Alerts > Statistics for File > File System
I/O, as shown in Figure 71.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 161
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Chapter 7: System Monitoring
EMC Confidential
Figure 71. File system performance panel
Latency
To observe file system latency, select System > Monitoring and Alerts >Statistics for
File > NFS in Unisphere, and examine the value for NFS: Average call time in Figure 72.
Figure 72. File storage all performance panel
162 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Chapter 7: System Monitoring
Summary
Consistent and thorough monitoring of the VSPEX Proven Infrastructure is a best
practice. Having baseline performance data helps to identify problems, while
monitoring key system metrics helps to ensure that the system functions optimally
and within designed parameters. The monitoring process can extend through
integration with automation and orchestration tools from key partners such as
Microsoft and VMware.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 163
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix A: Bill of Materials
164 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
EMC Confidential
Appendix A: Bill of Materials
Appendix A
Bill of Materials
This appendix presents the following topics:
Bill of materials ......................................................................................................166
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 165
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix A: Bill of Materials
EMC Confidential
Bill of materials
Figure 73. List of components used in the VSPEX solution for 200 virtual machines
Component
VMware
vSphere
servers
Solution for 200 virtual machines
CPU
1 vCPU per virtual machine
4 vCPUs per physical core
200 vCPUs
Minimum of 50 physical CPUs
Memory
2 GB RAM per virtual machine
2 GB RAM reservation per VMware vSphere host
Minimum of 400 GB RAM
Network
Block
2 x 10 GbE NICs per server
2 HBA per server
File
4 x 10 GbE NICs per server
Note: To implement VMware vSphere High-Availability (HA) functionality and
to meet the listed minimums, the infrastructure should have at least one
additional server beyond the number needed to meet the minimum
requirements.
Network
infrastructure
Minimum
switching
capacity
Block
2 physical switches
2 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 ports per VMware vSphere server, for storage
network
2 ports per SP, for storage data
File
2 physical switches
4 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 x 10 GbE ports per Data Mover for data
EMC Backup
Avamar
Refer to EMC Backup and Recovery Options for
Data Domain
Refer to EMC Backup and Recovery Options for
VSPEX Private Clouds White Paper
VSPEX Private Clouds White Paper
166 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Appendix A: Bill of Materials
Component
EMC VNX
series
storage array
Solution for 200 virtual machines
Block
 EMC VNX5200
 1 x 1 GbE interface per Control Station for
management
 1 x 1 GbE interface per SP for management
 2 front end ports per SP
 75 x 600 GB 15k rpm 3.5-inch SAS drives
 4 x 200 GB flash drives
 3 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
File
EMC VNX5200
 2 Data Movers (active / standby)
 2 x 10 GbE interfaces per Data Mover
 1 x 1 GbE interface per Control Station for
management
 1 x 1 GbE interface per SP for management
 75 x 600 GB 15k rpm 3.5-inch SAS drives
 4 x 200 GB flash drives.
 3 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare.
Shared
infrastructure
In most cases, a customer environment already
has infrastructure services such as Active
Directory, DNS, and other services configured.
The setup of these services is beyond the scope
of this document.
If implemented without existing infrastructure,
the new minimum requirements are:
 2 physical servers
 16 GB RAM per server
 4 processor cores per server
 2 x 1 GbE ports per server
Note: These services can be migrated into VSPEX post-deployment;
however, they must exist before VSPEX can be deployed.
Note: The solution recommends using a 10 GbE network or an equivalent 1 GbE network
infrastructure as long as the underlying requirements around bandwidth and redundancy are
fulfilled.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 167
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix A: Bill of Materials
Table 30.
EMC Confidential
List of components used in the VSPEX solution for 300 virtual machines
Component
VMware
vSphere
servers
Solution for 300 virtual machines
CPU
1 vCPU per virtual machine
4 vCPUs per physical core
300 vCPUs
Minimum of 75 physical CPUs
Memory
2 GB RAM per virtual machine
2 GB RAM reservation per VMware vSphere host
Minimum of 600 GB RAM
Network
Block
2 x 10 GbE NICs per server
2 HBA per server
File
4 x 10 GbE NICs per server
Note: To implement VMware vSphere High-Availability (HA) functionality and
to meet the listed minimums, the infrastructure should have at least one
additional server beyond the number needed to meet the minimum
requirements.
Network
infrastructure
Minimum
switching
capacity
Block
2 physical switches
2 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 ports per VMware vSphere server, for storage
network
2 ports per SP, for storage data
File
2 physical switches
4 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 x 10 GbE ports per Data Mover for data
EMC Backup
Avamar
Refer to EMC Backup and Recovery Options for
Data Domain
Refer to EMC Backup and Recovery Options for
VSPEX Private Clouds White Paper
VSPEX Private Clouds White Paper
168 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Appendix A: Bill of Materials
Component
EMC VNX
series
storage array
Solution for 300 virtual machines
Block
 EMC VNX5400
 1 x 1 GbE interface per Control Station for
management
 1 x 1 GbE interface per SP for management
 2 front end ports per SP
 110 x 600 GB 15k rpm 3.5-inch SAS drives
 6 x 200 GB flash drives
 4 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
File
EMC VNX5400
 2 Data Movers (active / standby)
 2 x 10 GbE interfaces per Data Mover
 1 x 1 GbE interface per Control Station for
management
 1 x 1 GbE interface per SP for management
 110 x 600 GB 15k rpm 3.5-inch SAS drives
 6 x 200 GB flash drives.
 4 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare.
Shared
infrastructure
In most cases, a customer environment already
has infrastructure services such as Active
Directory, DNS, and other services configured.
The setup of these services is beyond the scope
of this document.
If implemented without existing infrastructure,
the new minimum requirements are:
 2 physical servers
 16 GB RAM per server
 4 processor cores per server
 2 x 1 GbE ports per server
Note: These services can be migrated into VSPEX post-deployment;
however, they must exist before VSPEX can be deployed.
Note: The solution recommends using a 10 GbE network or an equivalent 1 GbE network
infrastructure as long as the underlying requirements around bandwidth and redundancy are
fulfilled.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 169
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix A: Bill of Materials
Table 31.
EMC Confidential
List of components used in the VSPEX solution for 600 virtual machines
Component
VMware
vSphere
servers
Solution for 600 virtual machines
CPU
1 vCPU per virtual machine
4 vCPUs per physical core
600 vCPUs
Minimum of 150 physical CPUs
Memory
2 GB RAM per virtual machine
2 GB RAM reservation per VMware vSphere host
Minimum of 1200 GB RAM
Network
Block
2 x 10 GbE NICs per server
2 HBA per server
File
4 x 10 GbE NICs per server
Note:: To implement VMware vSphere High-Availability (HA) functionality and
to meet the listed minimums, the infrastructure should have at least one
additional server beyond the number needed to meet the minimum
requirements.
Network
infrastructure
Minimum
switching
capacity
Block
2 physical switches
2 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 ports per VMware vSphere server, for storage
network
2 ports per SP, for storage data
File
2 physical switches
4 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 x 10 GbE ports per Data Mover for data
EMC Backup
Avamar
Refer to EMC Backup and Recovery Options for
Data Domain
Refer to EMC Backup and Recovery Options for
VSPEX Private Clouds Design and
Implementation Guide.
VSPEX Private Clouds Design and
Implementation Guide.
170 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Appendix A: Bill of Materials
Component
EMC VNX
series
storage array
Solution for 600 virtual machines
Block
 EMC VNX5600
 1 x 1 GbE interface per control station for
management
 1 x 1 GbE interface per SP for management
 2 front end ports per SP.
 220 x 600 GB 15k rpm 3.5-inch SAS drives
 10x 200 GB flash drives
 8 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drives as a hot spare
File
 EMC VNX5600
 2 Data Movers (active / standby)
 2 x 10 GbE interfaces per Data Mover
 1 x 1 GbE interface per Control Station for
management
 1 x 1 GbE interface per SP for management
 220 x 600 GB 15k rpm 3.5-inch SAS drives
 10 x 200 GB flash drives
 8 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
Shared
infrastructure
In most cases, a customer environment already
has infrastructure services such as Active
Directory, DNS, and other services configured.
The setup of these services is beyond the scope
of this document.
If implemented without existing infrastructure, a
minimum number of additional servers is
required:
2 physical servers
16 GB RAM per server
4 processor cores per server
2 x 1 GbE ports per server
Note: These services can be migrated into VSPEX post-deployment; however,
they must exist before VSPEX can be deployed.
Note:
The solution recommends using a 10 GbE network or an equivalent 1 GbE network
infrastructure as long as the underlying requirements around bandwidth and redundancy are
fulfilled.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 171
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix A: Bill of Materials
Table 32.
EMC Confidential
List of components used in the VSPEX solution for 1000 virtual machines
Component
VMware
vSphere
servers
Solution for 1000 virtual machines
CPU
1 vCPU per virtual machine
4 vCPUs per physical core
1000 vCPUs
Minimum of 250 physical CPUs
Memory
2 GB RAM per virtual machine
2 GB RAM reservation per VMware vSphere host
Minimum of 2000 GB RAM
Network
Block
2 x 10 GbE NICs per server
2 HBA per server
File
4 x 10 GbE NICs per server
Note: To implement VMware vSphere High Availability (HA) functionality and
to meet the listed minimums, the infrastructure should have at least one
additional server beyond the number needed to meet the minimum
requirements.
Network
infrastructure
Minimum
switching
capacity
Block
2 physical switches
2 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 ports per VMware vSphere server, for storage
network
2 ports per SP, for storage data
File
2 physical switches
4 x 10 GbE ports per VMware vSphere server
1 x 1 GbE port per Control Station for
management
2 x 10 GbE ports per Data Mover for data
EMC Backup
Avamar
Refer to EMC Backup and Recovery Options for
Data Domain
Refer to EMC Backup and Recovery Options for
VSPEX Private Clouds Design and
Implementation Guide.
VSPEX Private Clouds Design and
Implementation Guide.
172 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Appendix A: Bill of Materials
Component
EMC VNX
series
storage array
Solution for 1000 virtual machines
Block
 EMC VNX5800
 1 x 1 GbE interface per control station for
management
 1 x 1 GbE interface per SP for management
 2 front end ports per SP
 360 x 600 GB 15k rpm 3.5-inch SAS drives
 16 x 200 GB flash drives
 12 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare.
File
 EMC VNX5800
 3 Data Movers (2 x active /1 x standby)
 2 x 10 GbE interfaces per Data Mover
 1 x 1 GbE interface per Control Station for
management
 1 x 1 GbE interface per SP for management
 360 x 600 GB 15k rpm 3.5-inch SAS drives
 16 x 200 GB flash drives
 12 x 600 GB 15k rpm 3.5-inch SAS drives as
hot spares
 1 x 200 GB flash drive as a hot spare
Note: In VNX5800, it is recommended to run no
more than 600 virtual machines on a single
active Data Mover. Configure two active Data
Movers (2 x active/1 x standby) when scaling to
600 or larger in that case.
Shared
infrastructure
In most cases, a customer environment already has infrastructure services
such as Active Directory, DNS, and other services configured. The setup of
these services is beyond the scope of this document.
If this is being implemented without existing infrastructure, the new
minimum requirements are:
 2 physical servers
 16 GB RAM per server
 4 processor cores per server
 2 x 1 GbE ports per server
Note: These services can be migrated into VSPEX post-deployment; however,
they must exist before VSPEX can be deployed.
Note:
The solution recommends using a 10 GbE network or an equivalent 1 GbE network
infrastructure as long as the underlying requirements around bandwidth and redundancy are
fulfilled.
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 173
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix A: Bill of Materials
174 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
EMC Confidential
Appendix B: Customer Configuration Data Sheet
Appendix B
Customer Configuration Data Sheet
This appendix presents the following topics:
Customer configuration data sheet ........................................................................176
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 175
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix B: Customer Configuration Data Sheet
EMC Confidential
Customer configuration data sheet
Before you start the configuration, gather some customer-specific network, and host
configuration information. The following tables provide information on assembling
the required network and host address, numbering, and naming information. This
worksheet can also be used as a “leave behind” document for future reference.
The VNX File and Unified Worksheets should be cross-referenced to confirm customer
information.
Table 33.
Common server information
Server Name
Purpose
Primary IP
Domain Controller
DNS Primary
DNS Secondary
DHCP
NTP
SMTP
SNMP
vCenter Console
SQL Server
Table 34.
ESXi server information
Server Name
Purpose
Primary
IP
Private net (storage)
addresses
ESXi
Host 1
ESXi
Host 2
…
176 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
VMkernel IP
EMC Confidential
Appendix B: Customer Configuration Data Sheet
Table 35.
Array information
Array name
Admin account
Management IP
Storage pool name
Datastore name
Block
FC WWPN
FCOE WWPN
iSCSI IQN
iSCSI Port IP
File
Table 36.
Name
NFS Server IP
Network infrastructure information
Purpose
IP
Subnet mask
Default
gateway
Ethernet Switch 1
Ethernet Switch 2
…
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 177
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix B: Customer Configuration Data Sheet
Table 37.
Name
EMC Confidential
VLAN information
Network Purpose
VLAN ID
Allowed Subnets
Virtual machine
networking
ESXi Management
iSCSI storage network
(Block)
NFS storage network
(File)
vMotion
Table 38.
Account
Service accounts
Purpose
Password (optional, secure
appropriately)
Windows Server administrator
root
ESXi root
Array administrator
vCenter administrator
SQL Server administrator
178 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Appendix C: Server Resource Component Worksheet
Appendix C
Server Resource Component
Worksheet
This appendix presents the following topics:
Server resources component worksheet ................................................................180
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 179
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix C: Server Resource Component Worksheet
EMC Confidential
Server resources component worksheet
Table 39.
Blank worksheet for server resource totals
Application
Server resources
Storage resources
CPU
IOPS
(Virtual
CPUs)
Memory
(GB)
Resource
requirements
Capacity
(GB)
Reference
virtual
machines
N/A
Equivalent
reference virtual
machines
Resource
requirements
N/A
Equivalent
reference virtual
machines
Resource
requirements
N/A
Equivalent
reference virtual
machines
Resource
requirements
N/A
Equivalent
reference virtual
machines
Total Equivalent reference Virtual Machines
Server customization
Server component totals
NA
Storage customization
Storage component totals
NA
Storage component equivalent reference virtual machines
NA
Total equivalent reference virtual machines - storage
180 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Appendix D: References
Appendix D
References
This appendix presents the following topics:
References .............................................................................................................182
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 181
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix D: References
EMC Confidential
References
EMC
documentation
Other
documentation
The following documents, available on EMC Online Support, provide additional and
relevant information. If you do not have access to a document, contact your EMC
representative.

EMC VSI for VMware vSphere: Storage Viewer — Product Guide

EMC VSI for VMware vSphere: Unified Storage Management— Product Guide

PowerPath VE for VMware vSphere Installation and Administration Guide

VNX FAST Cache: A Detailed Review

EMC VNX Virtual Provisioning Applied Technology

EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 100 Virtual Machines

VNX5400 Unified Installation Guide

VNX5600 Unified Installation Guide

VNX5800 Unified Installation Guide

Using EMC VNX Storage with VMware vSphere
The following documents, located on the VMware website, provide additional and
relevant information:

vSphere Networking

vSphere Storage Guide

vSphere Virtual Machine Administration

vSphere Installation and Setup

vCenter Server and Host Management

vSphere Resource Management

Installing and Administering VMware vSphere Update Manager

vSphere Storage APIs for Array Integration (VAAI) Plug-in

Interpreting esxtop Statistics

Understanding Memory Resource Management in VMware vSphere 5.0
For documentation on Microsoft products, refer to the Microsoft websites:

Microsoft Developer Network

Microsoft TechNet
182 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
EMC Confidential
Appendix E: About VSPEX
Appendix E
About VSPEX
This appendix presents the following topics:
About VSPEX ..........................................................................................................184
EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines 183
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide
Appendix E: About VSPEX
EMC Confidential
About VSPEX
EMC has joined forces with the industry leading providers of IT infrastructure to create
a complete virtualization solution that accelerates deployment of cloud
infrastructure. Built with best-of-breed technologies, VSPEX enables faster
deployment, more simplicity, greater choice, higher efficiency, and lower risk.
Validation by EMC ensures predictable performance and enables customers to select
technology that uses their existing IT infrastructure while eliminating planning, sizing,
and configuration burdens. VSPEX provides a proven infrastructure for customers
looking to gain the simplicity that is characteristic of truly converged infrastructures,
while at the same time gaining more choice in individual solution components.
VSPEX solutions are proven by EMC, and packaged and sold exclusively by EMC
channel partners. VSPEX provides channel partners with more opportunity, faster
sales cycles, and end-to-end enablement. By working even more closely together,
EMC and its channel partners can now deliver infrastructure that accelerates the
journey to the cloud for even more customers.
184 EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 1,000 Virtual Machines
Enabled by Mircosoft Windows Server 2012 R2, EMc VNX Series, and EMc Powered
Backup-Proven Infrastructure Guide