1. technology and computing

version 6.0
Planning Guide
Mirantis OpenStack v5.1
Planning Guide
Contents
Preface
1
Intended Audience
1
Documentation History
1
Introduction to Mirantis OpenStack and Fuel
2
System Requirements
3
Fuel Master Node Hardware Recommendations
3
Target Node Server Hardware Recommendations
4
Sample Hardware Configuration for Target Nodes
4
Controller nodes (3)
4
Storage nodes (3)
5
Compute nodes (5)
5
Planning Summary
6
Choose Network Topology
7
Linux Distribution for Nodes
8
Choosing the Storage Model
9
Glance: Storage for Images
9
Object Storage for Applications
10
Cinder: Block Storage for Volumes
10
Nodes and Roles
12
Plan Monitoring Facilities
14
Ceilometer and MongoDB
14
Zabbix monitoring tool
15
Planning a Sahara Deployment
16
Preparing for vSphere Integration
17
vSphere Installation
17
ESXi Host Networks Configuration
18
Limitations
19
Preparing to run Fuel on vSphere
21
Preparing for VMware NSX integration
22
VMware NSX Installation
22
VMware NSX cluster configuration
22
©2014, Mirantis Inc.
Page i
Mirantis OpenStack v5.1
Planning Guide
Limitations
24
Calculate hardware requirements
26
Example of Hardware Requirements Calculation
27
Calculating CPU
27
Calculating Memory
28
Calculating Storage
28
Throughput
29
Calculating Network
30
Scalability and oversubscription
31
Hardware for this example
31
Mellanox ConnectX-3 network adapters
31
Summary
32
Reference configuration of hardware switches
33
Tagged ports
33
Untagged ports
34
Example 1: HA + Nova-network FlatDHCP manager
35
Detailed Port Configuration
36
Nova-network Switch configuration (Cisco Catalyst 2960G)
38
Nova-network Switch configuration (Juniper EX4200)
41
Example 2: HA + Neutron with GRE
48
Detailed port configuration
49
Neutron Switch configuration (Cisco Catalyst 2960G)
49
Neutron Switch configuration (Juniper EX4200)
53
Example 3: HA + Neutron with VLAN + SR-IOV & iSER
59
Neutron Switch configuration (Mellanox SX1036)
62
Index
©2014, Mirantis Inc.
65
Page ii
Mirantis OpenStack v5.1
Planning Guide
Preface
Preface
This documentation provides information on how to use Mirantis Fuel to deploy OpenStack environment. The
information is for reference purposes and is subject to change.
Intended Audience
This documentation is intended for OpenStack administrators and developers; it assumes that you have
experience with network and cloud concepts.
Documentation History
The following table lists the released revisions of this documentation:
Revision Date
October, 2014
©2014, Mirantis Inc.
Description
6.0 Technical Preview
Page 1
Mirantis OpenStack v5.1
Planning Guide
Introduction to Mirantis OpenStack and Fuel
Introduction to Mirantis OpenStack and Fuel
OpenStack is an extensible, versatile, and flexible cloud management platform. It is a portfolio of cloud
infrastructure services – compute, storage, networking and other core resources — that are exposed through
ReST APIs. It enables a wide range of control over these services, both from the perspective of an Integrated
Infrastructure as a Service (IaaS) controlled by applications and as a set of tools that enable automated
manipulation of the infrastructure itself.
Mirantis OpenStack is a productized snapshot of the open source technologies. It includes Fuel, a graphical web
tool that helps you to quickly deploy your cloud environment. Fuel includes scripts that dramatically facilitate
and speed up the process of cloud deployment, without requiring you to completely familiarize yourself with the
intricate processes required to install the OpenStack environment components.
This guide provides details to get you started with Mirantis OpenStack and Fuel on a set of physical servers
("bare-metal installation") See the User Guide for detailed instructions about how to download and install Fuel on
the Fuel Master Node and then how to use the Fuel interface to deploy your OpenStack environment.
Further reading is available in the following documents:
• Terminology Reference is an alphabetical listing of technologies and concepts that serves as both a glossary
and a master index of information in the Mirantis docs and the open source documentation.
• Operations Guide gives information about advanced tasks required to maintain the OpenStack environment
after it is deployed. Most of these tasks are done in the shell using text editors and command line tools.
• Reference Architecture provides background information about how Mirantis OpenStack and its supporting
HA architecture is implemented.
You have ways to use Mirantis OpenStack and Fuel other than the bare-metal installation:
• You can install Fuel and use it to deploy a Mirantis OpenStack Environment on Oracle VirtualBox. VirtualBox
deployment is useful for demonstrations and is a good way to begin your exploration of the tools and
technologies. See 0 to OpenStack in 60 Minutes or less to get started, with additional information in
Running Fuel on VirtualBox. Note that deployments on top of VirtualBox do not generally meet the
performance and robustness requirements of most production environments.
• You can install Fuel master node on vSphere and deploy a Mirantis OpenStack Environment on either
baremetal or on vSphere.
• Mirantis OpenStack is also available on-demand, preconfigured, and ready to use with our Hosted Private
Cloud product, Mirantis OpenStack Express.
For community members or partners looking to take Fuel even further, see the developer documentation for
information about the internal architecture of Fuel, instructions for building the project, information about
interacting with the REST API and other topics of interest to more advanced developers. You can also visit the
Fuel project for more detailed information and become a contributor.
©2014, Mirantis Inc.
Page 2
Mirantis OpenStack v5.1
Planning Guide
System Requirements
System Requirements
Fuel and Mirantis OpenStack run well on high-quality commodity hardware. The following sections give some
basic information about minimal hardware requirements; choosing the right hardware requires an understanding
of how your cloud environment will be used.
Some general remarks:
• For optimal performance in production environments, configure enough servers and network adapters to
minimize contention for resources in the environment.
• Understand the nature of the software that will run on your environment and configure the hardware
appropriately. For example, an environment that mostly provides remote storage has different hardware
requirements than an environment that is mostly used for extensive calculations.
• One Fuel Master node can manage multiple environments. If you need to support applications with very
different hardware needs, consider deploying separate environments that are targeted to these needs; for
example, deploy one environment that is optimized for data storage and another environment that is
optimized for compute power.
• Select peripheral hardware that is supported by the operating system distribution that you are using for the
target nodes (see Linux Distribution for Nodes) and for the VM instances that will be deployed.
Integrating new hardware drivers into the Mirantis OpenStack distribution is possible but complicated at
this time. This will be made easier through a simplified plug-in architecture that is planned for an upcoming
release.
Note also that some proprietary drivers may work only with specific Linux kernel versions and not be
compatible with the kernel versions that are shipped with Mirantis OpenStack. Other drivers may not be
included in the Mirantis OpenStack distribution because of licensing restrictions.
Fuel Master Node Hardware Recommendations
To install the Fuel Master Node, you should base your hardware on the anticipated load of your server. Logically,
deploying more node servers in your environment requires more CPU, RAM, and disk performance.
Suggested minimum configuration for installation in production environment:
• Quad-core CPU
• 4GB RAM
• 1 gigabit network port
• 128GB SAS Disk
• IPMI access through independent management network
Suggested minimum configuration for installation in lab environment:
• Dual-core CPU
• 2GB RAM
• 1 gigabit network port
• 50GB disk
©2014, Mirantis Inc.
Page 3
Mirantis OpenStack v5.1
Planning Guide
Target Node Server Hardware Recommendations
• Physical console access
Target Node Server Hardware Recommendations
Determining the appropriate hardware for the target nodes requires a good understanding of the software that
will be run on those nodes. The OpenStack community documentation provides basic guidelines:
• Compute and Image System Requirements
• System requirements for Object Storage
The OpenStack Operations Guide includes advice about capacity planning and scaling and should be read in
conjunction with this Planning Guide.
Other information is available elsewhere in this documentation set:
• For information about assigning roles to nodes, see Nodes and Roles.
• For information about storage requirements and other hardware issues for the MongoDB database that is
used with Ceilometer, see Ceilometer and MongoDB.
• For general information about calculating hardware requirements, see Calculate hardware requirements.
To help determine the correct sizing for OpenStack Node servers, use the Mirantis Hardware Bill of Materials
calculator.
For more information on the logic used in the utility and basic directions, see: “How do you calculate how much
hardware you need for your OpenStack cloud?”.
Sample Hardware Configuration for Target Nodes
This is a general-purpose medium-sized hardware configuration that is provides room for moderate growth and is
adequate for a variety of installations. Note that this is not a recommendation or some sort of "best practices"
guide; but provides a reasonable starting point that avoids common configuration mistakes.
The characteristics of the environment are:
• 12 servers (one Fuel Master node, 3 Controller nodes, 3 Storage nodes, 5 Compute nodes)
• High-availability configuration
• Neutron networking, using either the VLAN or GRE topology
• Ceph as the backend for Cinder, Glance, and Nova (for ephemeral storage)
• No Ceilometer/MongoDB monitoring is configured; three additional servers to be used as MongoDB nodes
would be required to add Ceilometer/MongoDB to the environment
Controller nodes (3)
Controllers require sufficient resources to run the Ceph monitors and MySQL in addition to the core OpenStack
components (Nova, Neutron, Cinder, and Glance).
Each controller should have:
• Single or dual-socket CPU with at least 4 physical cores
©2014, Mirantis Inc.
Page 4
Mirantis OpenStack v5.1
Planning Guide
Target Node Server Hardware Recommendations
• 32GB RAM recommended; 24GB minimum (Note that, for testing, a Controller node can run with as little as
2GB of memory, but production systems need the larger memory size.)
• Hardware RAID1 Controller with at least 500GB capacity for the Host operating system disk. Larger disks
may be warranted if you determine that you need large database capacity. Note that it is non-trivial to
expand the disk storage on running Controller nodes.
• 2 NICs, either 1 Gbit/s or 10 Gbit/s
Storage nodes (3)
We recommend separate Ceph nodes for scalability and robustness. The hardware estimate is based on the
requirement of .5 cores per Ceph-OSD CPU and 1GB of RAM per TB of Ceph-OSD space. All Ceph storage and
journal hard disks can be configured in JBOD (Just a Bunch of Disks) mode on the RAID controller, or can be
plugged into free MB ports.
For production installations, the Ceph replication factor should be set to 3 or more; see Storage.
• Single-socket CPU with at least 4 physical cores
• 24GB RAM
• RAID1 Controller with at least 500GB capacity for the Host operating system disk
• 2 NICs, either 1 Gbit/s or 10 Gbit/s
• 18 TB of Ceph storage (6 x 3TB)
• 1-2 SSDs, 64GB or more each, for the Ceph Journal
Compute nodes (5)
Virtual Machines are hosted on the Compute nodes. By default, Fuel configures the Scheduler to use a fairly
aggressive 8:1 CPU overcommit ratio on the Compute nodes; if you do not know what your VM workload is going
to be, reset this ratio to 1:1 to avoid CPU congestion.
Each Compute node should have:
• Dual-socket CPU with at least 4 physical cores per socket
• 64GB RAM
• RAID1 Controller with at least 500GB capacity for the Host operating system disk
• 2 NICs, either 1 Gbit/s or 10 Gbit/s
©2014, Mirantis Inc.
Page 5
Mirantis OpenStack v5.1
Planning Guide
Planning Summary
Planning Summary
Before installation, determine the deployment type that is appropriate for you configuration needs. You may want
to print this list and make notes indicating your selection so you can be sure you have planned your deployment
correctly.
The following table provides a list of configuration steps that you must complete to plan the Mirantis OpenStack
deployment.
Step Description
Additional Information
Select a network topology
See Choose Network Topology
Choose the Linux distro to use on your
nodes
See Linux Distribution for Nodes
Determine how many nodes to deploy
and which roles to assign to each and the
high-availability to implement.
See Nodes and Roles
Plan monitoring facilities
See Plan Monitoring Facilities
If you want to run Hadoop plan to install
Sahara.
See Planning a Sahara Deployment
If you want to run vCenter or
:ref`vsphere-term`
See Preparing for vSphere Integration
If you want to use NSX
See Preparing for VMware NSX integration
Calculate the server and network
hardware needed
See Calculate hardware requirements
Prepare an IP address management plan
and network association.
Identify the network addresses and VLAN IDs for your Public,
floating, Management, Storage, and virtual machine (fixed)
networks. Prepare a logical network diagram.
©2014, Mirantis Inc.
Page 6
Mirantis OpenStack v5.1
Planning Guide
Choose Network Topology
Choose Network Topology
Mirantis OpenStack supports two network modes, which in turn support five topologies. For architectural
descriptions of the five topologies, see:
• Neutron with VLAN segmentation and OVS
• Neutron with GRE segmentation and OVS
• Neutron with VMware NSX
• Nova-network FlatDHCP Manager
• Nova-network VLAN Manager
Nova-network is a simple legacy network manager. It can operate with predefined Private IP spaces only.
• If you do not want to split your VMs into isolated groups (tenants), you can choose the Nova-network with
FlatDHCP topology. In this case, you will have one big network for all tenants and their VMs.
• The Nova-network with VLANManager topology is a form of provider network. This allows you to have
multiple tenants, each of which supports approximately the same number of VMs and an equal amount of
Private IP space. The maximum number of tenants and the private IP space size must be defined before you
deploy your environment. You must also set up appropriate VLANs and gateways on your underlying
network equipment.
Neutron is a modern and more complicated network manager. It separates tenants, decreases the requirements
for the underlying network (physical switches and topology), and gives a great deal of flexibility for manipulating
Private IP spaces. You can create Private IP spaces with different sizes and manipulate them on the fly.
• The Neutron with VLAN topology, like Nova-network with VLANManager, requires a predefined maximum
number of tenants value and underlying network equipment configuration.
• The Neutron with GRE topology allows more flexibility in the maximum number of tenants (it supports up to
65535 tenants) and simplifies the network equipment configuration, but GRE encapsulation decreases the
speed of communication between the VMs and increases the CPU utilization on the Compute and Controller
nodes.
• The Neutron with VMware NSX topology utilizes the VMware NSX network hypervisor as a backend for
Neutron. As with GRE segmentation, it does not restrict the maximum number of available networks.
Before deploying the OpenStack environment that will be using NSX, the user must obtain the NSX OVS
packages for Ubuntu or CentOS and manually upload them to a web server (usually the Fuel master node).
Puppet can then download and install these packages on controllers and compute nodes during the
deployment stage.
Some other considerations when choosing a network topology:
• OVS (Open vSwitch), Bonding, and Murano can only be implemented on Neutron.
• VMWare vCenter and vSphere can only be implemented on Nova-network with the FlatDHCP topology.
• Neutron with VMware NSX can only be implemented with the KVM/QEMU hypervisor.
• Bonding is not supported for SR-IOV over the Mellanox ConnectX-3 network adapters family.
• Mellanox SR-IOV and iSER are supported only when choosing Neutron with VLAN.
©2014, Mirantis Inc.
Page 7
Mirantis OpenStack v5.1
Planning Guide
Linux Distribution for Nodes
Linux Distribution for Nodes
Fuel allows you to deploy either the CentOS or Ubuntu Linux distribution as the Host O/S on the nodes. All nodes
in the environment must run the same Linux distribution. Often, the choice is made based on personal
preference; many administrative tasks on the nodes must be performed at shell level and many people choose
the distribution with which they are most comfortable.
Some specific considerations:
• Each distribution has some hardware support issues. See Release Notes for Mirantis OpenStack 6.0 for details
about known issues.
• In particular, the CentOS version used for OpenStack does not include native support for VLANs while the
Ubuntu version does. In order to use VLANs on CentOS based nodes, you must configure VLAN splinters.
• CentOS supports .rpm packages; Ubuntu supports .deb packages.
©2014, Mirantis Inc.
Page 8
Mirantis OpenStack v5.1
Planning Guide
Choosing the Storage Model
Choosing the Storage Model
This section discusses considerations for choosing the storage model for your OpenStack environment. You need
to consider two types of data:
• Persistent storage exists outside an instance.
• Ephemeral storage is allocated for an instance and is deleted when the instance is deleted.
Fuel deploys storage for two types of persistent data:
• Glance, the image storage service, which can use either Swift or Ceph RBD as the storage backend
• Cinder, the block storage service, which can use either LVM or Ceph RBD as the storage backend
The Nova compute service manages ephemeral storage. By default, Nova stores ephemeral drives as files on local
disks on the Compute nodes but can instead use Ceph RBD as the storage backend for ephemeral storage.
See:
• Calculating Storage for information about choosing the hardware to use for your storage objects
• Storage Decisions is an OpenStack community document that gives guidelines for choosing the storage
model to use.
Glance: Storage for Images
Fuel configures a storage backend for the Glance image service. This provides data resilience for VM images;
multiple Glance instances running on controller nodes can store and retrieve images from the same data set,
while the object store takes care of data protection and HA.
When you create your OpenStack environment, you choose one of the following storage backends for Glance:
• Swift object store, the standard OpenStack object storage component
• Ceph RBD, the distributed block device provided by the Ceph storage platform.
Note
Older versions of Fuel provided the Multi-Node deployment mode that was used to deploy OpenStack
environments that had a single Compute node and used the file system backend for Glance. This mode is
still available in Fuel 5.1 but is deprecated; instead, use the HA deployment mode with a single controller.
This mode creates an environment that is not highly available but is managed using the same services as
those that manage the full HA environment and can be scaled up to have multiple controllers and be
highly available. In this mode, you can choose either Swift or Ceph RBD as the Glance storage backend.
Factors to consider when choosing between Swift and Ceph RBD for the storage backend for the Glance image
server include:
• Ceph provides a single shared pool of storage nodes that can handle all classes of persistent and ephemeral
data that is required for OpenStack instances.
©2014, Mirantis Inc.
Page 9
Mirantis OpenStack v5.1
Planning Guide
Object Storage for Applications
• When using Ceph, a single set of hard drives can serve as a backend for Glance, Cinder, and Nova.
Otherwise, you must have dedicated disks for image storage, block storage, and ephemeral disks.
• Ceph's copy-on-write facility allows you to clone a system image and to start different VMs based on that
image without any unnecessary data copy operations; this speeds up the time required to launch VMs and
create volumes from images. Without the Ceph copy-on-write facility, OpenStack images and volumes must
be copied to separate Glance, Cinder, and Nova locations.
• Ceph supports live migration of VMs with ephemeral drives whereas Cinder with LVM backend only supports
live migration of volume backed VMs. Note that live migration of Ceph backed VMs requires that Ceph RBD
is used for ephemeral volumes as well as Glance and Cinder.
• Swift provides multi-site support. Ceph is unable to replicate RBD block device data on a long-distance link
which means you cannot replicate between multiple sites. Starting with Ceph 0.80 "Firefly" (integrated into
Mirantis OpenStack beginning with Release 5.1), the Ceph Object Gateway (radosgw) supports multi-site
replication of object data but this support is more limited than that of Swift and does not apply to the RBD
backend for Glance.
• Ceph cannot tolerate clock drift greater than 50ms. If your servers drift out of sync, your Ceph cluster
breaks. When using Ceph, it is extremely important that you configure NTP or some other time
synchronization facility for your environment.
Object Storage for Applications
The object storage systems supported by Mirantis OpenStack can also be used by applications that need to store
data in an object store. Swift provides a native REST API as well as the S3 API compatibility layer that emulates
the Amazon S3 API on top of Swift Object Storage.
Ceph includes the optional Ceph Object Gateway component (radosgw) that applications can use to access RGW
objects.
Note that the radosgw implementation of the Swift API does not implement all operations.
Ceph RBD uses RADOS directly and does not use the Swift API, so it is possible to store Glance images in Ceph
and still use Swift as the object store for applications.
Because the Ceph Object Gateway replaces Swift as the provider of the Swift APIs, it is not possible to have both
radosgw and Swift running in the same OpenStack environment.
Cinder: Block Storage for Volumes
Cinder serves block devices for the OpenStack environment. You have two choices for the storage back-end:
• Cinder LVM (default): each volume is stored as a logical volume in an LVM volume group on one of your
Cinder nodes.
• Ceph: each volume is stored as an object in the Ceph RADOS object storage system.
Note
If you are using vCenter as the hypervisor, you must use the VMDK driver to store your volumes in the
vCenter datastore.
©2014, Mirantis Inc.
Page 10
Mirantis OpenStack v5.1
Planning Guide
Object Storage for Applications
Factors to consider when choosing between Cinder LVM and Ceph for the Cinder storage backend include:
• Ceph provides a single shared pool of storage nodes as discussed above for image storage.
• Ceph provides object replication capabilities by storing Cinder volumes as Ceph RBD objects; each Ceph
RBD object is stored as multiple RADOS objects. Ceph ensures that each replica of an object is stored on a
different node. This means that your volumes are protected against hard drive and node failures or even the
failure of the data center itself.
The Ceph data replication rules (CRUSH map) can be customized separately for each object pool to modify
the number of object replicas, add different types of failure domains, etc.
• LVM provides much less protection of your data than Ceph does. Even if you use RAID on each Cinder node,
your data is only protected against a hard drive failure. If the Cinder node itself is lost, all volumes that
were stored on that node are lost.
• Ceph consumes more disk space than LVM. LVM stores a single replica of the data whereas Ceph stores at
least two copies of your data so that your actual raw storage capacity must be two to three times bigger
than your data set. Beginning with the "Firefly" release of Ceph (supported by Mirantis OpenStack beginning
with Release 5.1), you can manually implement erasure coding striping to reduce the data multiplication
requirements of Ceph.
• Ceph provides multi-node striping and redundancy for block storage.
• If you combine Ceph RBD backends for Cinder and Glance, you gain another important advantage over
Cinder LVM: copy-on-write cloning of Glance images into bootable Ceph volumes.
• Ceph supports live migration of VMs with ephemeral drives whereas LVM only supports live migration of
volume backed VMs.
If you use Cinder LVM, you have the following configuation options:
• Let Fuel create a JBOD partition that spans all the storage drives in a node.
• You can join all drives into a RAID array before deployment and have the array appear to Fuel as a single
block device.
When deploying Ceph, Fuel partitions the Ceph storage nodes so that most of the space is reserved for Ceph-OSD
storage; all other partitions for the node consume a portion of the first drive. To improve system performance,
you might configure one or two SSDs and assign them the "Ceph General" role on the Assign a role or roles to each
node server screen. See Ceph Hardware Recommendations for details.
©2014, Mirantis Inc.
Page 11
Mirantis OpenStack v5.1
Planning Guide
Nodes and Roles
Nodes and Roles
Your OpenStack environment contains a set of specialized nodes and roles; see OpenStack Environment
Architecture for a description. When planning your OpenStack deployment, you must determine the proper mix of
node types and what roles will be installed on each. When you create your OpenStack environment, you will
assign a role or roles to each node server.
All production environments should be deployed for high availability although you can deploy your environment
without the replicated servers required for high availability and then add the replicated servers later. But part of
your Nodes and Roles planning is to determine the level of HA you want to implement and to plan for adequate
hardware.
Some general guiding principles:
• When deploying a production-grade OpenStack environment, it is best to spread the roles (and, hence, the
workload) over as many servers as possible in order to have a fully redundant, highly-available OpenStack
environment and to avoid performance bottlenecks.
• For demonstration and study purposes, you can deploy OpenStack on VirtualBox; see Running Fuel on
VirtualBox for more information. This option has the lowest hardware requirements
• OpenStack can be deployed on smaller hardware configurations by combining multiple roles on the nodes
and mapping multiple Logical Networks to a single physical NIC.
• When looking to maximize performance, carefully choose your hardware components and make sure that
the performance features of the selected hardware are supported and enabled. For an example of what tight
integration of hardware and software can do for you, see Mellanox ConnectX-3 network adapters.
This section provides information to help you decide how many nodes you need and which roles to assign to
each.
The absolute minimum requirement for a highly-available OpenStack deployment is to allocate 4 nodes:
• 3 Controller nodes, combined with Storage
• 1 Compute node
In production environments, it is highly recommended to separate storage nodes from controllers. This helps
avoid resource contention, isolates failure domains, and allows to optimize hardware configurations for specific
workloads. To achieve that, you will need a minimum of 5 nodes when using Swift and Cinder storage backends,
or 7 nodes for a fully redundant Ceph storage cluster:
• 3 Controller nodes
• 1 Cinder node or 3 Ceph OSD nodes
• 1 Compute node
Note
You do not need Cinder storage nodes if you are using Ceph RBD as storage backend for Cinder volumes.
©2014, Mirantis Inc.
Page 12
Mirantis OpenStack v5.1
Planning Guide
Nodes and Roles
Note
You do not need Compute nodes if you are using vCenter as the hypervisor.
Of course, you are free to choose how to deploy OpenStack based on the amount of available hardware and on
your goals (such as whether you want a compute-oriented or storage-oriented environment).
For a typical OpenStack compute deployment, you can use this table as high-level guidance to determine the
number of controllers, compute, and storage nodes you should have:
# of Nodes
Controllers
Computes
Storages
4-10
3
1-7
3 (on controllers)
11-40
3
3-32
3+ (swift) + 2 (proxy)
41-100
4
29-88
6+ (swift) + 2 (proxy)
>100
5
>84
9+ (swift) + 2 (proxy)
©2014, Mirantis Inc.
Page 13
Mirantis OpenStack v5.1
Planning Guide
Plan Monitoring Facilities
Plan Monitoring Facilities
Mirantis OpenStack can deploy two different metering and montoring facilities:
• Ceilometer to query and store metering data that can be used for billing purposes
• Zabbix to monitor the health of the OpenStack services running in your environment (experimental feature)
Ceilometer and MongoDB
Fuel can deploy Ceilometer (OpenStack Telemetry), the OpenStack Telemetry component. When enabled,
Ceilometer collects and shares measurement data gathered from all OpenStack components. This data can be
used for monitoring and capacity planning purposes as well as for an alarming service. Ceilometer's REST API can
also provide data to external monitoring software for a customer's billing system.
Mirantis OpenStack 5.0 and later installs MongoDB as the back-end database for OpenStack Telemetry. This
significantly improves the Ceilometer performance issues encountered in earlier releases that used the MySQL
database; MongoDB is better able to handle the volume of concurrent read/write operations.
Ceilometer collects two different categories of data:
• Billable events such as "instance X was created", "volume Z was deleted". These notifications appear in the
cloud ecosystem continuously; monitoring them does not consume significant amounts of resources in the
cloud environment.
• Metrics that analyze the activity in the cloud environment at the moment of sampling. This information is
gathered by polling and can consume significant amounts of I/O processing resources and large amounts of
database storage.
Ceilometer can be configured to collect a large amount of metering data and thus perform a high volume of
database writes. In planning the resources required, consider the following:
• The resources consumed by metrics sampling are determined by the polling interval, the number of metrics
being collected, and the number of resources from which metrics are collected. The amount of storage
required is also affected by the frequency with which you offload or purge the data from the database.
• Frequent polling of metrics yields a better picture of what is happening in the cloud environment but also
significantly increases the amount of data being processed and stored. For example, in one test sampling
the same metrics for the same (fairly small) number of resources in the same environment:
• 1 minute polling accumulated .8TB of data over a year.
• 30 second polling accumulated 1.4TB of data over a year.
• 5 second polling accumulated 14.5TB of data over a year.
• Ceilometer consumes fairly small amounts of CPU but the I/O processing is extremely intensive when the
data is written to the disk. This is why we recommend using dedicated MongoDB nodes rather than running
the MongoDB role on Controller nodes. In our lab tests, nearly 100% of the disk I/O resources on the
Controller nodes were sometimes consumed by Ceilometer writing data to MongoDB when the database
was located on the Controller and a small polling interval was used. This nearly halted all other processing
on the Controller node and prevented other processes from running.
To install Ceilometer:
©2014, Mirantis Inc.
Page 14
Mirantis OpenStack v5.1
Planning Guide
Zabbix monitoring tool
• Check the appropriate box on the Related projects screen when configuring your environment.
• Assign the Telemetry-MongoDB role to an appropriate number of servers on the Assign a role or roles to each
node server screen.
See Running Ceilometer for information about configuring and running Ceilometer.
Ideally, MongoDB should run on dedicated servers, with at least as many MongoDB nodes as Controller nodes in
the environment.
Zabbix monitoring tool
Zabbix is an open source infrastructure monitoring utility that Fuel 5.1 and later can deploy with your OpenStack
environment. See Zabbix implementation for details about how Zabbix is implemented in Mirantis OpenStack.
When planning your Mirantis OpenStack deployment, you must consider the following resource requirements if
you will be deploying Zabbix:
• The Zabbix server must run on its own dedicated node in Mirantis OpenStack 5.1. This server also stores the
Zabbix database.
• A Zabbix agent is installed on each Compute and Storage node in the environment. The agents send all
information to the Zabbix server immediately although some small amount of temporary data may be
written to the local disk for processing.
• Significant network traffic is generated on the Zabbix node as the agents report back to the server; the
agents themselves do not put much load on the network.
• The amount of storage required on the Zabbix node depends on the number of resources being monitored,
the amount of data being gathered for each, and so forth but our internal tests indicate that 30GB of data
storage is adequate for monitoring up to 100 nodes.
• The agents running on the Compute and Storage nodes run periodically; they mainly consume CPU
resources, although they are fairly light-weight processes.
See Performance tuning for information about how to maximize the performance of Zabbix.
To deploy Zabbix in your Mirantis OpenStack environment:
• Enable Experimental tools.
• Assign the Zabbix role to the appropriate node on the Assign a role or roles to each node server screen.
• If you like, reset the password used to access the Zabbix dashboard on the Resetting the Zabbix password
screen.
For information about using the Zabbix Web UI, see Zabbix Web interface.
Limitations:
• The Zabbix server cannot be replicated for high availability in Mirantis OpenStack.
• OVS bonding and Mellanox SR-IOV based networking over the Mellanox ConnectX-3 adapter family are not
supported on the Zabbix node.
©2014, Mirantis Inc.
Page 15
Mirantis OpenStack v5.1
Planning Guide
Planning a Sahara Deployment
Planning a Sahara Deployment
Sahara enables users to easily provision and manage Apache Hadoop clusters in an OpenStack environment.
Sahara works with either Release 1.x or 2.x of Hadoop.
The Sahara control processes run on the Controller node. The entire Hadoop cluster runs in VMs that run on
Compute Nodes. A typical set-up is:
• One VM that serves as the Hadoop master node to run JobTracker (ResourceManager for Hadoop Release
2.x) and NameNode.
• Many VMs that serve as Hadoop worker nodes, each of which runs TaskTracker (NodeManager for Hadoop
Release 2.x) and DataNodes.
You must have exactly one NameNode and one JobTracker running in the environment and you cannot run
Hadoop HA under Sahara. Other than that, you are free to use other configurations. For example, you can run the
TaskTracker/NodeManager and Datanodes in the same VM that runs JobTracker/ResourceManager and
NameNode; such a configuration may not produce performance levels that are acceptable for a production
environment but it works for evaluation and demonstration purposes. You could also run DataNodes and
TaskTrackers in separate VMs.
Sahara can use either Swift Object Storage or Ceph for object storage. Special steps are required to implement
data locality for Swift; see Data-locality for details. Data locality is not available for Ceph.
Plan the size and number of nodes for your environment based on the information in Nodes and Roles.
When deploying an OpenStack Environment that includes Sahara for running Hadoop you need to consider a few
special conditions.
Floating IPs
Fuel configures Sahara to use floating IPs to manage the VMs. This means that you must provide a Floating IP
pool in each Node Group Template you define. See Public and Floating IP address requirements for general
information about floating IPs.
A special case is if you are using Nova-Network and you have set the auto_assign_floating_ip parameter to true
by checking the appropriate box on the Fuel UI. In this case, a floating IP is automatically assigned to each VM
and the "floating ip pool" dropdown menu is hidden in the OpenStack Dashboard.
In either case, Sahara assigns a floating IP to each VM it spawns so be sure to allocate enough floating IPs.
Security Groups
Sahara does not configure OpenStack Security Groups so you must manually configure the default security group
in each tenant where Sahara will be used. See Ports Used by Sahara for a list of ports that need to be opened.
VM Flavor Requirements
Hadoop requires at least 1G of memory to run. That means you must use flavors that have at least 1G of memory
for Hadoop cluster nodes.
Communication between virtual machines
Be sure that communication between virtual machines is not blocked.
For additional information about using Sahara to run Apache Hadoop, see the Sahara documentation.
©2014, Mirantis Inc.
Page 16
Mirantis OpenStack v5.1
Planning Guide
Preparing for vSphere Integration
Preparing for vSphere Integration
Fuel 5.0 and later can deploy a Mirantis OpenStack environment that boots and manages virtual machines in
VMware vSphere. VMware provides a vCenter driver for OpenStack that enables the Nova-compute service to
communicate with a VMware vCenter server that manages one or more ESX host clusters. If your vCenter
manages multiple ESX host clusters, Fuel 5.1 allows you to specify several or all clusters for a single OpenStack
environment, so that one Nova-compute service manages multiple ESX host clusters via single vCenter server.
Note
In 5.x environments that use vCenter as the hypervisor, the Nova-compute service runs only on Controller
nodes.
In future Fuel releases, the plan is to change the relation between a Nova-compute service and an ESX
host cluster from one-to-many to one-to-one. In other words, to manage multiple ESX host clusters, you
will need to run multiple Nova-compute services.
The vCenter driver makes management convenient from both the OpenStack Dashboard (Horizon) and from
vCenter, where advanced vSphere features can be accessed.
This section summarizes the planning you should do and other steps that are required before you attempt to
deploy Mirantis OpenStack with vCenter integration.
For more information:
• See VMware vSphere Integration for information about how vCenter support is implemented in Mirantis
OpenStack;
• vSphere deployment notes gives instructions for creating and deploying a Mirantis OpenStack environment
that is integrated with VMware vSphere.
• For background information about VMware vSphere support in OpenStack, see the VMware vSphere OpenStack Manuals.
• The official vSphere installation guide can be found here: vSphere Installation and Setup.
vSphere Installation
Before installing Fuel and using it to create a Mirantis OpenStack environment that is integrated with VMware
vSphere, the vSphere installation must be up and running. Please check that you completed the following steps: *
Install vSphere * Install vCenter * Install ESXi * Configure vCenter
• Create DataCenter
• Create vCenter cluster
• Add ESXi host(s)
©2014, Mirantis Inc.
Page 17
Mirantis OpenStack v5.1
Planning Guide
ESXi Host Networks Configuration
ESXi Host Networks Configuration
The ESXi host(s) network must be configured appropriately in order to enable integration of Mirantis OpenStack
with vCenter. Follow the steps below:
1. Open the ESXi host page, select the "Manage" tab and click on "Networking". vSwitch0 and all its networks
are shown. Click the Add Network button:
2. In the "Add Networking" wizard, select the Virtual Machine Port group:
©2014, Mirantis Inc.
Page 18
Mirantis OpenStack v5.1
Planning Guide
Limitations
3. On the next page, select the "Virtual Machine Port Group" option to ensure that the network will be created
in vSwitch0:
4. Always name the network br100; this is the only value that works with Fuel; type a VLAN Tag in the VLAN
ID field; (the value must be equal to the VLAN Tag at VM Fixed on Fuel’s Network settings tab):
Limitations
• Only vCenter versions 5.1 and later are supported
• It is not possible to specify the vCenter cluster where virtual instances will be launched.
©2014, Mirantis Inc.
Page 19
Mirantis OpenStack v5.1
Planning Guide
Limitations
• Only Nova Network with FlatDHCP mode is supported in the current version of the integration.
• Each OpenStack environment can support one vCenter cluster.
• Security groups are not supported.
• The only supported backend for Cinder is VMDK.
• Volumes that are created by Cinder appear as SCSI disks. To be able to read/write that disk, be sure that the
operating system inside the instance supports SCSI disks. The CirrOS image that is shipped with Fuel
supports only IDE disks, so even if the volume is attached to it, CirrOS is not able to use it.
• The Ceph backend for Glance, Cinder and RadosGW object storage is not supported.
• The VMware vCenter-managed datastore is not supported as a backend for Glance.
• Murano is not supported. It requires Neutron and vCenter utilizes nova-network.
• Fuel does not configure Ceilometer to collect metrics from vCenter virtual resources. For more details about
the Ceilometer plugin for vCenter, see Support for VMware vCenter Server
For background information about how vCenter support is integrated into Mirantis OpenStack, see VMware
vSphere Integration.
Follow the instructions in vSphere deployment notes to deploy your Mirantis OpenStack environment with vCenter
support.
©2014, Mirantis Inc.
Page 20
Mirantis OpenStack v5.1
Planning Guide
Preparing to run Fuel on vSphere
Preparing to run Fuel on vSphere
For information about how Fuel runs on vSphere, see Fuel running under vSphere.
For instructions for setting up your vSphere and installing Fuel on vSphere, follow the instructions in Installing
Fuel Master Node on vSphere.
©2014, Mirantis Inc.
Page 21
Mirantis OpenStack v5.1
Planning Guide
Preparing for VMware NSX integration
Preparing for VMware NSX integration
Fuel 5.1 and later can deploy a Mirantis OpenStack environment that can manage virtual networks in VMware
NSX. VMware provides an NSX plug-in for OpenStack that enables the Neutron service to communicate and
provision virtual networks in NSX that can manage Open vSwitches on controller and compute nodes.
This section summarizes the planning you should do and other steps that are required before you attempt to
deploy Mirantis OpenStack with NSX integration.
For more information:
• See Neutron with VMware NSX for information about how NSX support is implemented in Mirantis
OpenStack;
• NSX deployment notes gives instructions for creating and deploying a Mirantis OpenStack environment that
is integrated with an NSX networking backend that utilizes the NSX Neutron plug-in.
• The official VMware NSX installation guide can be found here: NSX Installation and Upgrade Guide.
VMware NSX Installation
Before installing Fuel and using it to create a Mirantis OpenStack environment that is integrated with VMware
NSX, the VMware NSX installation must be up and running. Please check that you completed the following steps:
• Install NSX Controller node
• Install NSX Gateway node
• Install NSX Manager node
• Install NSX Service node
Note
According to VMware documentation, an NSX cluster can operate successfully without an NSX Service
node, but its presence is mandatory for deploying Mirantis OpenStack. Support of NSX clusters without a
Service node might appear in future versions of Fuel.
VMware NSX cluster configuration
• Configure NSX Controller
• Assign IP address to NSX controller. If the controller is going to be placed in any of the OpenStack
logical networks (Public, Management, Storage), you must assign an IP address that does not
overlap with IP addresses that are managed by OpenStack. For example if the Public network has
range 172.16.0.0/24 and addresses 172.16.0.1 - 172.16.0.126 are managed, any IP address in the
range 172.16.0.127 - 172.16.0.254 can be used for the NSX controller. If the controller IP belongs
to a separate network, there must be L3 connectivity between the Public network and the
network where the VMware NSX controller resides.
©2014, Mirantis Inc.
Page 22
Mirantis OpenStack v5.1
Planning Guide
Preparing for VMware NSX integration
• Configure NSX Gateway node
• Configure NSX Service node
• Create NSX cluster in NSX Manager
• Create new cluster
• Create new Transport Zone. You need to write down the Transport Zone UUID; you will use this
value when configuring parameters on the Settings tab in the Fuel web UI.
• Add Gateway node to the NSX cluster
• When you add the Gateway node, you must select the Transport Type the Gateway node will be
using.
• You need to write down the Transport Type you chose. Later, you will provide this value on the
Settings tab in the Fuel web UI.
• Add the L3 Gateway Service to NSX cluster. You need to write down the Gateway Service UUID;
later you need to provide this value on the Settings tab in the Fuel web UI.
Attention!
You must specify the same transport type on the Settings tab in FUEL web UI.
©2014, Mirantis Inc.
Page 23
Mirantis OpenStack v5.1
Planning Guide
Limitations
• Obtain and put NSX specific packages on the Fuel Master node
• Upload NSX package archives to the Fuel Master node which has IP address 10.20.0.2 in this
example:
$ scp nsx-ovs-2.0.0-build30176-rhel61_x86_64.tar.gz [email protected]:
$ scp nsx-ovs-2.0.0-build30176-ubuntu_precise_amd64.tar.gz [email protected]:
• Go to the Fuel Master node and put the NSX packages in the /var/www/nailgun/ directory:
[root@fuel
[root@fuel
[root@fuel
[root@fuel
~]# mkdir /var/www/nailgun/nsx
~]# cd /var/www/nailgun/nsx
nsx]# tar -xf ~/nsx-ovs-2.0.0-build30176-rhel61_x86_64.tar.gz
nsx]# tar -xf ~/nsx-ovs-2.0.0-build30176-ubuntu_precise_amd64.tar.gz
• Check out that the files are listed by web server. Open the URL http://10.20.0.2:8080/nsx/ in a
web browser and check that the web server successfully lists the packages.
• Now you can provide the URL http://10.20.0.2:8080/nsx/ for the "URL for NSX bits" setting on the
Settings tab in the Fuel web UI.
Seealso
You can read blog posts NSX appliances installation and NSX cluster configuration for details about the
NSX cluster deployment process.
Limitations
• Only KVM or QEMU are supported as hypervisor options when using VMware NSX.
• Only VMware NSX 4.0 is supported
• Resetting or deleting the environment via "Reset" and "Delete" buttons on the Actions tab does not flush
the entities (logical switches, routers, load balancers, etc) that were created in the NSX cluster. Eventually,
the cluster may run out of resources; it is up to the cloud operator to remove unneeded entities from the
VMware NSX cluster. Each time the deployment fails or is interrupted; after solving the problem, restart the
deployment process.
To cleanup the NSX cluster, log into the NSX Manager, open the dashboard and click on numbered link in
"Hypervisor Software Version Summary":
©2014, Mirantis Inc.
Page 24
Mirantis OpenStack v5.1
Planning Guide
Limitations
Tick all registered nodes and press "Delete Checked" button:
Then click on "Logical Layer" in the "category" column, tick all remaining logical entities and remove them
by pressing the corresponding "Delete Checked" button:
©2014, Mirantis Inc.
Page 25
Mirantis OpenStack v5.1
Planning Guide
Calculate hardware requirements
Calculate hardware requirements
You can use the Fuel Hardware Calculator to calculate the hardware required for your OpenStack environment.
When choosing the hardware on which you will deploy your OpenStack environment, you should think about:
• CPU -- Consider the number of virtual machines that you plan to deploy in your cloud environment and the
CPU per virtual machine. Also consider how the environment will be used: environments used for heavy
computational work may require more powerful CPUs than environments used primarily for storage, for
example.
• Memory -- Depends on the amount of RAM assigned per virtual machine and the controller node.
• Storage -- Depends on the local drive space per virtual machine, remote volumes that can be attached to a
virtual machine, and object storage.
• Networking -- Depends on the Choose Network Topology, the network bandwidth per virtual machine, and
network storage.
See Example of Hardware Requirements Calculation for some specific calculations you can make when choosing
your hardware.
©2014, Mirantis Inc.
Page 26
Mirantis OpenStack v5.1
Planning Guide
Example of Hardware Requirements Calculation
Example of Hardware Requirements Calculation
When you calculate resources for your OpenStack environment, consider the resources required for expanding
your environment.
The example described in this section presumes that your environment has the following prerequisites:
• 100 virtual machines
• 2 x Amazon EC2 compute units 2 GHz average
• 16 x Amazon EC2 compute units 16 GHz maximum
Calculating CPU
Use the following formula to calculate the number of CPU cores per virtual machine:
max GHz /(number of GHz per core x 1.3 for hyper-threading)
Example:
16 GHz / (2.4 x 1.3) = 5.12
Therefore, you must assign at least 5 CPU cores per virtual machine.
Use the following formula to calculate the total number of CPU cores:
(number of VMs x number of GHz per VM) / number of GHz per core
Example:
(100 VMs * 2 GHz per VM) / 2.4 GHz per core = 84
Therefore, the total number of CPU cores for 100 virtual machines is 84.
Depending on the selected CPU you can calculate the required number of sockets. Use the following formula:
total number of CPU cores / number of cores per socket
For example, you use Intel E5 2650-70 8 core CPU:
84 / 8 = 11
Therefore, you need 11 sockets. To calculate the number of servers required for your deployment, use the
following formula:
total number of sockets / number of sockets per server
©2014, Mirantis Inc.
Page 27
Mirantis OpenStack v5.1
Planning Guide
Calculating Memory
Round the number of sockets to an even number to get 12 sockets. Use the following formula:
12 / 2 = 6
Therefore, you need 6 dual socket servers. You can calculate the number of virtual machines per server using the
following formula:
number of virtual machines / number of servers
Example:
100 / 6 = 16.6
Therefore, you can deploy 17 virtual machines per server.
Using this calculation, you can add additional servers accounting for 17 virtual machines per server.
The calculation presumes the following conditions:
• No CPU oversubscribing
• If you use hyper-threading, count each core as 1.3, not 2.
• CPU supports the technologies required for your deployment
Calculating Memory
Continuing to use the example from the previous section, we need to determine how much RAM will be required
to support 17 VMs per server. Let's assume that you need an average of 4 GBs of RAM per VM with dynamic
allocation for up to 12GBs for each VM. Calculating that all VMs will be using 12 GBs of RAM requires that each
server have 204 GBs of available RAM.
You must also consider that the node itself needs sufficient RAM to accommodate core OS operations as well as
RAM for each VM container (not the RAM allocated to each VM, but the memory the core OS uses to run the VM).
The node's OS must run it's own operations, schedule processes, allocate dynamic resources, and handle network
operations, so giving the node itself at least 16 GBs or more RAM is not unreasonable.
Considering that the RAM we would consider for servers comes in 4 GB, 8 GB, 16 GB and 32 GB sticks, we would
need a total of 256 GBs of RAM installed per server. For an average 2-CPU socket server board you get 16-24
RAM slots. To have 256 GBs installed you would need sixteen 16 GB sticks of RAM to satisfy your RAM needs for
up to 17 VMs requiring dynamic allocation up to 12 GBs and to support all core OS requirements.
You can adjust this calculation based on your needs.
Calculating Storage
When planning disk space, you need to consider several types of data:
• Ephemeral (the local drive space for a VM)
• Persistent object storage
• Persistent block storage
©2014, Mirantis Inc.
Page 28
Mirantis OpenStack v5.1
Planning Guide
Calculating Memory
See Choosing the Storage Model for more information about the options for storage and how to choose the
appropriate model.
As far as local drive space that must reside on the compute nodes, in our example of 100 VMs we make the
following assumptions:
• 150 GB local storage per VM
• 5 TB total of local storage (100 VMs * 50 GB per VM)
• 500 GB of persistent volume storage per VM
• 50 TB total persistent storage
Returning to our already established example, we need to figure out how much storage to install per server. This
storage will service the 17 VMs per server. If we are assuming 50 GBs of storage for each VMs drive container,
then we would need to install 2.5 TBs of storage on the server. Since most servers have anywhere from 4 to 32
2.5" drive slots or 2 to 12 3.5" drive slots, depending on server form factor (i.e., 2U vs. 4U), you will need to
consider how the storage will be impacted by the intended use.
If storage impact is not expected to be significant, then you may consider using unified storage. For this example
a single 3 TB drive would provide more than enough storage for seventeen 150 GB VMs. If speed is really not an
issue, you might even consider installing two or three 3 TB drives and configure a RAID-1 or RAID-5 for
redundancy. If speed is critical, however, you will likely want to have a single hardware drive for each VM. In this
case you would likely look at a 3U form factor with 24-slots.
Don't forget that you will also need drive space for the node itself, and don't forget to order the correct
backplane that supports the drive configuration that meets your needs. Using our example specifications and
assuming that speed is critical, a single server would need 18 drives, most likely 2.5" 15,000 RPM 146 GB SAS
drives.
Throughput
As far as throughput, that's going to depend on what kind of storage you choose. In general, you calculate IOPS
based on the packing density (drive IOPS * drives in the server / VMs per server), but the actual drive IOPS will
depend on the drive technology you choose. For example:
• 3.5" slow and cheap (100 IOPS per drive, with 2 mirrored drives)
• 100 IOPS * 2 drives / 17 VMs per server = 12 Read IOPS, 6 Write IOPS
• 2.5" 15K (200 IOPS, four 600 GB drive, RAID-10)
• 200 IOPS * 4 drives / 17 VMs per server = 48 Read IOPS, 24 Write IOPS
• SSD (40K IOPS, eight 300 GB drive, RAID-10)
• 40K * 8 drives / 17 VMs per server = 19K Read IOPS, 9.5K Write IOPS
Clearly, SSD gives you the best performance, but the difference in cost between SSDs and the less costly
platter-based solutions is going to be significant, to say the least. The acceptable cost burden is determined by
the balance between your budget and your performance and redundancy needs. It is also important to note that
the rules for redundancy in a cloud environment are different than a traditional server installation in that entire
servers provide redundancy as opposed to making a single server instance redundant.
In other words, the weight for redundant components shifts from individual OS installation to server redundancy.
It is far more critical to have redundant power supplies and hot-swappable CPUs and RAM than to have
©2014, Mirantis Inc.
Page 29
Mirantis OpenStack v5.1
Planning Guide
Calculating Network
redundant compute node storage. If, for example, you have 18 drives installed on a server and have 17 drives
directly allocated to each VM installed and one fails, you simply replace the drive and push a new node copy. The
remaining VMs carry whatever additional load is present due to the temporary loss of one node.
Remote storage
IOPS will also be a factor in determining how you plan to handle persistent storage. For example, consider these
options for laying out your 50 TB of remote volume space:
• 12 drive storage frame using 3 TB 3.5" drives mirrored
• 36 TB raw, or 18 TB usable space per 2U frame
• 3 frames (50 TB / 18 TB per server)
• 12 slots x 100 IOPS per drive = 1200 Read IOPS, 600 Write IOPS per frame
• 3 frames x 1200 IOPS per frame / 100 VMs = 36 Read IOPS, 18 Write IOPS per VM
• 24 drive storage frame using 1TB 7200 RPM 2.5" drives
• 24 TB raw, or 12 TB usable space per 2U frame
• 5 frames (50 TB / 12 TB per server)
• 24 slots x 100 IOPS per drive = 2400 Read IOPS, 1200 Write IOPS per frame
• 5 frames x 2400 IOPS per frame / 100 VMs = 120 Read IOPS, 60 Write IOPS per frame
You can accomplish the same thing with a single 36 drive frame using 3 TB drives, but this becomes a single
point of failure in your environment.
Object storage
When it comes to object storage, you will find that you need more space than you think. For example, this
example specifies 50 TB of object storage.
Object storage uses a default of 3 times the required space for replication, which means you will need 150 TB.
However, to accommodate two hands-off zones, you will need 5 times the required space, which actually means
250 TB. The calculations don't end there. You don't ever want to run out of space, so "full" should really be more
like 75% of capacity, which means you will need a total of 333 TB, or a multiplication factor of 6.66.
Of course, that might be a bit much to start with; you might want to start with a happy medium of a multiplier of
4, then acquire more hardware as your drives begin to fill up. That calculates to 200 TB in our example. So how
do you put that together? If you were to use 3 TB 3.5" drives, you could use a 12 drive storage frame, with 6
servers hosting 36 TB each (for a total of 216 TB). You could also use a 36 drive storage frame, with just 2 servers
hosting 108 TB each, but its not recommended due to the high cost of failure to replication and capacity issues.
Calculating Network
Perhaps the most complex part of designing an OpenStack environment is the networking.
An OpenStack environment can involve multiple networks even beyond the Public, Private, and Internal
networks. Your environment may involve tenant networks, storage networks, multiple tenant private networks,
and so on. Many of these will be VLANs, and all of them will need to be planned out in advance to avoid
configuration issues.
©2014, Mirantis Inc.
Page 30
Mirantis OpenStack v5.1
Planning Guide
Mellanox ConnectX-3 network adapters
In terms of the example network, consider these assumptions:
• 100 Mbits/second per VM
• HA architecture
• Network Storage is not latency sensitive
In order to achieve this, you can use two 1 Gb links per server (2 x 1000 Mbits/second / 17 VMs = 118
Mbits/second).
Using two links also helps with HA. You can also increase throughput and decrease latency by using two 10 Gb
links, bringing the bandwidth per VM to 1 Gb/second, but if you're going to do that, you've got one more factor to
consider.
Scalability and oversubscription
It is one of the ironies of networking that 1 Gb Ethernet generally scales better than 10Gb Ethernet -- at least
until 100 Gb switches are more commonly available. It's possible to aggregate the 1 Gb links in a 48 port switch,
so that you have 48 x 1 Gb links down, but 4 x 10 Gb links up. Do the same thing with a 10 Gb switch, however,
and you have 48 x 10 Gb links down and 4 x 100b links up, resulting in oversubscription.
Like many other issues in OpenStack, you can avoid this problem to a great extent with careful planning.
Problems only arise when you are moving between racks, so plan to create "pods", each of which includes both
storage and compute nodes. Generally, a pod is the size of a non-oversubscribed L2 domain.
Hardware for this example
In this example, you are looking at:
• 2 data switches (for HA), each with a minimum of 12 ports for data (2 x 1 Gb links per server x 6 servers)
• 1 x 1 Gb switch for IPMI (1 port per server x 6 servers)
• Optional Cluster Management switch, plus a second for HA
Because your network will in all likelihood grow, it's best to choose 48 port switches. Also, as your network
grows, you will need to consider uplinks and aggregation switches.
Mellanox ConnectX-3 network adapters
Beginning with version 5.1, Fuel can configure Mellanox ConnectX-3 network adapters to accelerate the
performance of compute and storage traffic.
This implements the following performance enhancements:
• Compute nodes use SR-IOV based networking.
• Cinder nodes use iSER block storage as the iSCSI transport rather than the default iSCSI over TCP.
These features reduce CPU overhead, boost throughput, reduce latency, and enable network traffic to bypass the
software switch layer (e.g. Open vSwitch).
The Mellanox ConnectX-3 adapters family supports up to 40/56GbE. To reach 56 GbE speed in your network with
ConnectX-3 adapters, you must use Mellanox Ethernet switches (e.g. SX1036) with the additional 56GbE license.
The switch ports should be configured specifically to use 56GbE speed. No additional configuration is required on
©2014, Mirantis Inc.
Page 31
Mirantis OpenStack v5.1
Planning Guide
Summary
the adapter side. For additional information about how to run in 56GbE speed, see HowTo Configure 56GbE Link
on Mellanox Adapters and Switches.
Summary
In general, your best bet is to choose a 2 socket server with a balance in I/O, CPU, Memory, and Disk that meets
your project requirements. Look for a 1U R-class or 2U high density C-class servers. Some good options from Dell
for compute nodes include:
• Dell PowerEdge R620
• Dell PowerEdge C6220 Rack Server
• Dell PowerEdge R720XD (for high disk or IOPS requirements)
You may also want to consider systems from HP (http://www.hp.com/servers) or from a smaller systems builder
like Aberdeen, a manufacturer that specializes in powerful, low-cost systems and storage servers
(http://www.aberdeeninc.com).
©2014, Mirantis Inc.
Page 32
Mirantis OpenStack v5.1
Planning Guide
Reference configuration of hardware switches
Reference configuration of hardware switches
This section describes reference configuration for Cisco, Arista, Mellanox and Juniper network switches.
Tagged ports
Cisco Catalyst
interface [Ten]GigabitEthernet[interface number]
description [port description]
switchport trunk encapsulation dot1q
switchport trunk allowed vlan [vlan IDs for specific networks]
switchport mode trunk
spanning-tree portfast trunk
switchport trunk native vlan [vlan ID] - if necessary one untagged VLAN
Cisco Nexus/ Arista
interface ethernet[interface number]
description [port description]
switchport
switchport mode trunk
switchport trunk allowed vlan [vlan IDs for specific networks]
switchport trunk native vlan [vlan ID] - if necessary one untagged VLAN
Mellanox
interface ethernet [interface number]
description [port description]
switchport mode hybrid
switchport hybrid allowed-vlan [add|remove|except] [vlan ID|vlan IDs range]
switchport hybrid allowed-vlan [vlan ID|vlan IDs range]
switchport access vlan [vlan ID]
Juniper
interfaces {
[interface_name]-[interface_number] {
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members [ vlan IDs or names of specific
networks];
}
native-vlan-id [vlan ID] if necessary one untagged VLAN
©2014, Mirantis Inc.
Page 33
Mirantis OpenStack v5.1
Planning Guide
Untagged ports
}
}
}
}
Untagged ports
Cisco Catalyst
interface [Ten]GigabitEthernet[interface number]
description [port description]
switchport access [vlan ID for specific network]
switchport mode access
spanning-tree portfast
Cisco Nexus/Arista
interface ethernet[interface number]
description [port description]
switchport
switchport access vlan [vlan ID for specific network]
Mellanox
interface ethernet [interface number]
description [port description]
switchport mode access
switchport access vlan [vlan ID]
Juniper
interfaces {
[interface_name]-[interface_number] {
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members [vlan ID or name for specific network];
}
}
}
}
}
©2014, Mirantis Inc.
Page 34
Mirantis OpenStack v5.1
Planning Guide
Example 1: HA + Nova-network FlatDHCP manager
Example 1: HA + Nova-network FlatDHCP manager
As a model example, the following configuration is used:
• Deployment mode: Multi-node HA
• Networking model: Nova-network FlatDHCP manager
Hardware and environment:
• 7 servers with two 1Gb/s Ethernet NIC and IPMI
• 1 Cisco Catalyst 2960G switch
• Independent out of band management network for IPMI
• Connection to the Internet or/and DC network via a router called Gateway and IP 172.16.1.1
Node Server roles:
• 1 server as Fuel Node
• 3 servers as Controller Node
• 1 server as Cinder Node
• 2 servers as Compute Node
Network configuration plan:
• Public network 172.16.1.0/24
• Floating network 172.16.0.0/24 in VLAN 100
• Management network 192.168.0.0/24 in VLAN 101
• Storage network 192.168.1.0/24 in VLAN 102
• Private (Fixed) network 10.0.0.0/24 in VLAN 103
• Administrative network (for Fuel) 10.20.0.0/24 in VLAN 104
Network Parameters:
• Fuel server IP: 10.20.0.2/24
• Default gateway: 10.20.0.1
• DNS 10.20.0.1
Note
The Internet and rest of DC access is available through the Public network (for OpenStack Nodes) and
Administrative network (Fuel server)
From the server node side, ports with the following VLAN IDs for networks are used:
• eth0 - Management VLAN 101 (tagged), Storage VLAN 102(tagged) and Administrative VLAN 104 (untagged)
©2014, Mirantis Inc.
Page 35
Mirantis OpenStack v5.1
Planning Guide
Detailed Port Configuration
• eth1 - Public/Floating VLAN 100 (tagged), Private VLAN 103 (tagged)
Detailed Port Configuration
The following table describes the detailed port configuration and VLAN assignment.
Switch
Port
Server name
Server
NIC
tagged /
untagged
VLAN ID
G0/1
Fuel
eth0
untagged
104
G0/2
Fuel
eth1
untagged
100
G0/3
Compute Node 1
eth0
tagged
101, 102, 104 (untagged)
G0/4
Compute Node 1
eth1
tagged
100, 103
G0/5
Compute Node n
eth0
tagged
101, 102, 104 (untagged)
G0/6
Compute Node n
eth1
tagged
100, 103
G0/7
Controller Node 1
eth0
tagged
101, 102, 104(untagged)
G0/8
Controller Node 1
eth1
tagged
100, 103
G0/9
Controller Node 2
eth0
tagged
101, 102, 104 (untagged)
G0/10
Controller Node 2
eth1
tagged
100, 103
G0/11
Controller Node 3
eth0
tagged
101, 102, 104 (untagged)
G0/12
Controller Node 3
eth1
tagged
100, 103
G0/13
Cinder Node
eth0
tagged
101, 102, 104 (untagged)
G0/14
Cinder Node
eth1
tagged
100, 103
G0/24
Router (default gateway)
---
untagged
100
Connect the servers to the switch as in the diagram below:
©2014, Mirantis Inc.
Page 36
Mirantis OpenStack v5.1
Planning Guide
Detailed Port Configuration
The following diagram describes the network topology for this environment.
©2014, Mirantis Inc.
Page 37
Mirantis OpenStack v5.1
Planning Guide
Nova-network Switch configuration (Cisco Catalyst
2960G)
Nova-network Switch configuration (Cisco Catalyst 2960G)
Use the following configuration to deploy Mirantis OpenStack using a Cisco Catalyst 2960G network switch.:
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname OpenStack\_sw1
!
logging count
logging buffered 64000 informational
logging rate-limit console 100 except errors
logging console informational
enable secret r00tme
!
username root privilege 15 secret r00tme
!
no aaa new-model
aaa session-id common
ip subnet-zero
ip domain-name domain.ltd
ip name-server [ip of domain name server]
!
©2014, Mirantis Inc.
Page 38
Mirantis OpenStack v5.1
Planning Guide
Nova-network Switch configuration (Cisco Catalyst
2960G)
spanning-tree mode rapid-pvst
spanning-tree loopguard default
spanning-tree etherchannel guard misconfig
spanning-tree extend system-id
!
ip ssh time-out 60
ip ssh authentication-retries 2
ip ssh version 2
!
vlan 100
name Public
vlan 101
name Management
vlan 102
name Storage
vlan 103
name Private
vlan 104
name Admin
!
interface GigabitEthernet0/1
description Fuel Node eth0
switchport access vlan 104
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2
description Fuel Node eth1 (optional to have direct access to Public net)
switchport access vlan 100
switchport mode access
spanning-tree portfast
interface GigabitEthernet0/3
description Compute Node 1 eth0
switchport trunk native vlan 104
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 101, 102, 104
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/4
description Compute Node 1 eth1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 103
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/5
description Compute Node 2 eth0
switchport trunk native vlan 104
switchport trunk encapsulation dot1q
©2014, Mirantis Inc.
Page 39
Mirantis OpenStack v5.1
Planning Guide
switchport trunk allowed vlan 101, 102,
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/6
description Compute Node 2 eth1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 103
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/7
description Controller Node 1 eth0
switchport trunk native vlan 104
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 101, 102,
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/8
description Controller Node 1 eth1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 103
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/9
description Controller Node 2 eth0
switchport trunk native vlan 104
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 101, 102,
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/10
description Controller Node 2 eth1
switchport trunk encapsulation dot1
switchport trunk allowed vlan 100, 103
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/11
description Controller Node 3 eth0
switchport trunk native vlan 104
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 101, 102,
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/12
description Controller Node 3 eth1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 103
switchport mode trunk
spanning-tree portfast trunk
©2014, Mirantis Inc.
Nova-network Switch configuration (Cisco Catalyst
2960G)
104
104
104
104
Page 40
Mirantis OpenStack v5.1
Planning Guide
Nova-network Switch configuration (Juniper EX4200)
interface GigabitEthernet0/13
description Cinder Node eth0
switchport trunk native vlan 104
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 101, 102, 104
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/14
description Cinder Node eth1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 103
switchport mode trunk
spanning-tree portfast trunk
interface GigabitEthernet0/24
description Connection to default gateway
switchport access vlan 100
switchport mode access
!
interface Vlan100
ip address 172.16.1.254 255.255.255.0
ip address 172.16.0.254 255.255.255.0 secondary
no shutdown
!
ip route 0.0.0.0 0.0.0.0 172.16.1.1
!
ip classless
no ip http server
no ip http secure-server
!
line con 0
session-timeout 15
privilege level 15
login local
password r00tme
!
line vty 0 15
session-timeout 15
login local
password r00tme
!
ntp server [ntp_server1] prefer
ntp server [ntp_server2]
Nova-network Switch configuration (Juniper EX4200)
Use the following configuration to deploy Mirantis OpenStack using Juniper EX4200 network switch.:
©2014, Mirantis Inc.
Page 41
Mirantis OpenStack v5.1
Planning Guide
Nova-network Switch configuration (Juniper EX4200)
system {
host-name OpenStack_sw1;
domain-name domain.ltd;
authentication-order [ password ];
root-authentication {
encrypted-password "xxxxxxxxxxxxxxxxxxx";
}
}
services {
ssh;
}
ntp {
server [ntp_server1] prefer;
server [ntp_server2];
}
}
interfaces {
ge-0/0/0 {
description Fuel Node eth0;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_104;
}
}
}
}
ge-0/0/1 {
description Fuel Node eth1 (optional to have direct access to Public
net);
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_100;
}
}
}
}
ge-0/0/2 {
description Compute Node 1 eth0;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
©2014, Mirantis Inc.
Page 42
Mirantis OpenStack v5.1
Planning Guide
Nova-network Switch configuration (Juniper EX4200)
}
native-vlan-id vlan_104;
}
}
}
ge-0/0/3 {
description Compute Node 1 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_100, vlan_103;
}
}
}
}
ge-0/0/4 {
description Compute Node 2 eth0;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_104;
}
}
}
ge-0/0/5 {
description Compute Node 2 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_100, vlan_103;
}
}
}
}
ge-0/0/6 {
description Controller Node 1 eth0;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_104;
©2014, Mirantis Inc.
Page 43
Mirantis OpenStack v5.1
Planning Guide
Nova-network Switch configuration (Juniper EX4200)
}
}
}
ge-0/0/7 {
description controller Node 1 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_100, vlan_103;
}
}
}
}
ge-0/0/8 {
description Controller Node 2 eth0;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_104;
}
}
}
ge-0/0/9 {
description Controller Node 2 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_100, vlan_103;
}
}
}
}
ge-0/0/10 {
description Controller Node 3 eth0;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_104;
}
}
©2014, Mirantis Inc.
Page 44
Mirantis OpenStack v5.1
Planning Guide
Nova-network Switch configuration (Juniper EX4200)
}
ge-0/0/11 {
description Controller Node 3 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_100, vlan_103;
}
}
}
}
ge-0/0/12 {
description Cinder Node 1 eth0;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_104;
}
}
}
ge-0/0/13 {
description Cinder Node 1 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_100, vlan_103;
}
}
}
}
ge-0/0/23 {
description Connection to default gateway;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_100;
}
}
}
}
vlan {
unit 100 {
©2014, Mirantis Inc.
Page 45
Mirantis OpenStack v5.1
Planning Guide
Nova-network Switch configuration (Juniper EX4200)
family inet {
address 172.16.1.254/24;
address 172.16.0.254/24;
}
}
}
}
routing-options {
static {
route 0.0.0.0/0 next-hop 172.16.1.1;
}
}
protocols {
dcbx {
interface all;
}
rstp {
bridge-priority 32k;
interface ge-0/0/0.0 {
edge;
}
interface ge-0/0/1.0 {
edge;
}
interface ge-0/0/23.0 {
edge;
}
bpdu-block-on-edge;
}
lldp {
interface all;
}
}
vlans {
vlan_1;
vlan_100 {
description Public;
vlan-id 100;
l3-interface vlan.100;
}
vlan_101 {
description Management;
vlan-id 101;
}
vlan_102 {
description Storage;
vlan-id 102;
}
©2014, Mirantis Inc.
Page 46
Mirantis OpenStack v5.1
Planning Guide
Nova-network Switch configuration (Juniper EX4200)
vlan_103 {
description Private;
vlan-id 103;
}
vlan_104 {
description Admin;
vlan-id 104;
}
}
©2014, Mirantis Inc.
Page 47
Mirantis OpenStack v5.1
Planning Guide
Example 2: HA + Neutron with GRE
Example 2: HA + Neutron with GRE
As a model example, the following configuration is used:
• Deploying mode: Multi-node HA
• Networking model: Neutron with GRE
Hardware and environment:
• 7 servers with two 1Gb/s ethernet NIC and IPMI
• 1 Cisco Catalyst 3750 switch
• Independent out of band management network for IPMI
• Connection to the Internet or/and DC network via a router called Gateway and IP 172.16.1.1
Node servers roles:
• 1 server as Fuel Node
• 3 servers as Controller Node
• 1 server as Cinder Node
• 2 servers as Compute Node
Network Configuration Plan:
• Floating/Public network 172.16.0.0/24 in VLAN 100 (untagged on servers)
• Floating IP range 172.16.0.130 - 254
• Internal network (private) 192.168.111.0/24
• Gateway 192.168.111.1
• DNS 8.8.4.4, 8.8.8.8
• Tunnel ID range 2 - 65535
• Management network 192.168.0.0/24 in VLAN 101
• Storage network 192.168.1.0/24 in VLAN 102
• Administrative network (for Fuel) 10.20.0.0/24 in VLAN 103
Network Parameters
• Fuel server: IP 10.20.0.2/24
• Default gateway: 10.20.0.1
• DNS: 10.20.0.1
Note
The Internet and rest of DC access via Public network (for OpenStack Nodes) and Administrative network
(Fuel server).
©2014, Mirantis Inc.
Page 48
Mirantis OpenStack v5.1
Planning Guide
Detailed port configuration
From server side, ports with following VLAN IDs are used:
• eth0 - Administrative VLAN 103 (untagged)
• eth1 - Public/Floating VLAN 100 (untagged), Management VLAN 101 (tagged), Storage VLAN 102 (tagged)
Detailed port configuration
The following table describes port configuration for this deployment.
Switch
Port
Server name
Server
NIC
tagged /
untagged
VLAN ID
G0/1
Fuel
eth0
untagged
103
G0/2
Fuel
eth1
untagged
100
G0/3
Compute Node 1
eth0
untagged
103
G0/4
Compute Node 1
eth1
tagged
100(untagged), 101, 102
G0/5
Compute Node n
eth0
tagged
103
G0/6
Compute Node n
eth1
tagged
100(untagged), 101, 102
G0/7
Controller Node 1
eth0
tagged
103
G0/8
Controller Node 1
eth1
tagged
100(untagged), 101, 102
G0/9
Controller Node 2
eth0
tagged
103
G0/10
Controller Node 2
eth1
tagged
100(untagged), 101, 102
G0/11
Controller Node 3
eth0
tagged
103
G0/12
Controller Node 3
eth1
tagged
100(untagged), 101, 102
G0/13
Cinder Node
eth0
tagged
103
G0/14
Cinder Node
eth1
tagged
100(untagged), 101, 102
G0/24
Router (default gateway)
untagged
100
• Neutron Switch configuration (Cisco Catalyst 2960G)
Use the following configuration to deploy Mirantis OpenStack using a Cisco Catalyst 2960G network switch.
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname OpenStack_sw1
!
logging count
©2014, Mirantis Inc.
Page 49
Mirantis OpenStack v5.1
Planning Guide
Detailed port configuration
logging buffered 64000 informational
logging rate-limit console 100 except errors
logging console informational
enable secret r00tme
!
username root privilege 15 secret r00tme
!
no aaa new-model
aaa session-id common
ip subnet-zero
ip domain-name domain.ltd
ip name-server [ip of domain name server]
!
spanning-tree mode rapid-pvst
spanning-tree loopguard default
spanning-tree etherchannel guard misconfig
spanning-tree extend system-id
!
ip ssh time-out 60
ip ssh authentication-retries 2
ip ssh version 2
!
vlan 100
name Public
vlan 101
name Management
vlan 102
name Storage
vlan 103
name Admin
!
interface GigabitEthernet0/1
description Fuel Node eth0
switchport access vlan 103
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2
description Fuel Node eth1 (optional to have direct access to Public net)
switchport access vlan 100
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/3
description Compute Node 1 eth0
switchport access vlan 103
switchport mode access
spanning-tree portfast
©2014, Mirantis Inc.
Page 50
Mirantis OpenStack v5.1
Planning Guide
!
interface GigabitEthernet0/4
description Compute Node 1 eth1
switchport trunk native vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 101
switchport mode trunk
spanning-tree portfast trunk
!
interface GigabitEthernet0/5
description Compute Node 2 eth0
switchport access vlan 103
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/6
description Compute Node 2 eth1
switchport trunk native vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 101
switchport mode trunk
spanning-tree portfast trunk
!
interface GigabitEthernet0/7
description Controller Node 1 eth0
switchport access vlan 103
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/8
description Controller Node 1 eth1
switchport trunk native vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 101
switchport mode trunk
spanning-tree portfast trunk
!
interface GigabitEthernet0/9
description Controller Node 2 eth0
switchport access vlan 103
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/10
description Controller Node 2 eth1
switchport trunk native vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 101
©2014, Mirantis Inc.
Detailed port configuration
102
102
102
102
Page 51
Mirantis OpenStack v5.1
Planning Guide
Detailed port configuration
switchport mode trunk
spanning-tree portfast trunk
!
interface GigabitEthernet0/11
description Controller Node 3 eth0
switchport access vlan 103
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/12
description Controller Node 3 eth1
switchport trunk native vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 101 102
switchport mode trunk
spanning-tree portfast trunk
!
interface GigabitEthernet0/13
description Cinder Node eth0
switchport access vlan 103
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/14
description Cinder Node eth1
switchport trunk native vlan 100
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100, 101 102
switchport mode trunk
spanning-tree portfast trunk
!
interface GigabitEthernet0/24
description Connection to default gateway
switchport access vlan 100
switchport mode access
!
interface Vlan100
ip address 172.16.1.254 255.255.255.0
ip address 172.16.0.254 255.255.255.0 secondary
no shutdown
!
ip route 0.0.0.0 0.0.0.0 172.16.1.1
!
ip classless
no ip http server
no ip http secure-server
!
line con 0
©2014, Mirantis Inc.
Page 52
Mirantis OpenStack v5.1
Planning Guide
Neutron Switch configuration (Juniper EX4200)
session-timeout 15
privilege level 15
login local
password r00tme
!
line vty 0 15
session-timeout 15
login local
password r00tme
!
ntp server [ntp_server1] prefer
ntp server [ntp_server2]
Neutron Switch configuration (Juniper EX4200)
Use the following configuration to deploy Mirantis OpenStack using Juniper EX4200 network switch.
system {
host-name OpenStack_sw1;
domain-name domain.ltd;
authentication-order [ password ];
root-authentication {
encrypted-password "xxxxxxxxxxxxxxxxxxx";
}
}
services {
ssh;
}
ntp {
server [ntp_server1] prefer;
server [ntp_server2];
}
}
interfaces {
ge-0/0/0 {
description Fuel Node eth0;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_103;
}
}
}
}
ge-0/0/1 {
©2014, Mirantis Inc.
Page 53
Mirantis OpenStack v5.1
Planning Guide
Neutron Switch configuration (Juniper EX4200)
description Fuel Node eth1 (optional to have direct access to Public
net);
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_100;
}
}
}
}
ge-0/0/2 {
description Compute Node 1 eth0;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_103;
}
}
}
}
ge-0/0/3 {
description Compute Node 1 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_100;
}
}
}
ge-0/0/4 {
description Compute Node 2 eth0;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_103;
}
}
}
}
ge-0/0/5 {
description Compute Node 2 eth1;
unit 0 {
©2014, Mirantis Inc.
Page 54
Mirantis OpenStack v5.1
Planning Guide
Neutron Switch configuration (Juniper EX4200)
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_100;
}
}
}
ge-0/0/6 {
description Controller Node 1 eth0;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_103;
}
}
}
}
ge-0/0/7 {
description controller Node 1 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_100;
}
}
}
ge-0/0/8 {
description Controller Node 2 eth0;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_103;
}
}
}
}
ge-0/0/9 {
description Controller Node 2 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
©2014, Mirantis Inc.
Page 55
Mirantis OpenStack v5.1
Planning Guide
Neutron Switch configuration (Juniper EX4200)
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_100;
}
}
}
ge-0/0/10 {
description Controller Node 3 eth0;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_103;
}
}
}
}
ge-0/0/11 {
description Controller Node 3 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
}
native-vlan-id vlan_100;
}
}
}
ge-0/0/12 {
description Cinder Node 1 eth0;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_103;
}
}
}
}
ge-0/0/13 {
description Cinder Node 1 eth1;
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members vlan_101, vlan_102;
©2014, Mirantis Inc.
Page 56
Mirantis OpenStack v5.1
Planning Guide
Neutron Switch configuration (Juniper EX4200)
}
native-vlan-id vlan_100;
}
}
}
ge-0/0/23 {
description Connection to default gateway;
unit 0 {
family ethernet-switching {
port-mode access;
vlan {
members vlan_100;
}
}
}
}
vlan {
unit 100 {
family inet {
address 172.16.1.254/24;
address 172.16.0.254/24;
}
}
}
}
routing-options {
static {
route 0.0.0.0/0 next-hop 172.16.1.1;
}
}
protocols {
dcbx {
interface all;
}
rstp {
bridge-priority 32k;
interface ge-0/0/0.0 {
edge;
}
interface ge-0/0/1.0 {
edge;
}
interface ge-0/0/2.0 {
edge;
}
interface ge-0/0/4.0 {
edge;
©2014, Mirantis Inc.
Page 57
Mirantis OpenStack v5.1
Planning Guide
Neutron Switch configuration (Juniper EX4200)
}
interface ge-0/0/6.0 {
edge;
}
interface ge-0/0/8.0 {
edge;
}
interface ge-0/0/10.0 {
edge;
}
interface ge-0/0/12.0 {
edge;
}
interface ge-0/0/23.0 {
edge;
}
bpdu-block-on-edge;
}
lldp {
interface all;
}
}
vlans {
vlan_1;
vlan_100 {
description Public;
vlan-id 100;
l3-interface vlan.100;
}
vlan_101 {
description Management;
vlan-id 101;
}
vlan_102 {
description Storage;
vlan-id 102;
}
vlan_103 {
description Admin;
vlan-id 103;
}
}
©2014, Mirantis Inc.
Page 58
Mirantis OpenStack v5.1
Planning Guide
Example 3: HA + Neutron with VLAN + SR-IOV & iSER
Example 3: HA + Neutron with VLAN + SR-IOV & iSER
As a model example, the following configuration is used:
• Deploying mode: Multi-node HA
• Networking model: Neutron with VLAN
Hardware and environment:
• 16 servers (1U) with two 1Gb/s ethernet NIC on-board, IPMI
• 16 Mellanox ConnectX-3 Pro 40/56GbE adapter card
• 16 40/56GbE cables (Mellanox OPN MC2207130)
• 1 Mellanox Ethernet switch SX1036 (36 ports of 40/56GbE)
• 1 (or 2) 1G Ethernet switch
• 1 storage server (2U) with two 1Gb/s ethernet NIC on-board, IPMI
• 2 External JBODs + SATA cables - Optional disk + adaptec RAID controller
Note
The Fuel node doesn't have to be equipped with Mellanox ConnectX-3 Pro adapter card, as it is not
connected to the high speed network.
Node servers roles:
• 1 server as Fuel Node
• 3 servers as Controller Nodes
• 1 server as Storage Node (Cinder)
• 12 servers as Compute Nodes
Network Configuration Plan:
The switches and servers is designed in this example to support five networks as follows:
Cloud Networks
• Admin (PXE) network - 10.142.10.0/24 (no gateway, untagged via the 1GbE switch)
• Managemenet network - 192.168.0.0/24 (no gateway, VLAN 3 via Mellanox SX1036 40/56GbE switch)
• Storage network - 192.168.1.0/24 (no gateway, VLAN 4 via Mellanox SX1036 40/56GbE switch)
• Public network - 10.7.208.0/24 (gateway 10.7.208.1, untagged via the 1GbE switch)
• Private network - <any> (no gateway, use range of VLANs e.g. 5-15 via Mellanox SX1036 40/56GbE switch)
©2014, Mirantis Inc.
Page 59
Mirantis OpenStack v5.1
Planning Guide
Example 3: HA + Neutron with VLAN + SR-IOV & iSER
Note
Internet access is acheived via the Public network.
Note
All nodes should be connected to all networks (besides the Fuel node).
Note
The 1GbE switch configuraiton for the Public and Admin (PXE) networks are similar to examples 1 and 2
above.
Network Configuation
• Floating IP range 10.7.208.101-200
• DNS 8.8.4.4, 8.8.8.8
• Fuel server: Admin (PXE) IP 10.142.10.10/24
From server side (all nodes), ports with following VLAN IDs are used:
• eth0 - Admin (PXE) - untagged
• eth1 - Public - untagged
• eth2 (40/56GbE port) - Management VLAN 3, Storage VLAN 4 (+ VLANS 5-15 for the privte networks)
©2014, Mirantis Inc.
Page 60
Mirantis OpenStack v5.1
Planning Guide
Example 3: HA + Neutron with VLAN + SR-IOV & iSER
Here is an example of the network diagram:
©2014, Mirantis Inc.
Page 61
Mirantis OpenStack v5.1
Planning Guide
Neutron Switch configuration (Mellanox SX1036)
Rack Design
Here is a recommended rack design configuration. Design the rack as the 1G switches are on top, followed by the
servers, then the 40/56GbE switch and then the storage server and JBODs.
Neutron Switch configuration (Mellanox SX1036)
Use the following configuration to deploy Mirantis OpenStack using a Mellanox SX1036 40/56GE 36 port switch
The switch configuration is required prior to the Fuel instllation. Prior the installation, the network connectivity
between all hosts should be ready.
Here is an example of Mellanox switch VLAN configuation and flow control:
switch
switch
switch
switch
> enable
# configure terminal
(config) # vlan 1-20
(config vlan 1-20) # exit
# Note
switch
switch
switch
switch
switch
switch
that VLAN 1 is an untagged VLAN by default
(config) # interface ethernet 1/1 switchport
(config) # interface ethernet 1/1 switchport
(config) # interface ethernet 1/2 switchport
(config) # interface ethernet 1/2 switchport
(config) # interface ethernet 1/3 switchport
(config) # interface ethernet 1/3 switchport
©2014, Mirantis Inc.
mode hybrid
hybrid allowed-vlan 2-20
mode hybrid
hybrid allowed-vlan 2-20
mode hybrid
hybrid allowed-vlan 2-20
Page 62
Mirantis OpenStack v5.1
Planning Guide
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
switch
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
(config)
©2014, Mirantis Inc.
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
Neutron Switch configuration (Mellanox SX1036)
interface ethernet 1/4 switchport mode hybrid
interface ethernet 1/4 switchport hybrid allowed-vlan 2-20
interface ethernet 1/5 switchport mode hybrid
interface ethernet 1/5 switchport hybrid allowed-vlan 2-20
interface ethernet 1/6 switchport mode hybrid
interface ethernet 1/6 switchport hybrid allowed-vlan 2-20
interface ethernet 1/7 switchport mode hybrid
interface ethernet 1/7 switchport hybrid allowed-vlan 2-20
interface ethernet 1/8 switchport mode hybrid
interface ethernet 1/8 switchport hybrid allowed-vlan 2-20
interface ethernet 1/9 switchport mode hybrid
interface ethernet 1/9 switchport hybrid allowed-vlan 2-20
interface ethernet 1/10 switchport mode hybrid
interface ethernet 1/10 switchport hybrid allowed-vlan 2-20
interface ethernet 1/11 switchport mode hybrid
interface ethernet 1/11 switchport hybrid allowed-vlan 2-20
interface ethernet 1/12 switchport mode hybrid
interface ethernet 1/12 switchport hybrid allowed-vlan 2-20
interface ethernet 1/13 switchport mode hybrid
interface ethernet 1/13 switchport hybrid allowed-vlan 2-20
interface ethernet 1/14 switchport mode hybrid
interface ethernet 1/14 switchport hybrid allowed-vlan 2-20
interface ethernet 1/15 switchport mode hybrid
interface ethernet 1/15 switchport hybrid allowed-vlan 2-20
interface ethernet 1/16 switchport mode hybrid
interface ethernet 1/16 switchport hybrid allowed-vlan 2-20
interface ethernet 1/1-1/16 flowcontrol receive on force
interface ethernet 1/1-1/16 flowcontrol send on force
configuration write
Page 63
Mirantis OpenStack v5.1
Planning Guide
Index
Index
P
Preparing for the Mirantis OpenStack Deployment
S
System Requirements
©2014, Mirantis Inc.
Page 65