1. technology and computing

DDN | Whitepaper
A Beginner’s Guide To Next
Generation Object Storage
Tom Leyden, Director of Product Marketing WOS
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
Executive Summary
Object Storage is the new storage paradigm. There is a high level of interest from organizations, as this new
approach resolves the challenges of efficiently storing massive volumes of unstructured data - Big Unstructured
Data. This paper addresses the why, what and how of object storage.
Why should companies use Object Storage for unstructured data and how is it different from NAS or SAN?
The biggest problem with traditional approaches is scalability. NAS lacks the ability to scale as a single system,
especially in Petabyte environments. Today’s SANs are already complex, when deployed with a file system layer on
top. Scaling-out makes the problem a lot worse.
Object Storage is essentially just a different way
of storing, organizing and accessing data on disk.
An Object Storage platform provides a storage
infrastructure to store files with lots of metadata
added to them – referred to as objects. The backend
architecture of an object storage platform is
designed to present all the storage nodes as one
single pool. With Object Storage, there is no file
system hierarchy. The architecture of the platform,
and its new data protection schemes (vs. RAID, the
de-facto data protection scheme for SAN and NAS),
allow this pool to scale virtually to an unlimited size,
Figure 1 Object Storage Architecture
while keeping the system simple to manage.
Users access object storage through applications that typically use a REST API (an internet protocol, optimized for
online applications). This makes object storage ideal for all online, cloud, environments. When objects are stored, an
identifier is created to locate the object in the pool. Applications can very quickly retrieve the right data for the users
through the object identifier or by querying the metadata (information about the objects, like the name, when it
was created, by who etc.). This approach enables significantly faster access and much less overhead than locating a
file through a traditional file system.
DDN | WOS is a true object storage platform, designed to scale beyond petabytes as a single system, optimizing TCO
without compromising performance or durability. This makes WOS a perfect platform for a variety of storage cloud
solutions, including online collaboration, active archives, cloud backup and worldwide data distribution.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
2
Table of Contents
Executive Summary
2
History of Object Storage4
SAN vs NAS5
Object Storage, The Third Paradigm6
Cloud Storage, Storage Clouds, Object Storage7
REST API’s7
Object Storage Summary8
Why Object Storage? 8
Massive Data Growth8
Always Online8
Power to the Applications9
The Big Data Explosion9
We All Use Object Storage Everyday10
Use Cases10
How Does Object Storage Work?
11
Issues with File Storage11
Data Protection: Erasure Coding or Not?14
WOS15
True Object Storage Platform16
Optimized for Small and Large Files16
Choice of Data Protection Schemes16
Self-healing Architecture16
Single Storage Infrastructure16
Widest Selection of Interfaces; Out of the Box Applications
16
Enterprise-grade Platform17
WOS Benefits18
Ecosystem18
Resources20
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
3
History of Object Storage
Object Storage is not a new concept, it even predates the 2006 launch of Amazon’s S3®. Much advancement
have been made, and today’s current generation of Object Storage platforms cannot be compared to the earlier
generations - which were merely black boxes designed to store immutable copies of documents, mostly for
compliance environments. EMC® Centera®, based on Content Addressed Storage (CAS) innovator Filepool, was one
such early implementation of an object-based construct. Today, Centera users face a big challenge to move to newer,
faster storage infrastructures: Centera provides no interfaces to migrate data to other platforms.
Figure 2 Object Storage Timeline
The current generation of object storage platforms is designed with this “openness” & flexibility in mind. Most
platforms support a subset of Amazon’s REST API and some platforms are designed to be independent of the
hardware platform. The industry has learned some tough lessons from using proprietary systems. One initiative
to prevent Vendor Lock-in, is SNIA’s Cloud Data Management Interface (CDMI). This is a set of pre-defined RESTful
HTTP operations “for assessing the capabilities of the cloud storage system, allocating and accessing containers
and objects, managing users and groups, implementing access control, attaching metadata, billing, moving data
between cloud systems, exporting data, etc.”1
What is Object Storage?
Object Storage is essentially just a different way of storing, organizing and accessing data on disk. To really
understand how object storage is different from traditional storage platforms, it is important to understand the
“what and how” of traditional storage, and what the challenges are2.
From Wikipedia: http://en.wikipedia.org/wiki/Cloud_Data_Management_Interface
1
For the sake of briefness, we will stick to the very basics. There are hundreds, if not thousands of blog articles and papers about this topic available online.
2
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
4
SAN vs NAS
A SAN is block storage device, not that different from an external USB disk drive, just bigger. Systems connect to a
SAN with a block interface; common protocols for block storage include iSCSI, Fibre Channel, Fibre Channel over
Ethernet (FCoE), etc. A device attaching to SAN will see the storage presented as a disk drive. SANs allow multiple
servers to share a pool of storage that cannot be accessed by individual users. This is to prevent the overwriting each
other’s data. SANs are typically used by large applications, such as enterprise databases, that handle data locking
through the application. SAN storage can be presented as a file system (by putting a file system layer on top), which
is generally referred to as a clustered file system. As we will explain later in this document, SANs are complex systems
to manage, especially when used for file storage.
Figure 3 Simplified SAN infrastructure with Clustered file system and enterprise applications
A NAS is a file storage device. NAS exposes its storage as a network file system. Devices that attach to a NAS see a
mountable file system. Common protocols for file storage devices include, NFS and SMB / CIFS. A NAS operates at
the file level and is accessible to users with proper access rights - so it needs to manage user privileges, file locking
and other security measures. A NAS environment is a much better fit than SANs for to store files.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
5
Object Storage, The Third Paradigm
So, if NAS & SAN can store files, then, why Object Storage? How is it different? As we will explain in greater detail
to follow, the biggest problem with both systems is scalability. NAS cannot scale as a single system in petabytesize environments. To scale-out a NAS environment requires a combination of multiple systems (management!) or
forklift upgrades, with labor-intensive data migration projects. As we mentioned before, SANs are pretty complex
when deployed with a file system layer on top. Scaling-out makes the problem a lot worse. Again, with lots of
management!
Also, most of the unstructured data that is stored
online (or in active archives) is immutable data –
meaning the file will not be modified. Much of
the functionality built-in to traditional file systems
addresses user access rights and permissions for
appending and amending files. These complex
functions create a lot of overhead in terms of
performance, IOs required to access data and the
ability to scale. Object Storage does not have this
functionality. If a user modifies a file, the new version
is simply stored as a new object. This results in a much
simpler architecture than traditional file systems have.
An Object Storage platform is a storage infrastructure
to store … objects. For now we will refer to objects
Figure 4 NAS Storage is presented as a file system to the clients
as similar to files (collection of data blocks with
metadata), later in this document we will explain how
this is actually only partly true. The backend architecture of an object storage platform is designed so that all the
storage nodes are presented as one single pool. There is no file system hierarchy. The architecture of the platform,
and new data protection schemes (vs. RAID, the de-facto data protection scheme for SAN and NAS) allow this pool
to scale to virtually unlimited capacities, while keeping the system simple to manage.
Users access object storage through applications that will typically use a REST API. They use a set of simple
commands: GET (read), PUT (save) and DELETE. REST is an internet protocol, optimized for online applications. This
makes object storage ideal for all online, Cloud, environments. When objects are stored, an identifier is created to
locate the object in the pool. Applications can very quickly retrieve the right data for the users through the object
identifier - or by querying the metadata (information about objects: name, when it was created, by who, etc.). This
is much faster than attempting to locate a file through a traditional file system. Applications also handle user access
management. Each time a file (object) is changed, it is stored as a new object. This prevents corruption through
simultaneous changes.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
6
Figure 5 Scale out object storage with simple REST API for applications
Cloud Storage, Storage Clouds, Object Storage
What is the difference between Cloud Storage and Storage Clouds? How does Object Storage fit? The answer is
pretty straightforward. Cloud Storage is the storage used for Compute Cloud infrastructures - in other words: to run
VM’s on. Compute Clouds are very IOPS intensive and usually block storage is used in these applications. Storage
Clouds are “storage in the cloud”, whether public or private. So, Storage Clouds are simply storage capacity that is
made available through the Internet. Most of today’s storage clouds use object storage technologies.
REST API’s
REST stands for Representational State Transfer. It is a software architecture that is used for distributed application
environments, such as the internet. An API, short for Application Programming Interface, is an interface used for an
application (client) to talk to its environment (backend servers, storage, databases etc.). With the success of cloudstyle computing (running applications in the cloud, rather than on the user’s computer), REST API’s have become the
predominant interface for cloud applications to connect to the cloud. For storage-centric cloud applications, a REST
API is the interface between the application and the object storage platform.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
7
The three most common commands in REST API’s for object storage environments are GET, PUT and DELETE, which
are the equivalents of reading a file, saving a file (technically “save as” – because object storage does not allow you to
update an object), or deleting a file.
Since the early days of Cloud Computing, there’s been a lot of discussion about standardizing on a specific REST API
to avoid vendor lock-in. The general idea behind this is, if all vendors (of applications, cloud infrastructures, object
storage platforms etc.) use a standard API, users will never be locked-in to a specific environment. Without having to
reprogram their applications, they would be able to freely move their data from one platform to another - or keep
it on more than one platform. Little progress has been made on the standardization front however, and the result is
that object storage platforms will either support the Amazon S3 API, the OpenStack API or a native API (i.e. an API of
their own, typically a very easy to use, lightweight interface).
Object Storage Summary
••
••
••
••
Data is stored as objects in one large, scalable pool of storage
••
Objects are immutable; edits are saved as a new object
Objects are stored with metadata – information about the object
An Object ID is stored, to locate the data
REST is the standard interface, simple commands used by applications
Why Object Storage?
Massive Data Growth
Depending on which analyst firm you talk to, you will hear storage growth predictions that vary between 30x-40x for
the next decade. That means we will all be storing 30 to 40 times as much digital data ten years from now, compared
to today. At the same time, companies will only invest an additional 50% in personnel to manage their storage
infrastructures. This means that the average storage operator will have to manage 15-20 times as much storage a
decade from now. This will drive the need for storage platforms that require little management effort and scale out
to virtually unlimited capacities.
Always Online
Much of that data growth is driven by the recent innovations in cloud and mobile computing. We already mentioned
Amazon S3, but there are also Google®, Facebook®, Apple® and several smaller public storage cloud offerings that set
a new level of expectations where all data needs to be available anywhere at anytime.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
8
Power to the Applications
File-based storage platforms not only fail to scale sufficiently, they also become obsolete as more and more
applications are designed to use REST API’s (the default interface for object storage platforms) to talk directly to
the storage, without additional (file system) layers in between. This greatly simplifies architectures and delivers
significant performance gains.
The Big Data Explosion
Essentially, there are three types of Big Data: Big Structured Data, Big Semi-structured Data (Big Data Analytics) and
Big Unstructured Data. All three require one or more of the “three V’s”, the commonly accepted definition of Big Data:
“Big Data refers to any set of data that comes in great Volumes, has a large Variety of information and/or is consumed
at high Velocity.
Big Structured Data refers to large enterprise databases. Velocity is key here, hence the success of the superfast
SSD drives. Big Semi-structured Data refers to massive volumes of small log files (often sensor information), that is
collected for analytics. Therefore, we also talk about Big Data Analytics. This data is stored in distributed frameworks
that support distributed processing. Think of Hadoop® and MapReduce. Object Storage is not commonly used for
structured or semi-structured Big Data, unless it is for archival purposes.
The sweet spot for object storage is Big Unstructured Data, which refers to all data that users best understand
as files. Think of image and movie data – always growing and always in higher resolution – music files, office
documents etc. Analysts believe that 80% or more of the expected data growth will be unstructured data. File based
storage platforms cannot support this growth. This is the problem Object Storage solves.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
9
We All Use Object Storage Everyday
Possibly the largest object storage infrastructure, and one of the drivers for the adoption of object storage is
Amazon’s S3. This “public” storage cloud service was launched in 2006 and has stimulated many application
developers to have their applications use S3 as backend storage. The benefits were clear: no hassle with a private
infrastructure, relatively low cost, pay as you go, scale as needed and a very simple interface.
However, while Amazon advertises very low cost to store data on their infrastructure, there are some hidden costs
such as network traffic. At a certain volume of data, there is a point of cost inflection. Many of the startups that
launched on Amazon over the past years and who clearly see the benefits of object storage, are now deploying their
own infrastructure using object storage platforms.
Also, not everyone wants their data in a public environment with debatable SLA’s and moderate security at best.
More and more enterprises are choosing to deploy their own internal storage clouds to facilitate cloud-based
applications. These infrastructures need similar or better availability and durability than the available public services.
Use Cases
Object Storage is more than a smarter paradigm that allows you to store large volumes of unstructured data.
Features like massive scalability, REST APIs, geographic distribution, enable a series of compelling use cases. An
interesting side effect is that solutions tend to overlap. Dropbox® is not just file sharing, it’s backup, collaboration,
archiving and mobile storage. Here are a few popular use cases:
Online Web Services: As we mentioned earlier, one of the drivers for object storage is the trend to use more and more online cloud applications. Previously, without Amazon’s S3, none of this would have been possible. The more successful web services companies are now gradually making the move to in-house infrastructures. Also, with corporate security policies, IP and compliance considerations, most enterprises prefer to run cloud
applications on private storage infrastructures.
File Sharing is by far the most popular object storage use case. Dropbox offered a solution for a need that most of us did not know we had. Today, service providers are now deploying similar services and enterprises are deploying private file sharing services – as people utilize a variety of devices at home, at work and on-the-go. They collaborate with people across the office or around the world.
Cloud Backup is increasingly popular. There are dozens and dozens of online services for backup. For enterprises, the idea of backing up to low cost, highly scalable disk infrastructures - rather than tape, which can be cumbersome for recovery - is also very compelling
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
10
Cloud Archives: Data archiving decisions used to be very simple: data that was infrequently accessed was moved off disk to tape. Very few arguments could beat the low TCO of tape. Disk archives were hard to justify and reserved for those exceptional use cases where latency outweighed the huge cost of disk archives. With object storage, it is now possible to deploy disk archives at an acquisition cost and TCO close to that of tape. Many organizations are opting for hybrid environments - with a really, really superfast “hot” disk tier and a very cheap “cold” tape tier.
Worldwide Collaboration: Globally distributed teams have become standard practice. Think of researchers from different institutions working on the same project. Think of a movie being shot in New Zealand and produced in Los Angeles - or software being developed in California and then tested in India. Geographically distributed storage pools enable teams to work in real-time on the same datasets.
How Does Object Storage Work?
Issues with File Storage
As we explained earlier in this document, file based storage is a great concept: users can access the same resources
through a corporate network. The file system takes care of permissions, access rights and avoids users overwriting
each other’s data. File systems can even present data in a hierarchical “Directory” structure, which until now has been
a very useful tool to keep data organized. The underlying software for such file systems contains a lot of “ingenuity”,
which rapidly becomes “complexity” when scaling-out the infrastructure.
The concept of a “file” on a computer system is so well ingrained that it is often difficult to think of computer storage
in different terms. It is clearly a very powerful and natural way to think of data. Object Storage is distinct from “file
storage”, but in some ways is even a more natural and powerful way to organize data.
The “file” concept is an abstraction. In actuality, data in a computer system is stored in fixed size “blocks” which are
“addressed” with a number – which ultimately is a physical location in a storage device. This is the case for data
stored in a NAS, a SAN or when using Object Storage. The system presents those blocks to the user or application in
a form that is useful. For non-transactional data, that form is usually a file.
File storage systems will also store a small amount of information that tells which of those data blocks make up
the file, in which order, and finally what “name” has been applied to the collection of blocks. This additional “data
about data” is called “file system metadata”. Keeping track of the file system metadata is the responsibility of the “file
system”. To keep even more than a few files organized, a file system imposes a hierarchy on the metadata in the form
of “directory structure”. A key concept of the file system is the notion that the files themselves have relationships to
one another – as one could think of files being “co- located” in a directory.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
11
When the system is instructed to “read a file”, the repository of file system metadata is consulted and the required
data blocks are retrieved from the storage device. Writing data into a file system has the additional complexity
of requiring that the file system metadata must be written or updated - potentially by several users or processes
simultaneously. Numerous techniques and designs exist that attempt to minimize the impact of dealing with file
system metadata, and the “locking” problem associated with simultaneous access. Unfortunately, as the number
of files in the system grows large, keeping the file system metadata correctly organized (so that the names and the
data blocks that make up files can be found) becomes increasingly complex. As this requirement increases, keeping
track of billions of “files” (which may be distributed across a number of network connected computer systems),
the abstraction of the “file system” begins to breakdown. Moreover, the hierarchical structure of the “file system” is
insufficient to adequately categorize the data in the system.
File systems require at least three layers of software constructs to execute any file operation. As they allow files to
be amended by multiple users, they must maintain complex lock structures with OPEN and CLOSE semantics. These
lock structures must be distributed coherently to all of the servers used for access.
Also, as data is placed (based on random block availability), traditional file systems are always fragmented. This
is especially true in environments where the data is unstructured and it is not uncommon to write widely varied
file sizes. Using a traditional file system designed for amendable data, storing immutable data constitutes an
inappropriate and wasteful use of bandwidth and compute resources. This highly inefficient approach requires
a great deal of additional hardware and network resources to achieve data distribution goals. These systems now
become exponentially more complex as they are scaled-out.
Object storage systems dispense with the overburdened concept of file system metadata. This approach allows the
system to separate the storage of data from the relationship that the individual data items have to each other. In an
object storage system, the physical storage blocks are organized into “objects” which are collections of data blocks
represented by an identifier. There is no “hierarchy” imposed on the data and no repository of the objects’ metadata
to be consulted when reads or writes are requested. This approach allows an object storage system to scale with
both the requirements and size of the system, well beyond the technical & practical boundaries of traditional file
systems.
While Object Storage systems do not use file system metadata, they do employ object metadata (customizable
information about the objects). This information can later be used to query or analyze the information stored. Object
metadata for a photo could be the day it was taken, the last time it was modified, the type of camera that was used,
whether a flash was used, where it was taken, etc. Object metadata will play an increasingly important role as we
store more and more information, but it does not add complexity to the system like file system metadata does.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
12
•
At the highest level, storage servers are, like NAS and SAN, simply boxes with a lot of disks in there. Typically,
object storage vendors will use SATA disks in their systems, and may include SSDs for caching. Some
platforms opt for separate controllers, but in essence that does not make a difference, as the storage is
presented as one pool (namespace). When choosing an object storage platform, it’s important to understand
the limitations of the namespace and how the system combines different pools or namespaces. Many
vendors claim infinite scalability, but there is no such thing. The important thing is to understand how
namespaces are combined, presented and managed. How many such namespaces can be combined? Are
they managed as one system? The system software manages most of that.
•
The actual software layer is where vendors can differentiate. The list of possible features is endless. A single
management interface is always great. Self-healing capabilities are a must for environments that will scale
into the hundreds of petabytes. The software layer also provides data protection mechanisms, which we will
cover in the next section.An Object ID is stored, to locate the data
•
The standard interface to access data in an object storage platform is a RESTful interface or REST API. This is a
set of simple commands that application developers use in their code to let the application access the data.
The basic REST commands are LIST, GET, PUT and DELETE, which are used to list (a selection of ) objects, read
an object, store an object or delete it. There is no standard for REST yet, but the so called Amazon API is by
far the most popular amongst developers. Hence, most object storage providers will provide an “Amazon
compatible API”, which is typically a subset of the commands that are supported by Amazon S3. As most
legacy applications were designed to interface with a file system, most object storage platforms will also
provide one or more file interfaces (a file system layer on top of the object storage pool – also called a file
system gateway) and often a selection of programming language-specific API’s will be provided as well.
DDN’s WOS has the widest selection of interfaces on the market.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
13
Data Protection: Erasure Coding or Not?
Several data protection mechanisms are used for object storage. The one method that is being abandoned, however,
is RAID (which has been the de facto data protection scheme for SAN and NAS for the past two decades). The
problem with RAID is that it was originally designed for small capacity disks . The larger the drive capacity, the longer
it takes to restore a failed drive. During this restore, the data is less protected. If you are on RAID 5, and a second disk
fails in your RAID group during the lengthy restore, then data loss will occur. Also, as all processing capacity is used
for the restore, users will experience severe performance drops as data is being written to the replacement disk.
Large systems with hundreds of TBs or Petabytes, will routinely be in constant rebuild mode, as drives routinely fail.
In an effort to shorten these longer rebuild cycles, RAID systems ship with faster processors, which also consume
more energy, but this only masks the problem at best.
The simplest way to protect data is to make several copies, replication. A popular concept is called “three copies
in the cloud”, promoted mostly by public cloud platforms like Amazon S3 and Rackspace®. While three copies in
the cloud provides acceptable data protection, it is also very lucrative for the cloud provider as they are in the
business of selling more storage capacity. Swift™, the object storage component of Rackspace’s open source cloud
infrastructure also uses pure replication.
A more efficient data protection mechanism is erasure coding. Today, there are several flavors of erasure coding,
each one with its own benefits. Erasure coding’s key advantage is that you can break up your data into n fragments,
add m additional fragments, store the fragments across n+m devices, and then recover the original data from any n
of the devices. Survive 4 failures? 10? Pick a number! Also when a disk is lost, the system only has to calculate new
fragments, to be spread over any selection of disks with available capacity. This is a lot faster and more efficient than
restoring an entire RAID-based disk, even if it was only 20% full.
Erasure coding can be implemented locally or distributed, which means that fragments are spread over multiple
data centers (at least three) and the system can survive failure of a full datacenter. Distributed erasure coding
drastically reduces the overhead (the extra storage that is needed to protect the data). Five 9’s or more are
guaranteed with overhead numbers as low as 20% - as opposed to 3 copies requiring 200% overhead. The problem
with distributed erasure coding is that it creates a huge WAN cost, as rebuilds require data transfer between the data
centers. Also, the availability highly depends on the WAN connectivity. Each data read from a distributed erasure
coding pool, requires data to be read from three data centers. If any of these connections have high latency, the user
will notice the delayed response.
An interesting, “best of both worlds” solution is local erasure coding with replication. Such architecture combines
the benefits of erasure coding with those of replication (a full copy is present in each datacenter). While such a setup
requires more overhead than the distributed alternative, the TCO is typically a lot lower due to the reduced WAN
traffic.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
14
WOS
DDN’s legacy is designing high-performance storage systems, but without making things more complex than they
need to be. WOS is the perfect example of achieving operational excellence through reverse engineering - stripping
the architecture down to the very basics. The architecture of WOS consists of three components: WOS building
blocks, WOS Core software and a choice of simple interfaces.
•
The backend of a WOS storage infrastructure are the WOS
storage nodes. The storage nodes are essentially 4U servers
filled with 60 SATA disks. A WOS infrastructure can contain as few
as 3 nodes and scales to virtually unlimited capacity by adding
more nodes.
•
Smart storage requires intelligent software. WOS Core has
a single, straightforward management console for the entire
infrastructure - even when distributed across multiple sites. WOS
Core’s self-healing capabilities and other features drastically
reduce operator-driven maintenance interventions.
•
WOS provides the most complete choice of interfaces, including
a set of native API’s, file access interfaces and S3 REST.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
15
True Object Storage Platform
Most object storage platforms still have a POSIX file system layer on the disk level. WOS, however, was designed as a
true object storage platform, a flat, single layer, address structure where objects are stored in a contiguous group of
blocks so that disk operations are minimized (single-disk-operation reads – dual-operation writes), performance is
maximized and disks are used at full capacity.
Optimized for Small and Large Files
WOS is the only object storage system that is optimized for high-speed throughput of large data volumes and superfast I/O operations for small files. For multi-site deployments, the built-in WOS Latency-Aware Access Manager will
automatically address data access requests to the location with the lowest latency.
Choice of Data Protection Schemes
WOS offers a choice of data protection mechanisms to ensure the highest data durability AND availability. Reduce
your storage overhead while maximizing durability for single site deployments with local ObjectAssure™, DDN’s
implementation of Erasure Coding. Alternatively, you can choose Replicated ObjectAssure, to improve availability
without increasing WAN costs. Finally, ObjectAssure™ can be implemented in a distributed way to ensure higher
durability, at a lower cost.
Self-healing Architecture
Keeping traditional storage infrastructures healthy is management-intensive. Disks need to be replaced and
restored. Rebuild windows need to be kept to a minimum to avoid data loss and preserve application performance.
This is not the case with WOS. The built-in data protection algorithm, ObjectAssure has unique self-healing
capabilities that further reduce the management effort. Also, in case of a broken disk, ObjectAssure only has to
reconstruct the actual data that was lost - as opposed to the entire disk. This dramatically reduces the rebuild
window.
Single Storage Infrastructure
WOS is the only object storage platform that seamlessly integrates with other storage tiers. It has one management
interface for the entire infrastructure and supports easy data movement between different tiers, e.g. from GPFS to
WOS and back, or from and to Lustre environments.
Widest Selection of Interfaces; Out of the Box Applications
WOS provides the most complete choice of interfaces, including a set of native API’s, file access interfaces and S3
REST. In addition, WOS can be configured with preinstalled applications such as iRODS for data management or WOS
Share for secure global file sharing.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
16
Enterprise-grade Platform
Most vendors recommend commodity hardware for their object storage platforms. In the short term, this could
mean initial CAPEX savings, but as such devices typically have shorter replacement cycles, this highly impacts the
OPEX further down the road. This is especially so for multi petabyte deployments. While WOS was designed to be
hardware agnostic, we designed the WOS 7000 hardware to reduce TCO. Unlike commodity hardware, the WOS 7000
has an ultra dense form factor, so there are fewer systems to house, manage, power, cool and maintain. Leveraging
over 15 years of hardware design for the most demanding HPC environments, WOS 7000 was built to run many more
years than cheaper commodity hardware.
WOS Benefits
Lowest Global Access Latency
WOS was designed with the intent of maximizing performance for storage of massive volumes of immutable data.
Scales with All Varieties of Applications
WOS scales virtually unlimited in clusters as large as 30PB. Those clusters can consist of any mix of small (kilobytes)
or large (terabytes) files.
Best Durability & System Availability
WOS’ choice of data protection schemes allows the customer to deploy object storage that combines durability with
availability.
Lowest Administration Overhead, Lowest TCO
Through automated management, lower hardware costs, less power usage, simple architecture, optimized disk
usage and reduced WAN bandwidth usage; WOS enables organizations to store more data at a much lower cost.
Simple Integration
Integrate WOS with your GRIDScaler GPFS storage or your EXAScaler Lustre platform. Use WOS as an archive for your
HScaler Big Data Storage, or build an Active Archive of WOS with a tape library for offline cold archiving.
Maximum Portability
WOS features the most complete set of interfaces to facilitate your application integration, including C++ and Java
APIs for direct application integration, REST for web applications (S3 or not) and file gateways to support file-based
workflows.
Best Data Center Density
Designed for massive HPC deployments, WOS 7000 provides the highest data center density possible.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
17
Ecosystem
Object Storage is clearly the hot space in the storage industry, with offerings from both startups and established
storage solution providers. But, there is more than just object storage on the market: object storage has fostered a
wave of innovation that enables or leverages the paradigm.
The list of Tier 2 object storage players can be endless, especially when including the application providers. Here is
a short selection of popular gateways, WAN optimizers, collaboration platforms and other applications. This should
help to provide a better understanding of the object storage ecosystem and the opportunities and use cases.
Ctera® CTERA leverages object storage to offer a range of solutions for SMBs, enterprise branch offices and remote
users, including: data backup and recovery, file-based collaboration and mobile access.
Mezeo® also provides a number of storage solutions that leverage object storage, including: an AWS compatible
REST API and a number of file sync and share clients that give users access and collaboration capabilities from their
PC/Mac, smartphone, tablet or browser interface.
Panzura® built a NAS gateway for storage clouds. The gateway enables enterprises to combine multiple storage
(cloud) resources and make them accessible to multiple locations, presented as a unified global file system.
Aspera® both leverages and facilitates object storage. On the one hand, they have a number of applications for
collaboration, distribution etc., but the core of their technology is a protocol that optimizes how data is sent from
the object storage pool, over the WAN to a user application - or between sites, if an object storage infrastructure is
distributed over multiple locations.
Bitspeed® and Silverpeak® are active in the same space: WAN optimization, which enables faster, more reliable
and] secure data transfer between storage sites - or between the object storage pool and the application. These
technologies are becoming increasingly important in the deployment of object storage based storage clouds.
Dropbox® is probably the best-know object storage success case. This early AWS S3 customer launched a file-sharing application when no one even knew they needed one. The power of Dropbox lies in their use of deduplication
(when multiple users store the same file in their Dropbox, only one copy is kept). This way, Dropbox saves a lot on
storage costs. Deduplication is not new, but Dropbox pioneered its use in an online, object storage based application. This also allowed them to quickly gain a large user base through a fermium model, which would have been
unaffordable otherwise.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
18
Box(.net)™ also started as an online file sharing application but with some very important differences. Box runs
on their own (object storage) infrastructure, which gave them more control over security, data integrity etc. (as
compared to using S3). This allowed them to bring their solution to the SMB and Enterprise markets. Today, Box.net
grew to what can probably best be described as a storage-centric Platform as a Service, enabling organizations to
customize apps, integrate with their own applications etc.
Netflix®, which launched as a DVD rental by mail is an early adopter of object storage: in 2007 it launched a movie
streaming service which would disrupt the market. Well before Apple added movies and tv shows to their store,
Netflix leveraged S3 to offer movies in an online format.
Apple®, Google® and Facebook® also have massive object storage deployments, but little is known about their
architectures. Apple and Google are going after the S3 end users with document sharing and other storage in the
cloud services. With this, they compete both with Amazon and the applications that use S3 such as Dropbox and
Evernote.
Resources
http://knowledgelayer.softlayer.com/learning/introduction-object-storage
http://docs.openstack.org/trunk/openstack-object-storage/admin/content/ch_introduction-to-openstack-object-storage.html
http://cloudarchitect.att.com/Articles/Introduction-Object-Based-Storage
http://www.conres.com/hitachi-hds-object-storage-content-platform
http://www.cs.cmu.edu/~garth/RAIDpaper/Patterson88.pdf
http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162
https://en.wikipedia.org/wiki/Representational_state_transfer#RESTful_web_APIs
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
19
DDN | About Us
DataDirect Networks (DDN) is the world leader in massively scalable storage.
Our data storage and processing solutions and professional services enable content-rich and
high growth IT environments to achieve the highest levels of systems scalability, efficiency
and simplicity. DDN enables enterprises to extract value and deliver business results from their
information. Our customers include the world’s leading online content and social networking
providers, high performance cloud and grid computing, life sciences, media production, and
security and intelligence organizations. Deployed in thousands of mission critical environments
worldwide, DDN’s solutions have been designed, engineered and proven in the world’s most
scalable data centers to ensure competitive business advantage for today’s information powered
enterprise.
For more information, go to www.ddn.com or call +1.800.837.2298
© 2013, DataDirect Networks, Inc. All Rights Reserved. DataDirect Networks, EXAScaler, GRIDScaler,
hScaler, ReACT, SFA12K, SFA, SFX, Storage Fusion Xceleration, Web Object Storage, WOS are trademarks of
DataDirect Networks. All other trademarks are the property of their respective owners.
ddn.com
©2013 DataDirect Networks. All Rights Reserved.
Version-1 5/13
20