1. technology and computing
2. software
3. databases

: what’s all the buzz about?
http://nosql-database.org/
Next generation databases are:
• Non-relational,
• Distributed,
• Open-source,
• Horizontal scalable
Often more characteristics:
Schema-free, easy replication support, simple API,
eventually consistent / BASE (not ACID), a huge data
amount
List of NoSQL databases [122+]
•
Wide Column Store / Column Families
HBase, Cassandra, Hypertable, Cloudata, Cloudera, Amazon SimpleDB
•
Document Stores
CouchDB, MongoDB, Terrastore, ThruDB, OrientDB, RavenDB, Citrusleaf, SisoDB
•
Key Value / Tuple Store
Azure Table Storage, MEMBASE, Riak, Redis, Chordless, GenieDB, Scalaris, Tokyo Cabinet / Tyrant, Keyspace
Berkeley DB, MemcacheDB, Faircom C-Tree, Mnesia, LightCloud, Hibari, HamsterDB, STSdb, Pincaster, RaptorDB
•
Eventually Consistent Key Value Stores
Amazon Dynamo, Voldemort, Dynomite, KAI
•
Graph Databases
Neo4J, Infinite Graph, Sones, InfoGrid, HyperGraphDB, Trinity, AllegroGraph, Bigdata, DEX, OpenLink, Virtuoso,
VertexDB, FlockDB
•
Object Databases
db4o, Versant, Objectivity, Gemstone, Progress, Starcounter, Perst, Caching, ZODB, NEO, PicoLisp, Sterling
•
More and more databases
So what’s wrong with relational
databases?
Main principals of RDBMS
• SQL
• ACID
• Atomic “all or nothing”
• Consistent means that data moves from one correct
state to another correct state, with no possibility that
readers could view different values that don’t make
sense together.
• Isolated means that transactions executing
concurrently will not become entangled with each
other.
• Durable once a transaction has succeeded, the
changes will not be lost.
Shortcomings of RDBMS
• Transactions under heavy load
• Complexities of vertical scaling
• 2 phase commit (2PC) protocol
Sharding
If you can’t split it, you can’t scale it (Randy
Shoup, distinguished architect, eBay)
• Sharging approach
• Feature-based shard or functional segmentation
• Key-based sharding
• Lookup table
• Shared-nothing or Cassandra like sharding
The real question is not
“What’s wrong with relational
databases?” but rather,
“What problem do you have?”
Brewer’s CAP Theorem
Availability
Consistency
Partition
Tolerance
Brewer’s CAP Theorem
Availability
Relational:
MySQL, Oracle, MSSQL
Amazon Dynamo derivatives:
Cassandra, Voldemort,
Riak, CouchDB
Partition
Tolerance
Consistency
Neo4j, Google Big Table and
its derivatives: MongoDB,
Redis, Hypertable
in 50 words or less
Apache Cassandra is an open source, distributed,
decentralized, elastically scalable, highly available,
fault-tolerant, tuneably consistent, column-oriented
database that bases its distribution design on
Amazon’s Dynamo and its data model on Google’s
Bigtable. Created at Facebook, it is now used at
some of the most popular sites on the Web.
Cassandra case studies
Cassandra outlines
• BASE (Basically Available Soft-state Eventual
consistency) and not ACID (Atomicity,
Consistency, Isolation, Durability)
• Distributed and decentralized
• Elastic scalability
• High availability and fault tolerance
• Tunable consistency
Use cases for Cassandra
•
•
•
•
Large deployments
Lots of writes, statistics and analysis
Geographical distribution
Evolving applications
Writes
Memtable
Commit
log
Write
Threshold
SSTable
SSTable
•
•
•
•
•
•
•
•
No reads
No seeks
Fast
Sequential disk access
Atomic within a column family
Any node
Always writable (hinted hand-off)
≈ 0.2 ms
Reads
Memtable
Read
Bf
Idx
SSTable
Bf
Idx
SSTable
• Bloomfilter field to determine
whether a provided key is in the
SSTable
• Index field for quick read
• Any node
• Read repair
• ≈ 15 ms
The tenets of column-oriented model
• Keyspace
Outer container, that contains column
families (is sort of like a relational database)
• Column Family
Logical division that associates similar data
(very roughly analogous to tables in the
relational world)
• Column
Name/value pair (and a client-supplied
timestamp of when it was last updated)
• Super Column Family
Container for super columns sorted by their
names
• Super Column
Structure with name and set of dependent
columns
Column Family\Column
Column
A name value pair (contains also a time-stamp for conflict resolution
on the server side)
column
name : byte[]
column
value : byte[]
+ timestamp : long
Column Family
A container for columns sorted by their names. Column Families are referenced and
sorted by row keys.
row key
column
name 1
column
name n
column
value 1
column
value n
Super Column Family\Super Column
Super Column
A sorted associative array of columns.
super column name
column
name 1
column
name n
column
value 1
column
value n
Super Column Family
A container for super columns sorted by their names. Like Column Families, Super Column Families
are referenced and sorted by row keys.
super column name 1
row key
super column name m
column
name 1
column
name n1
column
name 1
column
name nm
column
value 1
column
value n1
column
value 1
column
value nm
Addressing Column Family
row key
column
name 1
column
name n
column
value 1
column
value n
• Four-dimensional hash
• [Keyspace][ColumnFamily][Key][Column]
Addressing Super Column Family
super column name 1
row key
super column name m
column
name 1
column
name n1
column
name 1
column
name nm
column
value 1
column
value n1
column
value 1
column
value nm
• Five-dimensional hash
• [Keyspace][ColumnFamily][Key][SuperColumn][SubColumn]
Cassandra client options
Thrift (12 different languages)
Avro (data serialization system)
Java:
Hector: http://github.com/rantav/hector (abstraction over thrift)
Pelops: http://github.com/s7/scale7-pelops (abstraction over thrift)
CQL: JDBC driver for Cassandra version starting from 0.8 (SQL like language)
Hector JPA: https://github.com/riptano/hector-jpa (ORM client)
Cassandrelle: http://demoiselle.sf.net/component/demoiselle-cassandra/ (documentation ???)
Kundera: http://code.google.com/p/kundera/ (buggy ???)
Python:
Pycassa, Telephus
Grails:
grails-cassandra
.NET:
Aquiles, FluentCassandra
Ruby:
Cassandra
PHP:
phpcassa, SimpleCassie
Cassandra\RDBMS query differences
•
•
•
•
No update query
Record-level atomicity on writes
No duplicate keys
Basic write properties: consistency level
(ZERO, ANY, ONE, QUORUM, ALL)
• Basic read properties: consistency level (ONE,
QUORUM, ALL)
Integrating
Hadoop (http://hadoop.apache.org) is a set of open source projects that deal with
large amounts of data in a distributed way.
• Hadoop Distributed File System (HDFS): a distributed file system that provides
high-throughput access to application data.
• Hadoop MapReduce: a software framework for distributed processing of large
data sets on compute clusters.
Other Hadoop-related projects at Apache include:
• Cassandra™: a scalable multi-master database with no single points of failure.
• Hive™: a data warehouse infrastructure that provides data summarization and ad
hoc querying.
• Mahout™: a Scalable machine learning and data mining library.
• Pig™: a high-level data-flow language and execution framework for parallel
computation.
The end
Questions?