Design Patterns for LargeScale Data Management Robert Hodges OSCON 2013 ©Continuent 2013 The Start-Up Dilemma 1. You are releasing Online Storefront V 1.0 2. It could be a complete bust 3. But it could be *really* big How do you implement data management at hacking velocity without compromising the future? ©Continuent 2013 2 The Answer: Data Fabric Design App Tier App Server App Server App Server High Availability Large Data Volumes Data Tier Run Anywhere Cross-Site Distribution Polyglot DBMS Data Fabric ©Continuent 2013 3 Technology Upgrades Use App Design Principles for Data Client Interface(s) Responsibilities Capabilities DBMS Server ©Continuent 2013 Data Service 4 Compose Services via Design Patterns Connector Connector Connector Fabric Connector Transactional Data Service Bridge Fault-Tolerant Data Service Real-Time Data Bridge Sharded Data Service Multi-Site Data Service ©Continuent 2013 5 1. Transactional Data Service • Responsibility: Reliable and accessible data storage • Capabilities: ©Continuent 2013 • • • • Well-de!ned client interface Transactions and recovery Backup and restore Log for replication 6 MySQL: A Mature Building Block Relational modeling Single join type Sales Table id cust_id prod_id ... 1 335301 532 ... 2 2378 6235 ... 3 ... ... ... Prod_ID Index Product Table id sku type prod_id id 532 533 ... C00135 S09957 ... consumer specialty 532 6235 ... 1 2 ... Indexes speed query, slow updates Single-threaded query Transaction support ©Continuent 2013 Fast row operations, slow scans of columns 7 MySQL Implementation SQL Interface (MySQL Dialect) MySQL Wire Protocol MySQL 5.6/InnoDB Percona XtraBackup MySQL Binlog ©Continuent 2013 8 Backup Images Comments and Trade-O!s • • • • ©Continuent 2013 Interface need not be SQL Backups are critical to restore broken replicas Pay close attention to DBMS con"guration Design Key: Use ACID transactions (No MyISAM!) 9 2. Fault-Tolerant Data Service • Responsibility: Continously available data using shared-nothing replicas • Capabilities: ©Continuent 2013 • • • Real-time replication between replicas Provision/reprovision from backups Failover and rolling maintenance 10 Replication Technology Overview • • • • Synchronous vs. asynchronous Master/slave vs. multi-master Logical vs. binary Trigger vs. log-based Master/ Slave ©Continuent 2013 MultiMaster 11 Implementation using MySQL & MHA Master MHA Manager Daemon Oracle MySQL 5.6 (Performs failover and switch) Async Native Replication Oracle MySQL 5.6 Oracle MySQL 5.6 Slave Slave ©Continuent 2013 12 Comments and Trade-O!s • • • • ©Continuent 2013 Design key: Pick the right replication method Capabilities vary greatly Failover is hard to implement from scratch Design key: Address rolling maintenance 13 3. Fabric Connector • Responsibility: Connect applications transparently to replicas • Capabilities: ©Continuent 2013 • • • • Encapsulation of DBMS server location Transparent request proxying Noti!cation protocol for failover/switch operations Load balancing for performance 14 Lots of Places to Intercept Requests Application Logic Access Library DBMS driver Proxy Driver Client proxy Client Wire Protocol Level 7 router Virtual IP Address DNS name switching Proxy DBMS server (to other DBMS processes) DBMS Server ©Continuent 2013 Access Library Wrapper 15 Quorum and Split Brain App Server App Server Master/Online ©Continuent 2013 App Server Slave/Online 16 App Server Slave/Offline Library Wrapper + ZooKeeper Application Logic ZooKeeper Cluster Access Library Wrapper zk1 Java ORM MySQL JDBC Driver zk2 zk3 DBMS locations stored in ZK directory; failover through ZK event notifications Fault Tolerant Data Service ©Continuent 2013 17 Comments and Trade-O!s • VIPs and DNS renaming are easy but treacherous • • Wire protocol translation is hard • Design key: balance between availability and split brain ©Continuent 2013 Wrapper libraries are a good Goldilocks solution 18 4. Sharded Data Service • Responsibility: Partition a large dataset into manageable subsets • Capabilities: ©Continuent 2013 • • • • “Buckets” to contain groups of shards Partitioning function to locate shards Provisioning and shard migration Intra- and cross-shard query 19 Easy vs. Hard Sharding Problems Independent, small subsets ©Continuent 2013 Very large subsets with dependent data 20 Sharding Is Easy when It’s Easy Application Logic ZooKeeper Cluster Access Library Wrapper zk1 Java ORM MySQL JDBC Driver zk2 zk3 DBMS locations stored in ZK directory; failover through ZK event notifications Sharded Data Service ©Continuent 2013 21 Comments and Trade-O!s • Design key: factor data model to enable sharding • • • • Sharding scheme must be simple to work ©Continuent 2013 Moving shards is di#cult Push cross-shard queries into data warehouse Some problems really need Cassandra or HBase, not a SQL DBMS 22 5. Multi-Site Data Service • Responsibility: Spread data across multiple geographic locations • Capabilities: ©Continuent 2013 • • • Fast cross-site replication Primary/DR topology for business continuity Master/master topology to push writes closer to users 23 Multi-Master vs. Primary/DR Amazon East EU West US East APAC Tokyo Rackspace DFW Write-anywhere model No writes on DR sites Applications w/ few con!cts Any application Simpler failover model Easier to set up and manage Problems with constraints, large transactions Does not a"ect SQL features ©Continuent 2013 24 Tungsten Primary/DR Service App Server + Connector App Server + Connector San Jose NYC Slave Master ©Continuent 2013 25 Comments and Trade-O!s • • Primary/DR is robust, easy to add later • Design key: Keep data model simple to enable multi-master • SQL DBMS do not support multi-master as well as NoSQL (cf. Riak, Cassandra, HBase,...) ©Continuent 2013 Multi-master is a better model if you can make it work 26 6. Real-Time Data Bridge • Responsibility: Transfer data in real-time between di!erent DBMS types • Capabilities: ©Continuent 2013 • • • • Heterogeneous replication [Near] Real-time High performance without app changes E#cient conversion between storage models 27 Why Polyglot Persistence? Online Storefront Application User Session Service Web Sales Service Sales Analytics Service Key-value store with serialized user sessions Row store with strong transactional support Column store w/ compression + time series analysis ©Continuent 2013 28 Implementing Real-Time Data Bridge Sharded Data Service Tungsten Master Replication Services Tungsten Slave Replication Services oltp1 oltp1 oltp2 oltp2 oltp3 Parallel loading via CSV files into Vertica oltp3 Serial extraction from MySQL binlog ©Continuent 2013 29 Comments and Trade-O!s • Heterogeneous replication is time-consuming to deploy • • • Can apply the pattern after the fact ©Continuent 2013 Avoid transformation as much as possible Design key: Balance complexity of data movement vs. getting data into the right silo for processing 30 Data Fabric Design Pattern Recap Connector Connector Connector Fabric Connector Transactional Data Service Bridge Fault-Tolerant Data Service Real-Time Data Bridge Sharded Data Service Multi-Site Data Service ©Continuent 2013 31 Conclusion • Use application architecture principles to create data services • Use design patterns to combine services into data fabrics • • Apply patterns incrementally ©Continuent 2013 Early focus on design keys pays o! big in the long run 32 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: [email protected] Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://www.continuent.com/news/blogs Continuent Web Page: http://www.continuent.com Tungsten Replicator: http://code.google.com/p/tungsten-replicator ©Continuent 2013
© Copyright 2024