How to program MapReduce jobs in Hadoop with R Group 8 João Rosa, Mario Almeida, Alex Pérez Index ● ● ● ● ● ● ● ● ● Introduction Hadoop MapReduce R Why and how? Possible uses? Business opportunities? Conclusion Questions References Nowadays, we have lots of data. BIG DATA! If we need to analyse this we have a problem... ...but, if we need to analyse this we have a BIG DATA problem! How can we analyse this BIG DATA? + A possible solution! The Apache™ Hadoop™ project develops open-source software for reliable, scalable, distributed computing. The project includes these subprojects: Hadoop Common is a set of utilities that support the Hadoop subprojects. Hadoop Common includes FileSystem, RPC, and serialization libraries. The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. ● ● ● ● ● Highly fault-tolerant with hardware Failure Designed to be deployed on low-cost hardware Streaming Data Access Large Data Sets Portability Across Heterogeneous Hardware and Software Platforms Supports distributed computing on large data sets on clusters of computers Process large amounts of raw data Map + Reduce R is the language of Pirates!!! Rrrrr What is R? It's a language and environment for statistical computing and graphics! What is R? 2 million analysts! Quantitative finance! Google, Facebook and LinkedIn! Why R? ● Current analytic solutions are costly! ● New methods for analyzing complex datasets! Why Hadoop with R? "Easiest, most productive, most elegant way to write map reduce jobs." Why Hadoop with R? ● One-two orders of magnitude less code than Java Why Hadoop with R? Readable, reusable and extensible language. Why Hadoop with R? To give R analysts a way to access the mapreduce programming paradigm using big data sets. How to use Hadoop with R? RHadoop = rmr + rHDFS + rHBase ● rhdfs - functions providing file management of the HDFS from within R (RJava). ● rhbase - functions providing database management for the HBase distributed database from within R (Thrift). ● rmr - functions providing Hadoop MapReduce functionality in R. Business opportunities? xkcd.com Conclusions Productivity vs Efficiency Wide variety of statistical and graphical techniques Business orientation Questions? References http://hadoop.apache.org/ - Apache Hadoop's project http://www.r-bloggers.com/how-to-program-mapreduce-jobs-in-hadoop-with-r/ - teachers page http://static.usenix.org/event/osdi04/tech/full_papers/dean/dean.pdf - MapReduce https://github.com/RevolutionAnalytics/RHadoop/wiki/Tutorial - MapReduce in R tutorial http://www.inside-r.org/r-doc/base/lapply - R lapply http://www.inside-r.org/r-doc/base/tapply - R tapply http://www.revolutionanalytics.com/what-is-open-source-r/ - What is R? http://www.r-project.org/ - What is R? Official page http://en.wikipedia.org/wiki/R_(programming_language) - R wiki http://www.johndcook.com/R_language_for_programmers.html - R programming for those coming from other languages http://www.revolutionanalytics.com/why-revolution-r/whitepapers/r-is-hot.php- why are R is hot Pictures We tried to use CC pictures, bellow are their respective links: http://www.flickr.com/photos/nanagyei/4880468290 - pig pirates http://www.flickr.com/photos/timypenburg/5328226108 - maths and pen http://www.flickr.com/photos/48481327 - graduate http://s0.geograph.org.uk/geophotos/01/53/43/1534341_7dc47500.jpg - store http://www.flickr.com/photos/dizfunk/3066153143/ - nerd http://geekithawaii.com/wp-content/uploads/2011/01/7562581_l.jpg - sky http://www.robweir.com/blog/wp-content/uploads/2011/01/numbers.jpg - numbers http://delightfulchildrensbooks.files.wordpress.com/2011/02/read-around-the-world.jpg - children Others: http://www.xkcd.com