Introduction to Map Reduce with Hadoop (I)

I thought of compiling some of the basics concepts related to Map Reduce and Hadoop along with a trivial sample to get started with Hadoop. Also, I would like to discuss about the new MapReduce API of Hadoop and samples will be based on the new API.

Why Map Reduce?

Nowadays, we are surrounded by huge amount of data and each one of us keep on consuming and generating data at every second. Facebook, YouTube, Twitter, LinkedIn, Googling and every other thing that we do on the internet is dealing with a huge amount of data. The main challenge here is to analyze this sort of huge volumes of data and make decisions based on the analysis. Google was the first initiator who came up with an abstraction called Map Reduce to address the challenges in parallel processing of high volume of data.

Fundamentals of Map Reduce

MapReduce is a programming model and an associated implementation for processing and generating large data sets. It was originally developed by Google (MapReduce: Simplified Data Processing on Large Clusters) and built on well-known principles in parallel and distributed processing. Since then Map Reduce was extensively adopted for analyzing large data sets in its open source flavor Hadoop.




The original motivation behind Map Reduce aroused with the requirements of Google for processing large amount of raw data such as crawled documents, web requests logs etc and building up inverted index or graph representations. Despite those computations are not so complex, owing to the high volume of data, the computations has to be distributed across multiple machines (normally hundreds or thousands) along with the support for parallelization, fault-tolerance, data distribution and load balancing.

Map Reduce provide a new abstraction for all type of high volume data processing with allows the users to express the simple computations we were trying to perform but hides the messy details of parallelization, fault-tolerance, data distribution and load balancing in a library.

In most computation related to high data volumes, it is observed that two main phases are commonly used in most data processing components. The original authors of Map Reduce spotted this commonality and created an abstraction phases of Map Reduce model called 'mappers' and 'reducers' (Original idea was inspired from programming languages such as Lisp).

When it comes to processing large data sets, for each logical record in the input data it is often required to use a mapping function to create intermediate key value pairs . Then another phase called 'reduce' to be applied to the data that shares the same key, to derive the combined data appropriately.

Mapper

The mapper is applied to every input key-value pair (split across an arbitrary number of files) to generate an arbitrary number of intermediate key-value pairs. The standard representation of this is as follows:


map: (k1 , v1 ) → [(k2 , v2 )]

So, if we take an example of processing a large set of text files for obtaining word frequencies (word count), then the input of map function will be the file_name and the file_content which is denoted by k1 and v1. So, with in the map function user may emit the any arbitrary key/value pair as denoted in the list [k2, v2]. 

eg: Mapper for Word Count sample
Loading ....


Reducer

The reducer is applied to all values associated with the same intermediate key to generate output key-value pairs. It is important to keep in mind that, in between map and reduce jobs there exists an implicit distributed "group by" operation on intermediate keys and intermediate data arrive at each reducer
in order, sorted by the key.

Since we have an intermediate 'group by' operation, the input to the reducer function is a key value pair where the key-k2 is the one which is emitted from mapper and a list of values [v2] with shares the same key.

reduce: (k2 , [v2 ]) → [(k3 , v3 )]

eg: Reducer for Word Count sample

Loading ....



WordCount with Map Reduce


So, as we have covered all the fundamentals related to Map Reduce, its time to start writing our simple map reduce program to count words frequencies of a given data set.

images courtsey of : http://dme.rwth-aachen.de/de/research/projects/mapreduce
http://technorati.com/technology/article/massively-parallel-processing-of-big-data1/
http://www.rabidgremlin.com/data20/

Comentários

Postar um comentário

Postagens mais visitadas deste blog

Understanding Reactor Pattern with Java NIO

Imagem
Why non-blocking IO A typical server application, such as a web server, needs to process thousands of request concurrently. Therefore, the modern web server needs to meet the following requirements. Handling of thousands of connections simultaneously (significant number of connections may be in idle state as well) Handling high latency connections Request/response handling needs to be decoupled Minimize latency, maximize throughput and avoiding unnecessary CPU cycles Hence, to cater to such requirements, there can be several possibilities in the server application architecture. Having a pool of sockets for each client, and periodically polling them: This is the most straightforward approach; however, it is more or less impractical without non-blocking sockets. This is extremely inefficient and never scales with the increasing number of connections. Thread per socket: This is the conventional approach and was initially used in some applications and this is the only practical solution wi...
Leia mais

WSO2 ESB 4.5.0 Released !!!

Imagem
We have just released WSO2 ESB 4.5.0, the High Performance Open Source ESB,which is the first ESB version based on enhanced WSO2 Carbon 4 platform.  I would like to give an high level overview of WSO2 ESB 4.5.0's new features and improvements.  XPath 2.0 and XSLT 2.0 support In most of the integration scenarios, we need to leverage the new features of XPath 2.0 and XSLT 2.0. WSO2 ESB 4.5.0,  comes with XPath 2.0 and XSLT 2.0 support. Also, we did a major performance enhancement with the introduction of Saxon as the deafult XSLT/XPath engine.   JSON  Integration is all about handling diverse message types. In that context, JSON plays a major role as numerous APIs are exposed via JSON.  In ESB 4.5.0, we introduce a brand new JSON implementation, which can be used to Streaming JSON (JSONStreamFormatter and JSONStreamBuilder) messages through ESB. In case, if we need to mediate message based on the message content (rather than streaming), th...
Leia mais

The Romantic Opera – Kasun Kalhara

Imagem
Nowadays Sri Lankan music stream is flooded with a lot RnB and Hiphop music and many people merely want to listen to such styles of music. And all the radio stations are backing such music and ruining the natural music taste of Sri Lankans. All those commercial oriented artists tend to release their 5-6 tracks per a month. This is something simply related to the quantity and they simply want to create a music track using computers and other equipments. Once they create the track there are people who can write some strange words to tally with the track and the beat. Finally they’ll come up with a music video and all the TV stations eagerly waiting to play their music video.This is the story of Iraj, BnS, .. (you name it) But .. They have forgotten the fact that there are some (I think its few ) people in Sri Lanka who still loves the instinctive slow music (mood songs) which is a blend of soothing voices and acoustic music. Kasun Kalhara is one of the exceptions thrown from the Sri Lan...
Leia mais