Hadoop MR

Hadoop MapReduce (Hadoop MR) is a programming model and processing framework used for processing and generating large datasets in parallel across a distributed cluster of commodity hardware. It is a core component of the Apache Hadoop ecosystem and is designed to handle large-scale data processing tasks efficiently. Here’s an overview of Hadoop MapReduce:

1. MapReduce Model:

   • Hadoop MapReduce follows a functional programming model where the processing is divided into two main phases: the Map phase and the Reduce phase.
   • The Map phase processes input data and produces intermediate key-value pairs.
   • The Reduce phase takes the intermediate key-value pairs, groups them by key, and performs aggregation or other operations on the values associated with each key.

2. Key Concepts:

   • Mapper: The Mapper is responsible for processing input data and emitting key-value pairs as intermediate outputs. Custom mappers are developed to specify how input data is transformed.
   • Reducer: The Reducer takes the intermediate key-value pairs generated by the Mapper, groups them by key, and performs operations like aggregation, sorting, or filtering on the values.
   • Shuffling and Sorting: Hadoop handles the automatic sorting and shuffling of intermediate data between the Mapper and Reducer tasks, ensuring that data with the same key ends up on the same Reducer.
   • Input and Output Formats: Hadoop supports various input and output formats, allowing data to be read from and written to different sources like HDFS, HBase, or other data stores.

3. Hadoop MapReduce Workflow:

   • Data is typically stored in the Hadoop Distributed File System (HDFS).
   • Users submit MapReduce jobs to the Hadoop cluster, specifying input data, Map and Reduce tasks, and output locations.
   • The Hadoop YARN ResourceManager manages the allocation of resources (CPU and memory) to job tasks across the cluster.
   • Map tasks are executed in parallel across the cluster, processing input data and producing intermediate key-value pairs.
   • Reduce tasks run after the Map tasks, processing the intermediate data and producing the final output.

4. Custom MapReduce Jobs:

   • Users can develop custom Map and Reduce functions in Java to define the specific processing logic for their MapReduce jobs.
   • Hadoop also supports streaming, which allows you to use other programming languages like Python or Ruby for MapReduce tasks.

5. Fault Tolerance:

   • Hadoop MapReduce provides built-in fault tolerance by automatically restarting failed tasks on other cluster nodes.
   • It stores intermediate data on disk, ensuring that even in case of node failures, data can be recovered and the job can continue.