Hadoop Getting Started

 Here’s a step-by-step guide to help you get started with Hadoop:

Step 1: Prerequisites

Before you begin, ensure that you have the following prerequisites in place:

• A Linux-based system (such as Ubuntu or CentOS) or access to a virtual machine with Linux.
• Java Development Kit (JDK) installed (preferably Oracle JDK 8 or later).
• A user account with administrative privileges (for setting up Hadoop).

Step 2: Download and Install Hadoop

1. Visit the Apache Hadoop official website: https://hadoop.apache.org.
2. Navigate to the “Downloads” section and choose the latest stable release.
3. Download the Hadoop distribution (e.g., “hadoop-x.y.z.tar.gz”).
4. Extract the downloaded archive to a directory of your choice. You can use the following command:
   
   shellCopy code

   tar -xzvf hadoop-x.y.z.tar.gz


Step 3: Configure Hadoop

1. Navigate to the Hadoop installation directory:

   shellCopy code

   cd hadoop-x.y.z


2. Edit the Hadoop configuration files located in the etc/hadoop directory. Key configuration files include:

   • core-site.xml: Hadoop core configuration.
   • hdfs-site.xml: Hadoop Distributed File System (HDFS) configuration.
   • yarn-site.xml: YARN (Yet Another Resource Negotiator) configuration.

3. Set Hadoop environment variables by adding the following lines to your shell profile (e.g., .bashrc or .zshrc):

   shellCopy code

   export HADOOP_HOME=/path/to/hadoop-x.y.z
export PATH=$PATH:$HADOOP_HOME/bin


4. Save the changes and reload your shell profile:

   shellCopy code

   source ~/.bashrc


Step 4: Start Hadoop

1. Format the HDFS filesystem (do this only once after installing Hadoop):

   shellCopy code

   hdfs namenode -format


2. Start the HDFS service:

   shellCopy code

   start-dfs.sh


3. Start the YARN service (Resource Manager and Node Manager):

   shellCopy code

   start-yarn.sh


Step 5: Verify Hadoop Installation

• Open a web browser and visit the Hadoop web interfaces to verify that Hadoop is running:
  • NameNode web interface: http://localhost:50070
  • Resource Manager web interface: http://localhost:8088

Step 6: Run a Hadoop Example

To test your Hadoop installation, you can run a simple MapReduce example. Hadoop comes with several built-in examples.

1. Navigate to the Hadoop examples directory:

   shellCopy code

   cd $HADOOP_HOME
cd share/hadoop/mapreduce


2. Run a MapReduce example, such as the word count example:

   shellCopy code

   hadoop jar hadoop-mapreduce-examples-x.y.z.jar wordcount input output


Step 7: Explore Further

Now that you have Hadoop up and running, you can explore more advanced features, such as:

• Writing your MapReduce programs.
• Working with Hadoop’s distributed file system (HDFS).
• Configuring resource management and job scheduling with YARN.
• Integrating Hadoop with other big data tools like Hive, Pig, or Spark.