HDFS Docker

HDFS in a Docker container can be a useful way to set up a Hadoop environment for development, testing, or experimentation without the need to install it directly on your host machine. Here are the basic steps to run HDFS in Docker:

Note: Running a full-fledged Hadoop cluster with HDFS in Docker for production purposes can be complex and may require additional configuration and considerations. The following steps are a simplified way to get started with a single-node HDFS Docker container.

1. Install Docker: Ensure that you have Docker installed on your system. You can download and install Docker from the official website: Docker.

2. Pull Hadoop Docker Image: There are several Hadoop Docker images available on Docker Hub. You can pull an official image or one maintained by the community. For example, you can pull the sequenceiq/hadoop-docker image using the following command:

   bashCopy code

   docker pull sequenceiq/hadoop-docker:latest

3. Run HDFS Container: Once you have the Hadoop image, you can run a Docker container with HDFS. You can specify the ports you want to expose and mount local directories as volumes. The following command runs a single-node HDFS container:

   bashCopy code

   docker run -it -p 50070:50070 -p 8088:8088 -v /path/on/host:/usr/local/hadoop/input sequenceiq/hadoop-docker /etc/bootstrap.sh -bash
   • -it: Runs the container in interactive mode with a terminal.
   • -p: Maps host ports to container ports (e.g., 50070 for HDFS web UI, 8088 for YARN web UI).
   • -v: Mounts a host directory as a volume inside the container (for data storage).
   • /etc/bootstrap.sh -bash: This is the command to start the HDFS container.

4. Hadoop Commands: Once the container is running, you can access the Hadoop command-line tools and interact with the HDFS filesystem. For example:

   • To list files in HDFS: hadoop fs -ls /
   • To copy a file to HDFS: hadoop fs -copyFromLocal /local/path /hdfs/destination
   • To run a Hadoop job: hadoop jar /path/to/hadoop-example.jar input-dir output-dir

5. Web Interfaces: You can access Hadoop’s web interfaces, such as the HDFS Namenode UI and YARN Resource Manager UI, by opening a web browser and going to the respective ports you exposed in the Docker run command (e.g., http://localhost:50070 for HDFS and http://localhost:8088 for YARN).

6. Cleanup: When you’re done experimenting with the container, you can stop and remove it using Docker commands:

   • To stop the container: docker stop <container-id>
   • To remove the container: docker rm <container-id>