Hadoop FS

In the context of Apache Hadoop, “Hadoop FS” typically refers to the Hadoop FileSystem command-line interface (CLI) or commands used to interact with Hadoop Distributed File System (HDFS). HDFS is the primary storage system used by Hadoop to store and manage data across a distributed cluster. Hadoop FS commands allow users to perform various file and directory operations on HDFS from the command line. Here are some common Hadoop FS commands:

1. List Files and Directories:

   • hadoop fs -ls [path]: List files and directories at the specified HDFS path.

2. Create a Directory:

   • hadoop fs -mkdir [path]: Create a new directory in HDFS.

3. Copy Files/Directories to HDFS:

   • hadoop fs -copyFromLocal [local-source] [hdfs-destination]: Copy a file or directory from the local file system to HDFS.
   • hadoop fs -put [local-source] [hdfs-destination]: Another command to copy files or directories to HDFS.

4. Copy Files/Directories from HDFS to Local:

   • hadoop fs -copyToLocal [hdfs-source] [local-destination]: Copy a file or directory from HDFS to the local file system.
   • hadoop fs -get [hdfs-source] [local-destination]: Another command to copy files or directories from HDFS to the local file system.

5. Move or Rename Files/Directories:

   • hadoop fs -mv [source] [destination]: Move or rename a file or directory in HDFS.

6. Delete Files/Directories:

   • hadoop fs -rm [path]: Delete a file or directory in HDFS.
   • hadoop fs -rm -r [path]: Recursively delete a directory and its contents.

7. Check File/Directory Existence:

   • hadoop fs -test -e [path]: Check if a file or directory exists in HDFS.
   • hadoop fs -test -d [path]: Check if a directory exists in HDFS.

8. Change Permissions and Ownership:

   • hadoop fs -chmod [permissions] [path]: Change the permissions of a file or directory in HDFS.
   • hadoop fs -chown [owner]:[group] [path]: Change the ownership of a file or directory in HDFS.

9. View File Content:

   • hadoop fs -cat [file]: View the content of a file in HDFS.
   • hadoop fs -tail [file]: Display the last few lines of a file in HDFS.

10. File Count and Size:

    • hadoop fs -count [path]: Count the number of files and directories in HDFS and their total size.

11. Set Replication Factor:

    • hadoop fs -setrep -w [replication-factor] [path]: Set the replication factor for a file or directory in HDFS.

12. Copy Data Within HDFS:

    • hadoop fs -cp [source] [destination]: Copy data within HDFS from one location to another.