Hadoop FS Copy

Hadoop FS Copy is a command used to copy files and directories between Hadoop Distributed File System (HDFS) locations. Hadoop provides a command-line interface called hadoop fs that allows you to interact with HDFS and perform various file and directory operations, including copying data. The copyFromLocal and copyToLocal commands are commonly used for copying data to and from HDFS.

Here’s how you can use hadoop fs to copy data in HDFS:

1. Copy from Local to HDFS (copyFromLocal):

   • The copyFromLocal command is used to copy files or directories from your local file system into HDFS.
   • The basic syntax is:
   shellCopy code

   hadoop fs -copyFromLocal <local-source> <hdfs-destination>

   For example, to copy a local file named localfile.txt to an HDFS directory named /user/hadoop/hdfsdir, you would use:

   shellCopy code

   hadoop fs -copyFromLocal localfile.txt /user/hadoop/hdfsdir

2. Copy from HDFS to Local (copyToLocal):

   • The copyToLocal command is used to copy files or directories from HDFS to your local file system.
   • The basic syntax is:
   shellCopy code

   hadoop fs -copyToLocal <hdfs-source> <local-destination>

   For example, to copy a file named hdfspath/file.txt from HDFS to your local directory /localpath, you would use:

   shellCopy code

   hadoop fs -copyToLocal hdfspath/file.txt /localpath

3. Recursive Copy:

   • You can use the -r or --recursive option to perform a recursive copy of directories and their contents.
   shellCopy code

   hadoop fs -copyFromLocal -r localdir /user/hadoop/hdfsdir

   This command copies the entire contents of the local directory localdir to the HDFS directory /user/hadoop/hdfsdir.

4. Overwriting Existing Files:

   • By default, if a file with the same name already exists in the destination location, Hadoop will not overwrite it.
   • To force overwriting, you can use the -f option.
   shellCopy code

   hadoop fs -copyFromLocal -f localfile.txt /user/hadoop/hdfsdir