Spark HDFS

Apache Spark, a powerful open-source data processing framework, can be used to interact with data stored in Hadoop Distributed File System (HDFS) efficiently. Spark provides various APIs and libraries to read, process, and write data to and from HDFS. Here’s how you can work with Spark and HDFS together:

Reading Data from HDFS: Spark provides several methods for reading data from HDFS:

1. SparkContext.textFile(): You can use the textFile() method to read text files from HDFS. For example:

   scalaCopy code

   val textRDD = sc.textFile("hdfs://<HDFS_MASTER>:<HDFS_PORT>/path/to/your/file.txt")


2. SparkSession.read(): If you’re using Spark’s DataFrame API, you can use the read() method to read various data formats from HDFS, such as Parquet, Avro, JSON, and more. For example:

   scalaCopy code

   val df = spark.read.parquet("hdfs://<HDFS_MASTER>:<HDFS_PORT>/path/to/your/parquetfile")


Writing Data to HDFS: Spark also allows you to write data back to HDFS:

1. RDD.saveAsTextFile(): You can save an RDD as a text file in HDFS using the saveAsTextFile() method. For example:

   scalaCopy code

   textRDD.saveAsTextFile("hdfs://<HDFS_MASTER>:<HDFS_PORT>/path/to/your/output")


2. DataFrame.write(): If you’re working with DataFrames, you can use the write() method to write DataFrames to HDFS in various formats. For example:

   scalaCopy code

   df.write.parquet("hdfs://<HDFS_MASTER>:<HDFS_PORT>/path/to/your/output/parquetfile")


Configuration: To work with HDFS from Spark, you need to set the Hadoop configuration specifying the HDFS location. This typically includes specifying the HDFS master and port.