Orc Hadoop

ORC (Optimized Row Columnar) is a file format used in the Hadoop ecosystem for storing structured data efficiently. ORC is designed to provide high performance and space efficiency for processing large datasets in Hadoop-based applications. It is one of the several file formats commonly used alongside Apache Hadoop and Hive. Here are some key points about ORC in Hadoop:

1. Columnar Storage: ORC stores data in a columnar format, which means that data in each column is stored together. This allows for better compression and more efficient data processing because it enables the reading of only the necessary columns, reducing I/O and improving query performance.

2. Compression: ORC uses lightweight compression techniques to reduce storage space. It offers various compression algorithms, such as Zlib, Snappy, and Lz4, to choose from based on your needs.

3. Predicate Pushdown: ORC supports predicate pushdown, which means that it can skip unnecessary data during query execution, further improving query performance.

4. Type Evolution: ORC allows schema evolution, meaning you can add or modify columns to the schema without requiring a full data migration. This makes it flexible for changing data requirements over time.

5. Compatibility: ORC files can be used with various components in the Hadoop ecosystem, including Apache Hive, Apache Pig, and Apache Spark, making it a versatile choice for big data processing.

6. Hive Integration: ORC is often used in conjunction with Apache Hive, a data warehouse infrastructure built on top of Hadoop. Hive can create, read, and write ORC files, and it can use ORC as a storage format for optimized query performance.