One of the major advantage of using Hadoop is its ability to handle failures and allow jobs to complete successfully. In this article we are going to discuss about the different types of Failures that can occur in Hadoop and how they are handled. Let us begin with the HDFS…

In Hadoop 2, Map Reduce jobs are executed using the YARN(Yet Another Resource Negotiator). Let us understand the different id’s that are created while executing a mapreduce application. Application Id: When a MR job is submitted by the client, the resource manager will first create the application ID. Application ID…

In real world the clusters are busy and the resources are limited, as a result the applications often need to wait to have some of its resources fulfilled. The YARN scheduler takes the responsibility of allocating the resources to applications based on some defined policies. …

Before we start our discussion on what exactly is Erasure coding, let us understand the below two terms and see how HDFS achieve them. Durability: How many simultaneous failure can be tolerated ? It is also known as fault tolerance. Storage Efficiency: How much portion of storage is useful for…

Hadoop Distributed File System (HDFS) is file system of Hadoop designed for storing very large files running on clusters of commodity hardware. Generally, when dataset outgrows the storage capacity of a single machine, it is necessary to partition it across number of separate machines. …

In this blog we are going to discuss about how to integrate Apache Kafka with Spark using Python and its required configuration. How Kafka works ? Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. …

Introduction to Parquet Apache Parquet is a columnar open source storage format that can efficiently store nested data which is widely used in Hadoop and Spark. Initially developed by Twitter and Cloudera. Columnar formats are attractive since they enable greater efficiency, in terms of both file size and query performance…