Apache Hadoop YARN
Apache Hadoop YARN (Yet Another Resource Negotiator) is the resource management layer of the Hadoop ecosystem, introduced with Hadoop 2.x. It is responsible for allocating computational resources (CPU, memory) to applications running on a Hadoop cluster and managing their lifecycle. YARN…
Key facts
- First appeared
- 2012
- Category
- technology
- Problem solved
- Apache Hadoop YARN was created to address the significant limitations of Hadoop 1.x's MapReduce architecture, where the JobTracker was a monolithic component responsible for both resource management and job scheduling. This design created a single point of failure, scalability bottlenecks, and severely limited the ability to run diverse data processing frameworks (beyond just MapReduce) on the same shared Hadoop cluster efficiently and securely.
- Platforms
- Cloud environments (e.g., AWS, Azure, GCP), Linux (distributed clusters)
Related technologies
Notable users
- Many large enterprises leveraging Hadoop
- Facebook (Meta)
- Microsoft Azure (HDInsight)
- Google Cloud (Dataproc)
- Hortonworks (Cloudera)
- Amazon Web Services (EMR)
- Netflix
- Yahoo