Initial Development & Vision

Matei Zaharia begins developing Spark as a Ph.D. project, driven by the desire to overcome Hadoop MapReduce's limitations for iterative and interactive computations by leveraging in-memory processing and a more flexible API.

Significance

This marked the conceptual birth of Spark, laying the groundwork for its core architecture, the Resilient Distributed Datasets (RDD) abstraction, and its focus on speed and versatility.