Apache Spark
Apache Spark is a unified analytics engine for large-scale data processing, offering high performance for both batch and streaming data. It extends the MapReduce model to efficiently support more types of computations, including interactive queries, streaming analytics, machine learning, and…
Key facts
- First appeared
- 2010
- Category
- technology
- Problem solved
- Apache Spark was created to address the limitations of Apache Hadoop MapReduce, particularly its inefficiency for iterative algorithms (e.g., machine learning, graph processing) and interactive queries, which required repeated disk I/O between stages. MapReduce's batch-only processing model also struggled with real-time data streams and complex workflows that needed to chain multiple operations with shared data.
- Platforms
- Kubernetes, Linux, Apache Mesos, Cloud platforms (AWS EMR, Azure HDInsight/Synapse Analytics, Google Cloud Dataproc), macOS, Windows, Standalone cluster manager, Apache Hadoop YARN
Related technologies
Notable users
- Uber
- Netflix
- Capital One
- Microsoft
- Amazon
- Walmart
- Alibaba
- Adobe