Apache Livy
Apache Livy is an open-source REST service designed to enable simple interaction with Apache Spark clusters from anywhere. It allows users to submit Spark jobs or code, manage long-running Spark contexts, and access Spark's results via a programmatic interface, decoupling client applications…
Key facts
- First appeared
- 2015
- Category
- technology
- Problem solved
- Apache Livy addresses the challenge of providing a standardized, secure, and multi-tenant REST interface for remote interaction with Apache Spark clusters. Before Livy, developers often resorted to custom RPC solutions, embedding Spark code directly, or relying solely on `spark-submit` for job submission, which lacked interactive session management and easy programmatic access from external applications.
- Platforms
- Kubernetes, Apache Hadoop YARN, Windows (via Docker/VM or WSL), Apache Mesos, Linux, macOS
Related technologies
Notable users
- Cloudera
- Google Cloud Platform (via Dataproc, indirectly)
- Microsoft Azure (via HDInsight)
- Amazon Web Services (via EMR, indirectly)
- Enterprises running managed Spark clusters
- Hortonworks (merged with Cloudera)