Dask
Dask is an open-source Python library designed to natively scale the Python scientific stack (like NumPy, Pandas, and Scikit-learn) to handle datasets larger than memory or to execute computations across distributed clusters. It achieves parallel computing through dynamic task scheduling,…
Key facts
- First appeared
- 2014
- Category
- technology
- Problem solved
- Dask was created to address the 'memory wall' and 'single-core wall' faced by Python data scientists. While libraries like NumPy and Pandas excelled for in-memory, single-machine computations, they struggled when datasets grew beyond RAM or when extensive parallel processing was required, forcing users to either downsample data, rewrite code in JVM-based languages (like Scala for Spark), or manage complex custom parallel scripts.
- Platforms
- macOS, Linux, Cloud computing environments (AWS, Azure, GCP), Kubernetes, HPC clusters (via schedulers like SLURM, PBS), Windows
Related technologies
Notable users
- Capital One
- Pangeo Project
- Saturn Cloud
- Los Alamos National Laboratory
- Coiled
- NVIDIA