PySpark-02- Architecture of PySpark

Опубликовано: 26 Январь 2025
на канале: DataEdge Learning
55
5

As of writing this Spark with Python (PySpark) tutorial, Spark supports below cluster managers:
Standalone – a simple cluster manager included with Spark that makes it easy to set up a cluster.
Apache Mesos – Mesons is a Cluster manager that can also run Hadoop MapReduce and PySpark applications.
Hadoop YARN – the resource manager in Hadoop 2. This is mostly used, cluster manager.
Kubernetes – an open-source system for automating deployment, scaling, and management of containerized applications.