Spark 3.0 Features | Dynamic Partition Pruning (DPP) | Avoid Scanning Irrelevant Data
Spark 3.0 has introduced multiple optimization features. Dynamic Partition Pruning (DPP) is one among them, which is an optimization on Star schema queries(data warehouse architecture model). DPP is implemented using Broadcast hashing technique for passing the subquery results of dimension table to fact table before loading the complete data into memory.
Check this video to know more about DPP feature in Spark 3.0
Medium Blog - / spark-3-0-feature-dynamic-partition-prunin...
Check this video to know more about AQE feature in Spark 3.0
• Spark 3.0 Features | Adaptive Query E...
Content By - Prabhakaran Vijayanagulu [LinkedIn - / prabhakaran-vijayanagulu-248ba2118 ]
Editing By - Sivaraman Ravi [LinkedIn - / sivaraman-ravi-791838114 ]
Facebook Page - https://www.facebook.com/Tech-Island-...
Please SUBSCRIBE to our channel :)
Share your feedback with us.
[email protected]
