Source: dzone.com
Dynamic Partition Pruning in Spark 3.0Category: Database, Data
With the release of Spark 3.0, big improvements were implemented to enable Spark to execute faster and there came many new features along with it.
Before diving into the features which are new in Dynamic Partition Pruning let us understand what is Partition Pruning.
For example, In this simple query, we are trying to match and identify records in the Students table that belong to subject English.
So in this case, we can no longer apply static partition pruning because the filter is on one side of the join, and the table that is more appealing and more attractive to prune is on the other side of the join.
And in this way we can figure out that when we are planning the fact side of the join.
Before diving into the features which are new in Dynamic Partition Pruning let us understand what is Partition Pruning.
For example, In this simple query, we are trying to match and identify records in the Students table that belong to subject English.
So in this case, we can no longer apply static partition pruning because the filter is on one side of the join, and the table that is more appealing and more attractive to prune is on the other side of the join.
And in this way we can figure out that when we are planning the fact side of the join.
Related Articles
Community Partners
DevOps Careers