Apache spark join with dynamic re-partitionion

Question

I\'m trying to do a fairly straightforward join on two tables, nothing complicated. Load both tables, do a join and update columns but it keeps throwing an exception.

I

Answer 1

Your values for partitionColumn, upperBound, and lowerBound could cause this exact behavior if they aren't set correctly. For instance, if lowerBound == upperBound, then all of the data would be loaded into a single partition, regardless of numPartitions.

The combination of these attributes determines which (or how many) records get loaded into your DataFrame partitions from your SQL database.