问题
I am running spark in cluster mode and reading data from RDBMS via JDBC.
As per Spark docs, these partitioning parameters describe how to partition the table when reading in parallel from multiple workers:
partitionColumnlowerBoundupperBoundnumPartitions
These are optional parameters.
What would happen if I don\'t specify these:
- Only 1 worker read the whole data?
- If it still reads parallelly, how does it partition data?
回答1:
If you don't specify either {partitionColumn, lowerBound, upperBound, numPartitions} or {predicates} Spark will use a single executor and create a single non-empty partition. All data will be processed using a single transaction and reads will be neither distributed nor parallelized.
See also:
- How to optimize partitioning when migrating data from JDBC source?
- How to improve performance for slow Spark jobs using DataFrame and JDBC connection?
来源：https://stackoverflow.com/questions/43150694/partitioning-in-spark-while-reading-from-rdbms-via-jdbc