What happens if I try to use more cores than I have?

Question

In my sparkconf, i can set the number of cores to use, i have 4 physical, 8 logical on my laptop, what does spark do if I specify a number that was not possible on the machi

Answer 1

It heavily depends on your cluster manager. I assume that you're asking about local[n] run mode.

If so, the driver and the one and only one executor are the same JVM with n number of threads.

DAGScheduler - the Spark execution planner will use n threads to schedule as many tasks as you've told it should.

If you have more tasks, i.e. threads, than cores, your OS will have to deal with more threads than cores and schedule them appropriately.