getting number of visible nodes in PySpark

Question

I\'m running some operations in PySpark, and recently increased the number of nodes in my configuration (which is on Amazon EMR). However, even though I tripled the number

Answer 1

It should be possible to get the number of nodes in the cluster using this (similar to @Dan's method above, but shorter and works better!).

sc._jsc.sc().getExecutorMemoryStatus().keySet().size()