问题
I have followed the instructions here to enable the metrics export to Prometheus for spark. In order to enable metrics export not just from the job, but also from master and workers, I have enabled the jmx agent for all of spark driver, master, worker, and executor.
This causes a problem since spark worker and executor are collocated on the same machine and, thus, I need to pass in different jmx ports to them. This is not a problem if I have a 1-1 relationship between spark workers and executors, however, it breaks down in the multiple executors per worker scenario, as there is no way to specify a different port for a specific executor during the spark job submission.
The situation is even worse when the job is submitted in cluster mode, since the driver, worker, and executors are all potentially collocated on the same node.
How have you resolved this problem?
来源：https://stackoverflow.com/questions/48574377/exporting-spark-worker-executor-metrics-to-prometheus-using-jmxagent