Databricks spark_jar_task failed when submitted via API

问题

I am using to submit a sample spark_jar_task

My sample spark_jar_task request to calculate Pi :

"libraries": [
    {
      "jar": "dbfs:/mnt/test-prd-foundational-projects1/spark-examples_2.11-2.4.5.jar"
    }
  ],
  "spark_jar_task": {
    "main_class_name": "org.apache.spark.examples.SparkPi"
  }

Databricks sysout logs where it prints the Pi value as expected

....
(This session will block until Rserve is shut down) Spark package found in SPARK_HOME: /databricks/spark DATABRICKS_STDOUT_END-19fc0fbc-b643-4801-b87c-9d22b9e01cd2-1589148096455 
Executing command, time = 1589148103046. 
Executing command, time = 1589148115170. 
Pi is roughly 3.1370956854784273 
Heap
.....

Spark_jar_task though prints the PI value in log, the job got terminated with failed status without stating the error. Below are response of api /api/2.0/jobs/runs/list/?job_id=23.

{
    "runs": [
        {
            "job_id": 23,
            "run_id": 23,
            "number_in_job": 1,
            "state": {
                "life_cycle_state": "TERMINATED",
                "result_state": "FAILED",
                "state_message": ""
            },
            "task": {
                "spark_jar_task": {
                    "jar_uri": "",
                    "main_class_name": "org.apache.spark.examples.SparkPi",
                    "run_as_repl": true
                }
            },
            "cluster_spec": {
                "new_cluster": {
                    "spark_version": "6.4.x-scala2.11",
......
.......

Why the job failed here? Any suggestions will be appreciated!

EDIT : The errorlog says

20/05/11 18:24:15 INFO ProgressReporter$: Removed result fetcher for 740457789401555410_9000204515761834296_job-34-run-1-action-34
20/05/11 18:24:15 WARN ScalaDriverWrapper: Spark is detected to be down after running a command
20/05/11 18:24:15 WARN ScalaDriverWrapper: Fatal exception (spark down) in ReplId-a46a2-6fb47-361d2
com.databricks.backend.common.rpc.SparkStoppedException: Spark down: 
    at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:493)
    at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:597)
    at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:390)
    at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337)
    at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219)
    at java.lang.Thread.run(Thread.java:748)
20/05/11 18:24:17 INFO ShutdownHookManager: Shutdown hook called