How to use Spark-Scala to download a CSV file from the web?

Question

world,

How to use Spark-Scala to download a CSV file from the web and load the file into a spark-csv DataFrame?

Currently I depend on curl in a shell command

Answer 1

Found better answer from Process CSV from REST API into Spark

Here you go:

import scala.io.Source._
import org.apache.spark.sql.{Dataset, SparkSession}

var res = fromURL(url).mkString.stripMargin.lines.toList
val csvData: Dataset[String] = spark.sparkContext.parallelize(res).toDS()

val frame = spark.read.option("header", true).option("inferSchema",true).csv(csvData)
frame.printSchema()