How to create a source to export metrics from Spark to another sink (Prometheus)?

问题

I am trying to create a source for metrics from my spark application written in Scala to export data to another system, preferable to Prometheus. According to this site from Data bricks I need to create a source that extends the Source trait. However, the Source trait is private[spark] trait Source and my source cannot visualize it. When I create this class I get the error Symbol Source is inaccessible from this place.

package org.sense.spark.util

import org.apache.spark.metrics.source.Source
import com.codahale.metrics.{Counter, Histogram, MetricRegistry}

class MetricSource extends Source {
  override val sourceName: String = "MySource"

  override val metricRegistry: MetricRegistry = new MetricRegistry

  val FOO: Histogram = metricRegistry.histogram(MetricRegistry.name("fooHistory"))
  val FOO_COUNTER: Counter = metricRegistry.counter(MetricRegistry.name("fooCounter"))
}

How can I create my source to export data to Prometheus? I would like to export monitored values from a UDF inside the combineByKey transformation. The values would be latency to aggregate and throughput IN/OUT of this transformation.

This is my build.sbt file in case it is necessary to check the libraries that I am using.

name := "explore-spark"

version := "0.2"

scalaVersion := "2.12.3"

val sparkVersion = "3.0.0"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "com.twitter" %% "algebird-core" % "0.13.7",
  "joda-time" % "joda-time" % "2.5",
  "org.fusesource.mqtt-client" % "mqtt-client" % "1.16"
)

mainClass in(Compile, packageBin) := Some("org.sense.spark.app.App")
mainClass in assembly := Some("org.sense.spark.app.App")

assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
assemblyJarName in assembly := s"${name.value}_${scalaBinaryVersion.value}-fat_${version.value}.jar"

来源：https://stackoverflow.com/questions/63012890/how-to-create-a-source-to-export-metrics-from-spark-to-another-sink-prometheus

标签

scala

apache-spark