问题
So I am trying to implement a custom partitioner using Spark with Java, and I found a great example of how to do this online, but it is using Scala, and I cannot for the life of me figure out how it translates properly into Java so I can try to implement it. Can anyone help? Here is the example code I found for it in Scala:
class DomainNamePartitioner(numParts: Int) extends Partitioner {
override def numPartitions: Int = numParts
override def getPartition(key: Any): Int = {
val domain = new Java.net.URL(key.toString).getHost()
val code = (domain.hashCode % numPartitions)
if (code < 0) {
code + numPartitions // Make it non-negative
} else {
code
}
}
// Java equals method to let Spark compare our Partitioner objects
override def equals(other: Any): Boolean = other match {
case dnp: DomainNamePartitioner =>
dnp.numPartitions == numPartitions
case _ =>
false
}
}
回答1:
First of all, Scala is the first choice to write Spark.
Here is the corresponding Java code (it is not the unique version):
see more at : https://spark.apache.org/docs/latest/api/java/index.html
class DomainNamePartitioner extends Partitioner{
private int numParts;
public Partitioner()
{
}
public Partitioner(int numParts)
{
this.numParts = numParts;
}
@Override
public int numPartitions()
{
return numParts;
}
@Override
public int getPartition(Object key){
String domain = new Java.net.URL(key.toString).getHost();
int code = domain.hashCode % numPartitions;
if (code < 0){
return code + this.numPartitions();
}else{
return code;
}
}
@Override
public boolean equals(Object obj){
if (obj instanceof DomainNamePartitioner){
DomainNamePartitioner pObj = (DomainNamePartitioner)obj;
return pObj.numPartitions() == this.numPartitions;
}else{
return false;
}
}
}
来源:https://stackoverflow.com/questions/30137693/translating-scala-code-to-java-for-spark-partitioner