Translating Scala code to Java for Spark Partitioner

大兔子大兔子 提交于 2019-12-11 02:49:45

问题


So I am trying to implement a custom partitioner using Spark with Java, and I found a great example of how to do this online, but it is using Scala, and I cannot for the life of me figure out how it translates properly into Java so I can try to implement it. Can anyone help? Here is the example code I found for it in Scala:

class DomainNamePartitioner(numParts: Int) extends Partitioner {
  override def numPartitions: Int = numParts
  override def getPartition(key: Any): Int = {
    val domain = new Java.net.URL(key.toString).getHost()
    val code = (domain.hashCode % numPartitions)
    if (code < 0) {
      code + numPartitions  // Make it non-negative
    } else {
      code
    }
  } 
  // Java equals method to let Spark compare our Partitioner objects
  override def equals(other: Any): Boolean = other match {
    case dnp: DomainNamePartitioner =>
      dnp.numPartitions == numPartitions
    case _ =>
      false
  }
}

回答1:


First of all, Scala is the first choice to write Spark.

Here is the corresponding Java code (it is not the unique version):

see more at : https://spark.apache.org/docs/latest/api/java/index.html

class DomainNamePartitioner extends Partitioner{

private int numParts;

public Partitioner()
{

}
public Partitioner(int numParts)
{
  this.numParts = numParts;
}

@Override
public int numPartitions()
{
    return numParts;
}
@Override
public int getPartition(Object key){

   String domain = new Java.net.URL(key.toString).getHost();
   int code = domain.hashCode % numPartitions;
   if (code < 0){
    return code + this.numPartitions();
   }else{
    return code;
   }
}

@Override
public boolean equals(Object obj){

    if (obj instanceof DomainNamePartitioner){
        DomainNamePartitioner pObj = (DomainNamePartitioner)obj;
        return pObj.numPartitions() == this.numPartitions;
    }else{
        return false;
    }
}
}


来源:https://stackoverflow.com/questions/30137693/translating-scala-code-to-java-for-spark-partitioner

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!