Load Word2Vec model in Spark

问题

Is it possible to load a pretrained (binary) model to spark (using scala) ? I have tried to load one of the binary models which was generated by google like this:

    import org.apache.spark.mllib.feature.{Word2Vec, Word2VecModel}


    val model = Word2VecModel.load(sc, "GoogleNews-vectors-negative300.bin")

but it is not able to locate the metadata directory. I also created the folder and appended the binary file there but it cannot be parsed. I did not find any wrapper for this issue.

回答1:

I wrote a quick function to load in the google news pretrained model into a spark word2vec model. Enjoy.

def loadBin(file: String) = {
  def readUntil(inputStream: DataInputStream, term: Char, maxLength: Int = 1024 * 8): String = {
    var char: Char = inputStream.readByte().toChar
    val str = new StringBuilder
    while (!char.equals(term)) {
      str.append(char)
      assert(str.size < maxLength)
      char = inputStream.readByte().toChar
    }
    str.toString
  }
  val inputStream: DataInputStream = new DataInputStream(new GZIPInputStream(new FileInputStream(file)))
  try {
    val header = readUntil(inputStream, '\n')
    val (records, dimensions) = header.split(" ") match {
      case Array(records, dimensions) => (records.toInt, dimensions.toInt)
    }
    new Word2VecModel((0 until records).toArray.map(recordIndex => {
      readUntil(inputStream, ' ') -> (0 until dimensions).map(dimensionIndex => {
        java.lang.Float.intBitsToFloat(java.lang.Integer.reverseBytes(inputStream.readInt()))
      }).toArray
    }).toMap)
  } finally {
    inputStream.close()
  }
}

回答2:

It is an unresolved issue: https://issues.apache.org/jira/browse/SPARK-15328

Either look at the specific code and try to recreate something for yourself or maybe use a python or C script to convert the binary to txt data and work from there.

Convert word2vec bin file to text

来源：https://stackoverflow.com/questions/43866703/load-word2vec-model-in-spark

标签

scala

apache-spark

word2vec