scala-breeze

I'm developing a Scala-based extreme learning machine, in Apache Spark. My model has to be a Spark Estimator and use the Spark framework in order to fit into the machine learning pipeline. Does anyone know if Breeze can be used in tandem with Spark? All of my data is in Spark data frames and conceivably I could import it using Breeze, use Breeze DenseVectors as the data structure then convert to a DataFrame for the Estimator part. The advantage of Breeze is that it has a function pinv for the Moore-Penrose pseudo-inverse, which is an inverse for a non-square matrix. There is no equivalent

While trying to use MLlib from Java, what is the correct way to use breeze Matrix operations? For e.g. multiplication in scala it ist simply " matrix * vector ". How is the corresponding functionality expressed in Java? There are methods like " $colon$times " which might be invoked by the correct way breeze.linalg.DenseMatrix<Double> matrix= ... breeze.linalg.DenseVector<Double> vector = ... matrix.$colon$times( ... one might need an operator instance ... breeze.linalg.operators.OpMulMatrix.Impl2 But which exact typed Operation instance and parameters are to be used? It's honestly very hard.

This question is about MLlib (Spark 1.2.1+). What is the best way to manipulate local matrices (moderate size, under 100x100, so does not need to be distributed). For instance, after computing the SVD of a dataset, I need to perform some matrix operation. The RowMatrix only provide a multiply function. The toBreeze method returns a DenseMatrix<Object> but the API does not seem Java friendly: public final <TT,B,That> That $plus(B b, UFunc.UImpl2<OpAdd$,TT,B,That> op) In Spark+Java, how to do any of the following operations: transpose a matrix add/subtract two matrices crop a Matrix perform

I've a List[List[Int]] = List(List(1, 2, 3, 0, 0, 0, 0, 0, 0), List(0, 0, 0, 1, 2, 3, 0, 0, 0), List(0, 0, 0, 0, 0, 0, 1, 2, 3)) and I want to create a Matrix/DenseMatrix with 3 rows and 9 columns so that I can perform some Linear Algebra operations on it. There's not great syntax for this yet. I actually ran into it today, so I'll probably add better support for it soon. In the meantime, DenseMatrix(array1, array2, array3) works. So: val listOfLists : List[List[Int]] = ??? DenseMatrix(listOfLists.map(_.toArray):_*) Not the prettiest, but I'll add better support if you open a ticket. 来源： https

I am trying to develop my own feedforward nerual network using spark. However I can't find the operations like multiplication , add or divide in the spark's sparse vector. The document said it is implemented using breeze vector. But I can find add operations in breeze but not in spark vector. How to solve this problem? Spark's Vector implementations don't support algebraic operations. Unfortunately, the Spark API no longer supports to convert SparkVectors into BreezeVectors via the methods asBreeze and fromBreeze , because these methods have been made package private with respect to the spark