bigdata

On elastic search, when doing a simple query like: GET miindex-*/mytype/_search { "query": { "query_string": { "analyze_wildcard": true, "query": "*" } } } It returns a format like: { "took": 1, "timed_out": false, "_shards": { "total": 1, "successful": 1, "failed": 0 }, "hits": { "total": 28, "max_score": 1, "hits": [ ... So I parse like response.hits.hits to get the actual records. However if you are doing another type of query e.g. aggregation, the response is totally different like: { "took": 1, "timed_out": false, "_shards": { "total": 1, "successful": 1, "failed": 0 }, "hits": { "total":

Suppose I have a dataframe with multiple columns, I want to iterate each column, do some calculation and update that column. Is there any good way to do that? Update In below example I have a dataframe with two integer columns c1 and c2. each column's value is divided with the sum of its columns. import org.apache.spark.sql.expressions.Window val df = Seq((1,15), (2,20), (3,30)).toDF("c1","c2") val result = df.columns.foldLeft(df)((acc, colname) => acc.withColumn(colname, sum(acc(colname)).over(Window.orderBy(lit(1)))/acc(colname))) Output : scala> result.show() +---+------------------+ | c1|

I have wrote a code for finding connected component of a very big Graph (80 million edges) but it doesn't work, When the edges number is close to 40 million it crashed. int main(){ using namespace boost; { int node1,node2; typedef adjacency_list <vecS, vecS, undirectedS> Graph; Graph G; std::ifstream infile("pairs.txt"); std::string line; while (std::getline(infile,line)) { std::istringstream iss(line); iss >> node1 >> node2; add_edge(node1, node2, G);} cout <<"writing file"<<endl; int j = 0; ofstream out; out.open("connected_component.txt"); std::vector<int> component(num_vertices(G)); int