bigdata

Is there anyway I can rename an element in an existing elasticsearch mapping without having to add a new element ? If so whats the best way to do it in order to avoid breaking the existing mapping? e.g. from fieldCamelcase to fieldCamelCase { "myType": { "properties": { "timestamp": { "type": "date", "format": "date_optional_time" }, "fieldCamelcase": { "type": "string", "index": "not_analyzed" }, "field_test": { "type": "double" } } } } You could do this by creating an Ingest pipeline, that contains a Rename Processor in combination with the Reindex API . PUT _ingest/pipeline/my_rename

So far I have : (:require [clojure-csv.core :as csv]) (:require [clojure.java.io :as io])) (def csv-file (.getFile (clojure.java.io/resource "verbs.csv"))) (defn process-csv [file] (with-open [rdr (io/reader file)] (csv/parse-csv rdr))) But I am getting java.io.IOException: Stream closed . I am using clojure-csv and it exposes two methods, the first of which I am using, parse-csv , the doc says : Takes a CSV as a char sequence or string, and returns a lazy sequence of vectors of strings What I think I know : with-open is lazy, and the rdr in (csv/parse-csv rdr))) is a single line of the csv

I have decided to use memmapfile because my data (typically 30Gb to 60Gb) is too big to fit in a computer's memory. My data files consist two columns of data that correspond to the outputs of two sensors and I have them in both .bin and .txt formats. m=memmapfile('G:\E-Stress Research\Data\2013-12-18\LD101_3\EPS/LD101_3.bin','format','int32') m.data(1) I used the above code to memory map my data to a variable "m" but I have no idea what data format to use (int8', 'int16', 'int32', 'int64','uint8', 'uint16', 'uint32', 'uint64', 'single', and 'double'). In fact I tried all of the data formats

If the file is loaded from HDFS by default spark creates one partition per block. But how does spark decide partitions when a file is loaded from S3 bucket? See the code of org.apache.hadoop.mapred.FileInputFormat.getSplits() . Block size depends on S3 file system implementation (see FileStatus.getBlockSize() ). E.g. S3AFileStatus just set it equals to 0 (and then FileInputFormat.computeSplitSize() comes into play). Also, you don't get splits if your InputFormat is not splittable :) Spark will treat S3 as if it were a block-based filesystem, so partitioning rules for HDFS and S3 inputs are the

I'm using rmongodb to get every document in a a particular collection. It works but I'm working with millions of small documents, potentially 100M or more. I'm using the method suggested by the author on the website: cnub.org/rmongodb.ashx count <- mongo.count(mongo, ns, query) cursor <- mongo.find(mongo, query) name <- vector("character", count) age <- vector("numeric", count) i <- 1 while (mongo.cursor.next(cursor)) { b <- mongo.cursor.value(cursor) name[i] <- mongo.bson.value(b, "name") age[i] <- mongo.bson.value(b, "age") i <- i + 1 } df <- as.data.frame(list(name=name, age=age)) This

I am learning hive and came across skewed tables. Help me understanding it. What are skewed tables in Hive? How do we create skewed tables? How does it effect performance? Tariq What are skewed tables in Hive? A skewed table is a special type of table where the values that appear very often (heavy skew) are split out into separate files and rest of the values go to some other file.. How do we create skewed tables? create table <T> (schema) skewed by (keys) on ('value1', 'value2') [STORED as DIRECTORIES]; Example : create table T (c1 string, c2 string) skewed by (c1) on ('x1') How does it

Does flatMap in spark behave like the map function and therefore cause no shuffling, or does it trigger a shuffle. I suspect it does cause shuffling. Can someone confirm it? There is no shuffling with either map or flatMap. The operations that cause shuffle are: Repartition operations: Repartition: Coalesce: ByKey operations (except for counting): GroupByKey: ReduceByKey: Join operations: Cogroup: Join: Although the set of elements in each partition of newly shuffled data will be deterministic, and so is the ordering of partitions themselves, the ordering of these elements is not. If one