bigdata

What's the best way to do the equivalent of an SQL join in Elasticsearch? I have an SQL setup with two large tables: Persons and Items. A Person can own many items. Both Person and Item rows can change (i.e. be updated). I have to run searches which filter by aspects of both the person and the item. In Elasticsearch, it looks like you could make Person a nested document of Item, then use has_child . But: if you then update a Person, I think you'd need to update every Item they own (which could be a lot). Is that correct? Is there a nice way to solve this query in Elasticsearch? As already

What's the meaning of the frozen keyword in Cassandra? I'm trying to read this documentation page: Using a user-defined type , but their explanation for the frozen keyword (which they use in their examples) is not clear enough for me: To support future capabilities, a column definition of a user-defined or tuple type requires the frozen keyword. Cassandra serializes a frozen value having multiple components into a single value. For examples and usage information, see "Using a user-defined type", "Tuple type", and Collection type. I haven't found any other definition or a clear explanation for

In Apache Flink I have a stream of tuples. Let's assume a really simple Tuple1<String> . The tuple can have an arbitrary value in it's value field (e.g. 'P1', 'P2', etc.). The set of possible values is finite but I don't know the full set beforehand (so there could be a 'P362'). I want to write that tuple to a certain output location depending on the value inside of the tuple. So e.g. I would like to have the following file structure: /output/P1 /output/P2 In the documentation I only found possibilities to write to locations that I know beforehand (e.g. stream.writeCsv("/output/somewhere") ),

I'm a grad student in astrophysics. I run big simulations using codes mostly developed by others over a decade or so. For examples of these codes, you can check out gadget http://www.mpa-garching.mpg.de/gadget/ and enzo http://code.google.com/p/enzo/ . Those are definitely the two most mature codes (they use different methods). The outputs from these simulations are huge . Depending on your code, your data is a bit different, but it's always big data. You usually take billions of particles and cells to do anything realistic. The biggest runs are terabytes per snapshot and hundreds of snapshots

Trying to find a complete documentation about an internal architecture of Apache Spark, but have no results there. For example I'm trying to understand next thing: Assume that we have 1Tb text file on HDFS (3 nodes in a cluster, replication factor is 1). This file will be spitted into 128Mb chunks and each chunk will be stored only on one node. We run Spark Workers on these nodes. I know that Spark is trying to work with data stored in HDFS on the same node (to avoid network I/O). For example I'm trying to do a word count in this 1Tb text file. Here I have next questions: Does Spark will load