bigdata

I have two vectors, v1 and v2 . I'd like to subtract each value of v2 from each value of v1 and store the results in another vector. I also would like to work with very large vectors (e.g. 1e6 size), so I think I should be using numpy for performance. Up until now I have: import numpy v1 = numpy.array(numpy.random.uniform(-1, 1, size=1e2)) v2 = numpy.array(numpy.random.uniform(-1, 1, size=1e2)) vdiff = [] for value in v1: vdiff.extend([value - v2]) This creates a list with 100 entries, each entry being an array of size 100. I don't know if this is the most efficient way to do this though. I'd

If the Kappa-Architecture does analysis on stream directly instead of splitting the data into two streams, where is the datastored then, in a messagin-system like Kafka? or can it be in a database for recomputing? And is a seperate batch layer faster than recomputing with a stream processing engine for batch analytics? "A very simple case to consider is when the algorithms applied to the real-time data and to the historical data are identical. Then it is clearly very beneficial to use the same code base to process historical and real-time data, and therefore to implement the use-case using the

Is it a good idea to create per day collections for data on a given day (we could start with per day and then move to per hour if there is too much data). Is there a limit on the number of collections we can create in mongodb, or does it result in performance loss (is it an overhead for mongodb to maintain so many collections). Does a large number of collections have any adverse effect on performance? To give you more context, the data will be more like facebook feeds, and only the latest data (say last one week or month) is more important to us. Making per day collections keeps the number of

This question already has an answer here: What will spark do if I don't have enough memory? 3 answers As far as I know, Spark tries to do all computation in memory, unless you call persist with disk storage option. If however, we don't use any persist, what does Spark do when an RDD doesn't fit in memory? What if we have very huge data. How will Spark handle it without crashing? Sachin Gaikwad From Apache Spark FAQ's: Spark's operators spill data to disk if it does not fit in memory, allowing it to run well on any sized data. Likewise, cached datasets that do not fit in memory are either

I have a set of n (~1000000) strings (DNA sequences) stored in a list trans. I have to find the minimum hamming distance of all sequences in the list. I implemented a naive brute force algorithm, which has been running for more than a day and has not yet given a solution. My code is dmin=len(trans[0]) for i in xrange(len(trans)): for j in xrange(i+1,len(trans)): dist=hamdist(trans[i][:-1], trans[j][:-1]) if dist < dmin: dmin = dist Is there a more efficient method to do this? Here hamdist is a function I wrote to find hamming distances. It is def hamdist(str1, str2): diffs = 0 if len(str1) !=

My Hive table has a date column with UTC date strings. I want to get all rows for a specific EST date. I am trying to do something like the below: Select * from TableName T where TO_DATE(ConvertToESTTimeZone(T.date)) = "2014-01-12" I want to know if there is a function for ConvertToESTTimeZone, or how I can achieve that? I tried the following but it doesnt work (my default timezone is CST): TO_DATE(from_utc_timestamp(T.Date) = "2014-01-12" TO_DATE( from_utc_timestamp(to_utc_timestamp (unix_timestamp (T.date), 'CST'),'EST')) Thanks in advance. Update: Strange behavior. When I do this: select