large-data

Im trying to run Mort Canty's http://mcanty.homepage.t-online.de/ Python iMAD implementation on bitemporal RapidEye Multispectral images. Which basically calculates the canonical correlation for the two images and then substracts them. The problem I'm having is that the images are of 5000 x 5000 x 5 (bands) pixels. If I try to run this on the whole image I get a memory error. Would the use of something like pyTables help me with this? What Mort Canty's code tries to do is that it loads the images using gdal and then stores them in an 10 x 25,000,000 array. # initial weights wt = ones(cols*rows

I am collecting a large amount of data that will be saved into individual H5 files using h5py. I would like to patch these images together into one pcolormesh plot to be saved as a single image. A quick example I have been working on generates arrays of 2000x2000 random data points and saves them in H5 files using h5py. Then I try to import the data in these files and try to plot it in matplotlib as a pcolormesh, but I always run into a memoryError (which is expected). import numpy import h5py arr = numpy.random.random((2000,2000)) with h5py.File("TEST_HDF5_SAVE_FILES\\Plot_0.h5", "w") as f:

I'm looking to compute something like: Where f(i) is a function that returns a real number in [-1,1] for any i in {1,2,...,5000} . Obviously, the result of the sum is somewhere in [-1,1] , but when I can't seem to be able to compute it in Python using straight forward coding, as 0.5 5000 becomes 0 and comb(5000,2000) becomes inf , which result in the computed sum turning into NaN . The required solution is to use log on both sides. That is using the identity a × b = 2 log(a) + log(b) , if I could compute log(a) and log(b) I could compute the sum, even if a is big and b is almost 0 . So I guess

I am having a very common problem which it seems that all the available solutions found are not working. We have a LAMP server which is receiving high amount of traffic. Using this server, we perform a regular file submission upload. On small file uploads, it works perfectly. On files of around 4-5MB, this submission upload failed intermittently (sometimes it works but many times it failed). We have the following configuration on our PHP: max_input_time: 600 max_execution_time: 600 max_upload_size: 10M post_max_size: 10M Apache setting: Timeout: 600 Keep-Alive Timeout: 15 Keep-Alive: On Per

Update I have resolved and removed the distracting error. Please read the entire post and feel free to leave comments if any questions remain. Background I am attempting to write relatively large files (video) to disk on iOS using Swift 2.0, GCD, and a completion handler. I would like to know if there is a more efficient way to perform this task. The task needs to be done without blocking the Main UI, while using completion logic, and also ensuring that the operation happens as quickly as possible. I have custom objects with an NSData property so I am currently experimenting using an extension

I have a very long string. I want to find all the unique substrings of this string. I tried to write the code where I used a set (python) to store all the substrings to ensure uniqueness. I am getting correct result for many medium and large strings however in case of very large strings, I am getting a MemoryError. I googled a bit and found out that the set data structure in python has a large RAM footprint and maybe thats why I am getting a MemoryError. Here is my code : a = set() for i in range(n): string = raw_input() j = 1 while True: for i in xrange(len(string)-j+1): a.add(string[i:i+j])

I'm trying and need some help doing the following: I want to stream parse a large XML file ( 4 GB ) with PHP. I can't use simple XML or DOM because they load the entire file into memory, so I need something that can stream the file. How can I do this in PHP? What I am trying to do is to navigate through a series of <doc> elements. And write some of their children to a new xml file. The XML file I am trying to parse looks like this: <feed> <doc> <title>Title of first doc is here</title> <url>URL is here</url> <abstract>Abstract is here...</abstract> <links> <sublink>Link is here</sublink>