large-data

Say I have some huge amount of data stored in an HDF5 data file (size: 20k x 20k, if not more) and I want to create an image from all of this data using Python. Obviously, this much data cannot be opened and stored in the memory without an error. Therefore, is there some other library or method that would not require all of the data to be dumped into the memory and then processed into an image (like how the libraries: Image, matplotlib, numpy, etc. handle it)? Thanks. This question comes from a similar question I asked: Generating pcolormesh images from very large data sets saved in H5 files

I'm thinking about using Apache Solr. In my db I will have around 10.000.000 records.The worst case where I will use it has around 20 searchable/sortable fields. My problem is that these fields may change values frequently during the day. For example in my db I might change some fields at the same time of 10000 records and this may happen 0, 1 or 1000 times a day etc. The point is that each time I update a value in the db I want it to be updated in solr too so I can search with the updated data each time. For those of you that have used solr, how fast can re indexing in such volumes be? Will

So I have a simple layout of a page which includes a panel of filters and a html table of records using ng-repeat. I am using MVC5 and an angularJs controller I may have to deal with up to 100000 records. Filters will occur for most of the columns including dates and text fields The records need to deal with two way binding (user has to select records which will be returned to the server). I'd like get opinions on the best design ideas for this....i.e. Would you load all the data to the browser upfront. If not when would more data be requested from the server. If all upfront should two arrays

I need to design a table in Oracle, which will store 2-5 TB of data in a day. It can grow to 200TB, and records will purged when it crosses 200 TB. Is it a feasible choice to keep it in OLTP, or do I need to shift it to data warehouse DB? Please advice considerations I should keep in mind when designing the schema of this table, or the database. Also, please advice if it is a SQL server, as I can use either database. That size puts you in the VLDB territory (very large databases). Things are fundamentally different at that altitude. Your question cannot be answered without the full

I am using the following method to copy the contents of a directory to a different directory. public void DirCopy(string SourcePath, string DestinationPath) { if (Directory.Exists(DestinationPath)) { System.IO.DirectoryInfo downloadedMessageInfo = new DirectoryInfo(DestinationPath); foreach (FileInfo file in downloadedMessageInfo.GetFiles()) { file.Delete(); } foreach (DirectoryInfo dir in downloadedMessageInfo.GetDirectories()) { dir.Delete(true); } } //================================================================================= string[] directories = System.IO.Directory.GetDirectories