hdf5

Is there a way to get a numpy-style view to a slice of an array stored in a hdf5 file?

由 ﹥＞﹥吖頭↗ 提交于 2019-12-08 15:47:00
问题 I have to work on large 3D cubes of data. I want to store them in HDF5 files (using h5py or maybe pytables). I often want to perform analysis on just a section of these cubes. This section is too large to hold in memory. I would like to have a numpy style view to my slice of interest, without copying the data to memory (similar to what you could do with a numpy memmap). Is this possible? As far as I know, performing a slice using h5py, you get a numpy array in memory. It has been asked why I

Read the properties of HDF file in Python

由 谁说胖子不能爱 提交于 2019-12-08 12:18:03
问题 I have a problem reading hdf file in pandas. As of now, I don't know the keys of the file. How do I read the file [data.hdf] in such a case? And, my file is .hdf not .h5 , Does it make a difference it terms data fetching? I see that you need a 'group identifier in the store' pandas.io.pytables.read_hdf(path_or_buf, key, **kwargs) I was able to get the metadata from pytables File(filename=data.hdf, title='', mode='a', root_uep='/', filters=Filters(complevel=0, shuffle=False, fletcher32=False,

hdf5 error when format=table, pandas pytables

由 ▼魔方 西西 提交于 2019-12-08 09:05:09
问题 It seems that I get an error when format=table but no error with format=fixed . Here is the command. What's weird is that it still seems to load the data. I just have to figure out a way to move past this. And it would give me peace of mind to not have any error. The dataframe is preprocessed, types set within the columns. The command I run is: hdf = pd.HDFStore('path-to-file') hdf.put('df',df,format='table') The error I get is: HDF5ExtError: HDF5 error back trace File "../../../src/H5Dio.c",

how to specify a custom compression filter in h5py

由 落花浮王杯 提交于 2019-12-08 04:32:36
问题 According to they h5py documentation, "compression filters can be dynamically loaded by the underlying HDF5 library. This is done by passing a filter number to Group.create_dataset() as the compression parameter." I have the following code (which works fine): import numpy as np a = np.random.random(1e5) with h5py.File(args.baseName + '/allubf.h5', libver='latest') as f: dset = f.create_dataset('myData', (1000,), dtype=np.dtype(float), chunks=(1000,), compression='lzf', shuffle=True) I looked

merging several hdf5 files into one pytable

由 倾然丶 夕夏残阳落幕 提交于 2019-12-07 17:29:29
问题 I have several hdf5 files, each of them with the same structure. I'd like to create one pytable out of them by somehow merging the hdf5 files. What I mean is that if an array in file1 has size x and array in file2 has size y, the resulting array in the pytable will be of size x+y, containing first all the entries from file1 and then all the entries from file2. 回答1: How you want to do this depends slightly on the data type that you have. Arrays and CArrays have a static size so you need to

OSError: Input/Output Error - Google Colab

由 a 夏天 提交于 2019-12-07 15:56:27
Using h5py.File(path,'r') or (Keras) model.load_weights(path) on .h5 files yields the below error. I've trained models on Colab for months, never had this problem. The same files open just fine on my computer when downloaded from Drive. At first one notebook failed to open .h5 files - all others followed soon after. What's the error, and how to troubleshoot? Inability to save & load models renders Colab unusable - help is appreciated. UPDATE : What started as inability to open h5 files morphed into inability to import simple modules - which somehow restored ability to load some of the h5 files

Using std:: string in hdf5 creates unreadable output

由 亡梦爱人 提交于 2019-12-07 12:56:09
问题 I'm currently using hdf5 1.8.15 on Windows 7 64bit. The sourcecode of my software is saved in files using utf8 encoding. As soon as I call any hdf5 function supporting std:: string, the ouput gets cryptic But if I use const char* instead of std::string , everything works fine. This applies also to the filename. Here is a short sample: std::string filename_ = "test.h5"; H5::H5File file( filename_.c_str(), H5F_ACC_TRUNC); // works H5::H5File file( filename_, H5F_ACC_TRUNC); // filename is not

Reading hdf5 files to dynamic arrays in c++

由 ▼魔方 西西 提交于 2019-12-07 12:46:17
问题 I am trying to read large 3D hdf5 files into a dynamic array due to size limitations on the stack. I have tried several different methods and have failed via segmentation fault. Below is example code to show my issue. I would very much appreciate some help!! //This example was based on several examples which came in the c++ examples directory of the hdf5 package. #ifdef OLD_HEADER_FILENAME #include <iostream.h> #else #include <iostream> #endif #include <string> #include <new> #include "hdf5.h

Pandas read_hdf query by date and time range

由 ⅰ亾dé卋堺 提交于 2019-12-07 12:10:51
问题 I have a question regarding how to filter results in the pd.read_hdf function. So here's the setup, I have a pandas dataframe (with np.datetime64 index) which I put into a hdf5 file. There's nothing fancy going on here, so no use of hierarchy or anything (maybe I could incorporate it?). Here's an example: Foo Bar TIME 2014-07-14 12:02:00 0 0 2014-07-14 12:03:00 0 0 2014-07-14 12:04:00 0 0 2014-07-14 12:05:00 0 0 2014-07-14 12:06:00 0 0 2014-07-15 12:02:00 0 0 2014-07-15 12:03:00 0 0 2014-07

How does one store a Pandas DataFrame as an HDF5 PyTables table (or CArray, EArray, etc.)?

由 北战南征 提交于 2019-12-07 09:11:54
问题 I have the following pandas dataframe: import pandas as pd df = pd.read_csv(filename.csv) Now, I can use HDFStore to write the df object to file (like adding key-value pairs to a Python dictionary): store = HDFStore('store.h5') store['df'] = df http://pandas.pydata.org/pandas-docs/stable/io.html When I look at the contents, this object is a frame . store outputs <class 'pandas.io.pytables.HDFStore'> File path: store.h5 /df frame (shape->[552,23252]) However, in order to use indexing, one
订阅 hdf5