shelve

How to serialize hash objects in Python

有些话、适合烂在心里 提交于 2019-12-06 13:27:37
问题 How can I serialize hash objects?, I'm using shelve to store a lot of objects. Hierarchy: - user - client - friend user.py: import time import hashlib from localfile import localfile class user(object): _id = 0 _ip = "127.0.0.1" _nick = "Unnamed" _files = {} def __init__(self, ip="127.0.0.1", nick="Unnamed"): self._id = hashlib.sha1(str(time.time())) self._ip = ip self._nick = nick def add_file(self, localfile): self._files[localfile.hash] = localfile def delete_file(self, localfile): del

Storing a 7millions keys python dictionary in a database

与世无争的帅哥 提交于 2019-12-06 04:14:29
问题 I have to handle a 7 millions keys dictionary (the number of keys can eventually be up to ~50 millions). Since I have barely enough ram to keep it in memory I've decided to store it. My dictionary looks like this: dictionary={(int1,int2):int3,...} First I tried to store it in a sqlite database using sqlite3. The amount of time required to store it is perfectly ok (around 70 secs). Using timeit : >>>import sqlite3 >>>conn=sqlite3.connect('test_sqlite.sqlite') >>>c=conn.cursor() >>>c.execute(

Is shelve really slow and taking a lot of memory or am I doing something wrong?

放肆的年华 提交于 2019-12-06 03:32:23
I'm trying to write a program that uses a shelve database of sorted letters as key, and a list of words that can be created from them as values. e.g: db['mnoo'] = ['moon', 'mono'] So I wrote a function that takes a filename and loads it into a shelve. The first part, that turns the file into a dictionary with the same layout as the shelve works fine, but the shelve part takes really long. I'm trying it with a dictionary of ~100k entries, each value being a list. It seems to take 15-20 seconds for each 1000 enteries, and each entry seems to take ~1kb of space. Is this nromal? the code: def save

Python shelve having items that aren't listed

痞子三分冷 提交于 2019-12-05 09:43:23
I've been saving a bunch of dictionaries to file using Python's shelve module (with Python 3.4 on OSX 10.9.5). Each key is a string of an int (e.g., "84554" ), and each value is a dictionary of dictionaries of a few smallish strings. No keys are used twice, and I know the total superset of all possible keys. I am adding these key-value pairs to the shelf via threads and which keys/values are added changes each time I run it (which is expected). The problem I've been having is that the number of keys iterable/visible with shelve 's shelf.keys() and the number of unique keys for which key in

Shelve module in python not working: “db type cannot be determined”

不打扰是莪最后的温柔 提交于 2019-12-05 07:04:02
I am trying to make a simple password-storing program in Python, and it seems pretty simple so I am wondering if I am using shelve wrong. I have the main .py file: import shelve passwords = shelve.open('./passwords_dict.py') choice = raw_input("Add password (a) or choose site (c)?") if choice[0] == 'a': site_key = raw_input("Add for which site? ").lower() userpass = raw_input("Add any info such as username, email, or passwords: ") passwords[site_key] = userpass else: site = raw_input("Which site? ").lower() if site in passwords: print "Info for " + site + ": " + passwords[site] else: print

How to keep uncommitted changes in a local mercurial repository, while still pushing/pulling?

让人想犯罪 __ 提交于 2019-12-05 06:06:32
If i'm working on some files that I don't want to commit, I just save them. I then have other files I want to push to the server, however if someone else has made changes to the repository, and I pull them down, it asks me to merge or rebase.. but either of these options will cause me to lose my local changes that I have not committed . What are other people doing to get around this? I find the documentation for the shelving extension hard to get my head around. Note: I'm using Mercurial Eclipse to push and pull files to/from the server. Any explanation of this would be greatly appreciated!

How to serialize hash objects in Python

那年仲夏 提交于 2019-12-04 18:50:32
How can I serialize hash objects?, I'm using shelve to store a lot of objects. Hierarchy: - user - client - friend user.py: import time import hashlib from localfile import localfile class user(object): _id = 0 _ip = "127.0.0.1" _nick = "Unnamed" _files = {} def __init__(self, ip="127.0.0.1", nick="Unnamed"): self._id = hashlib.sha1(str(time.time())) self._ip = ip self._nick = nick def add_file(self, localfile): self._files[localfile.hash] = localfile def delete_file(self, localfile): del self._files[localfile.hash] if __name__ == "__main__": pass client.py: from user import user from friend

Persistent multiprocess shared cache in Python with stdlib or minimal dependencies

我的未来我决定 提交于 2019-12-04 06:29:20
I just tried a Python shelve module as the persistent cache for data fetched from the external service. The complete example is here . I was wondering what would the best approach if I want to make this multiprocess safe? I am aware of redis, memcached and such "real solutions", but I'd like to use only the parts of Python standard library or very minimal dependencies to keep my code compact and not introduce unnecessary complexity when running the code in single process - single thread model. It's easy to come up with a single-process solution, but this does not work well current Python web

Can serialized objects be accessed simultaneously by different processes, and how do they behave if so?

谁说我不能喝 提交于 2019-12-04 06:02:18
问题 I have data that is best represented by a tree. Serializing the structure makes the most sense, because I don't want to sort it every time, and it would allow me to make persistent modifications to the data. On the other hand, this tree is going to be accessed from different processes on different machines, so I'm worried about the details of reading and writing. Basic searches didn't yield very much on the topic. If two users simultaneously attempt to revive the tree and read from it, can

hg unshelve seems to have no effect?

China☆狼群 提交于 2019-12-04 00:51:44
Our team is just getting started with Mercurial. One of the first things we've started to play with is hg shelve . Locally, I have no problem shelving changes. It all works perfectly from what I can tell. However, when I try to unshelve, I get the restoring backup files message, but when I run hg diff , there are no changes, and my changes are missing from the code. If i do hg unshelve -i I can see the diff, but again, trying to unshelve seems to have no effect. I've been trying to test it with some very simple changes that shouldn't be a problem in terms of conflicts, e.g. adding a test