java disc based hashmap

Question

I\'m working on a web crawler (please don\'t suggest an existing one, its not an option). I have it working the way it is expected to. My only issue is that currently I\'m u

Answer 1

There is Tokyo Cabinet, which is a fast implementation of a disk-based hash table.

In your case, I think the best way to store values in such a setup would be to prefix the metadata keys with the url:

[url]_[name] => [value]
[url]_[name2] => [value2]

Unfortunately, I'm not sure you can enumerate the metadata for a given URL, using this solution.

If you want to use a more structured data store, there are also MongoDB, and SQLite, which I would recommend.