Which embedded database capable of 100 million records has an efficient C or C++ API

Question

I\'m looking for a cross-platform database engine that can handle databases up hundreds of millions of records without severe degradation in query performance. It needs to h

Answer 1

I believe what you are looking for is BerkeleyDB: http://www.oracle.com/technology/products/berkeley-db/db/index.html

Never mind that it's Oracle, the license is free, and it's open-source -- the only catch is that if you redistribute your software that uses BerkeleyDB, you must make your source available as well -- or buy a license.

It does not provide SQL support, but rather direct lookups (via b-tree or hash-table structure, whichever makes more sense for your needs). It's extremely reliable, fast, ACID, has built-in replication support, and so on.

Here is a small quote from the page I refer to above, that lists a few features:

Data Storage

Berkeley DB stores data quickly and easily without the overhead found in other databases. Berkeley DB is a C library that runs in the same process as your application, avoiding the interprocess communication delays of using a remote database server. Shared caches keep the most active data in memory, avoiding costly disk access.

• Local, in-process data storage
• Schema-neutral, application native data format
• Indexed and sequential retrieval (Btree, Queue, Recno, Hash)
• Multiple processes per application and multiple threads per process
• Fine grained and configurable locking for highly concurrent systems
• Multi-version concurrency control (MVCC)
• Support for secondary indexes
• In-memory, on disk or both
• Online Btree compaction
• Online Btree disk space reclamation
• Online abandoned lock removal
• On disk data encryption (AES)
• Records up to 4GB and tables up to 256TB

Update: Just ran across this project and thought of the question you posted: http://tokyocabinet.sourceforge.net/index.html . It is under LGPL, so not compatible with your restrictions, but an interesting project to check out, nonetheless.