How to store dynamic data (unknown number of fields) to a file?

Question

I need to store some data in a file. FOR THE MOMENT each record (data set) consists in:

• a string (variable length),
• an array of integers (variable len

Answer 1

Take a look at our Synopse Big Table unit.

With its recent upgrade, it could fit perfectly your need.

Here is how you create your field layout:

var Table: TSynBigTableRecord;
    FieldText, FieldInt: TSynTableFieldProperties;
begin
  Table := TSynBigTableRecord.Create('FileName.ext','TableName');
  FieldText := Table.AddField('text',tftWinAnsi,[tfoIndex]);
  FieldInt := Table.AddField('Int',tftInt32,[tfoIndex,tfoUnique]);
  Table.AddFieldUpdate;

For storing an array of bytes or Integers, just use the tftWinAnsi or even better tftBlobInternal kind of field (this is a true variable-length field), then map it to or from a dynamic array, just like a RawByteString.

You can safely add fields later, the data file will be processed for you.

There are several ways of handling the data, but I've implemented a variant-based way of using a record, with true late-binding:

var vari: Variant;

  // initialize the variant
  vari := Table.VariantVoid;
  // create record content, and add it to the database
  vari.text := 'Some text';
  vari.int := 12345;
  aID := Table.VariantAdd(vari);
  if aID=0 then
    ShowMessage('Error adding record');
  // how to retrieve it
  vari := Table.VariantGet(aID);
  assert(vari.ID=aID);
  assert(vari.INT=12345);
  assert(vari.Text='Some text');

About speed, you can't find anything faster IMHO. Creating 1,000,000 records with some text and an integer value, both fields using an index, and the integer field set as unique is less than 880 ms on my laptop. It will use very little disk space, because all storage is variable-length encoded (similar to Google's Protocol Buffers).

It needs only two units, works for Delphi 6 up to XE (and is already Unicode ready, so using this unit you could upgrade safely to a newer Delphi version when you want to). There is no installation need, and it's only a few KB added to your executable. It's just a small but powerful NoSQL engine written in pure Delphi, but with the power of use of a database (i.e. pure field layout) and the speed of a in-memory engine, with no limit in size.

And it's full OpenSource, with a permissive licence.

Note that we also provide a SQLite3 wrapper, but it's another project. Slower but more powerful, with SQL support and an integrated Client/Server ORM.