HSQLDB: enable LOB compression for existing database

问题:

I'm using an embedded HSQLDB 2.3.2 instance to store XML documents as LOBs for a while in an application which has intermittent connection to wherever the documents are supposed to end.

To limit HSQBDL's .lobs file size growth, I enabled LOB compression through the jdbc connection URL as mentioned in the documentation (using hsqldb.lob_compressed=true), but from what I've recently discovered this URL parameter has no effect whatsoever.

If I understand the HSQLDB JDBC URL parsing code correctly, org.hsqldb.persist.Logger#setVariables() should at some point check HsqlDatabaseProperties.hsqldb_lob_file_compressed to set propCompressLobs, like it does for LOB crypto & cryptLobs, but never does so.

A database.script file from a database created with such parameter in the URL has no mention of LOB compression despite the URL parameter, and does not compress LOBs.

I've tried issuing a "SET FILES LOB COMPRESSED TRUE" statement just after opening the database, and this works correctly for a brand new database (lobs get compressed, database.script mentions LOB compression).

When I try this with one of the existing databases around I get an error due to the existing LOBs (error message is "data file in use"). I understand the reasoning behind this, since compressing all LOBs might take a while and would probably be a very long running operation which might leave the db in a bad state if interrupted.

I thought I could work around that limitation since there are times when the XML documents are deleted (after they eventually get sent), so theoretically the DB is empty of LOBs at some point.

Unfortunately this error also happens on empty databases because there are unused LOB entries of deleted LOBs, which I assume are kept there to recycle .lobs file space. It seems the org.hsqldb.persist.Logger#setLobFileCompressed() method which is responsible for allowing LOB compression mode changes only checks the number of entries in SYSTEM_LOBS.LOB_IDS, whether they are currently in use or not.

From what I've read, there is no way to purge the deleted LOB entries (for example to reduce .lobs file size) so basically even if all the XML documents currently stored in LOBs get sent and purged from the database, this is not a good time to enable compression either because deleted LOBs leave footprints which prevent enabling compression.

So short of closing the DB when it has no documents left, destroying it, and creating a new DB and then manually issuing the lob compression statement, I don't see any way of enabling LOB compression for applications with existing databases.

I'm not particularly fond of that option which seems way to hackish.

I haven't tried 2.3.3 yet, but from what I see of the sources, it doesn't look any better in that version.

Is there any other way to reliably enable the compression of LOBs of existing HSQLDB databases ? Even "empty" ones ?

回答1:

The compression setting on the URL is effective only when the database is created. The SQL setting can be executed on an empty database that has no lob history.

Compression capabilities were added a long time after version 2.0 which supported non-compressed lob data. There is no ability to change an existing database's setting.

If the database is empty of lobs, you should be able to perform CHECKPOINT, then use the SQL setting. If this doesn't work, there is the next option.

If a database is empty of lobs (but can have other data), you can shutdown the database then edit the .script file and delete the INSERT INTO ... entries for SYSTEM_LOBS and add the default entry INSERT INTO BLOCKS VALUES(0,2147483647,0) that appear in a new database. If you do so, you should delete the .lobs file as well.

You can use SQL to select form SYSTEM_LOBS tables to check their contents but cannot modify them.

回答2:

I've found a way to do it reliably with only SQL, whether the db is a new one or an existing one which has seen some activity.

• ensure there are 0 existing LOBs using select count(*) from SYSTEM_LOBS.LOB_IDS
• insert a row in any table with a LOB so LobManager usageChanged is set to true.
• commit.
• delete the row you just created.
• commit again.
• execute a CHECKPOINT so all lob metadata is cleaned up. Since a LOB has recently been added, the checkpoint will actually run LobManager.deleteUnusedLobs() and cleanup the unused lob entries.
• enable compression using SET FILES LOB COMPRESSED TRUE. This now works because there is no longer anything in the SYSTEM_LOBS.LOB_IDS table.

If you then want to reliably trim the HSQLDB .lobs file which grew huge because compression was not active:

• insert another row with a LOB so HSQLDB can compute the lob space usage (LobManager.getLobUseLimit doesn't work when there are no LOBs during checkpoint)
• commit
• execute a CHECKPOINT again so lob file gets trimmed to usage.
• delete the row you created before checkpoint.
• commit

文章来源: HSQLDB: enable LOB compression for existing database