LZW compression/decompression under low memory conditions

Question

Can anybody give pointers how I can implement lzw compression/decompression in low memory conditions (< 2k). is that possible?

Answer 1

If the choice of compression algorithm isn't set in stone, you might try gzip/LZ77 instead. Here's a very simple implementation I used and adapted once:

ftp://quatramaran.ens.fr/pub/madore/misc/myunzip.c

You'll need to clean up the way it reads input, error handling, etc. but it's a good start. It's probably also way too big if your data AND code need to fit in 2k, but at least the data size is small already.

Big plus is that it's public domain so you can use it however you like!

Answer 2

I have used LZSS. I used code from Haruhiko Okumura as base. It uses the last portion of uncompressed data(2K) as dictionary. The code I linked can be modified to use almost no memory if you have all the uncompressed data available in memory. With a bit of googling you will find that a lot of different implementations.

Answer 3

The lowest dictionary for lzw is trie on linked list. See original implementation in LZW AB. I've rewrited it in fork LZWS. Fork is compatible with compress. Detailed documentation here.

n bit dictionary requires (2 ** n) * sizeof(code) + ((2 ** n) - 257) * sizeof(code) + (2 ** n) - 257.

So:

1. 9 bit code - 1789 bytes.
2. 12 bit code - 19709 bytes.
3. 16 bit code - 326909 bytes.

Please be aware that it is a requirements for dictionary. You need to have about 100-150 bytes for state or variables in stack.

Decompressor will use less memory than compressor.

So I think that you can try to compress your data with 9 bit version. But it won't provide good compression ratio. More bits you have - ratio is better.

Answer 4

My best recommendation is to examine the BusyBox source and see if their LZW implementation is sufficiently small to work in your environment.

Answer 5

typedef   unsigned int     UINT;
typedef   unsigned char    BYTE;

BYTE *lzw_encode(BYTE *input ,BYTE *output, long filesize, long &totalsize);
BYTE *lzw_decode(BYTE *input ,BYTE *output, long filesize, long &totalsize);

Answer 6

Can anybody give pointers how I can implement lzw compression/decompression in low memory conditions (< 2k). is that possible?

Why LZW? LZW needs lots of memory. It is based on a hash/dictionary and compression ratio is proportional to the hash/dictionary size. More memory - better compression. Less memory - output can be even larger than input.

I haven't touched encoding for very long time, but IIRC Huffman coding is little bit better when it comes to memory consumption.

But it all depends on type of information you want to compress.