Best Compression algorithm for a sequence of integers

Question

I have a large array with a range of integers that are mostly continuous, eg 1-100, 110-160, etc. All integers are positive. What would be the best algorithm to compress thi

Answer 1

The basic idea you should probably use is, for each range of consecutive integers (I will call these ranges), to store the starting number and the size of the range. For example, if you have a list of 1000 integers, but there are only 10 separate ranges, you can store a mere 20 integers (1 start number and 1 size for each range) to represent this data which would be a compression rate of 98%. Fortunately, there are some more optimizations you can make which will help with cases where the number of ranges is larger.

1. Store the offset of the starting number relative to the previous starting number, rather than the starting number itself. The advantage here is that the numbers you store will generally require less bits (this may come in handy in later optimization suggestions). Additionally, if you only stored the starting numbers, these numbers would all be unique, while storing the offset gives a chance that the numbers are close or even repeat which may allow for even further compression with another method being applied after.

2. Use the minimum number of bits possible for both types of integers. You can iterate over the numbers to obtain the largest offset of a starting integer as well as the size of the largest range. You can then use a datatype that most efficiently stores these integers and simply specify the datatype or number of bits at the start of the compressed data. For example, if the largest offset of a starting integer is only 12,000, and the largest range is 9,000 long, then you can use a 2 byte unsigned integer for all of these. You could then cram the pair 2,2 at the start of the compressed data to show that 2 bytes is used for both integers. Of course you can fit this information into a single byte using a little bit of bit manipulation. If you are comfortable with doing a lot of heavy bit manipulation you could store each number as the minimum possible amount of bits rather than conforming to 1, 2, 4, or 8 byte representations.

With those two optimizations lets look at a couple of examples (each is 4,000 bytes):

1. 1,000 integers, biggest offset is 500, 10 ranges
2. 1,000 integers, biggest offset is 100, 50 ranges
3. 1,000 integers, biggest offset is 50, 100 ranges

WITHOUT OPTIMIZATIONS

1. 20 integers, 4 bytes each = 80 bytes. COMPRESSION = 98%
2. 100 integers, 4 bytes each = 400 bytes. COMPRESSION = 90%
3. 200 integers, 4 bytes each = 800 bytes. COMPRESSION = 80%

WITH OPTIMIZATIONS

1. 1 byte header + 20 numbers, 1 byte each = 21 bytes. COMPRESSION = 99.475%
2. 1 byte header + 100 numbers, 1 byte each = 101 bytes. COMPRESSION = 97.475%
3. 1 byte header + 200 numbers, 1 byte each = 201 bytes. COMPRESSION = 94.975%