I- Introduction

This library contains many methods of data compression. They can be included in a program, to provide fast, embedded compression of data files for example. They provide also features like CRC calculus on 16 or 32 bits, as well as Hash coding techniques.

The following compression techniques are provided:

1. Huffman (statistic method on bytes with header)
2. Adaptative Huffman (statistic method on bytes blocks without header)
3. LZW (Lempel Ziv Welch) (dictionnary method with variable length codes and improvments to the "standard" algorithm)
4. LZSS (Lempel Ziv Storer Szymanski) (dictionnary method with circular buffer of data and improvements)

At the moment, two of these techniques can compress data blocks, instead of files: Lzw and Lzss. The drawback is, naturally, a memory cost. It's use is restricted to memory-consuming programs (let's say, more than 512Kb of data), and to selective compression in files (for example, to compress only given block of data in a file, which can be useful too).

II- File compression and decompression

2.1- Generic interface

All file compression techniques have a similar interface. They take three arguments:

1. The file descriptor of the input file
2. The file descriptor of the output file
3. A pointer to a float for the compression rate

All file decompression techniques have a similar interface. They take two arguments:

1. The file descriptor of the input file
2. The file descriptor of the output file

The compression and decompression functions return a value of 0 on success, and a value of -1 on failure. Error message are displayed if it's a standard error (I/O, memory allocation for example). If compression is successful, the gain is wrote in the float value (between -1.0 and 1.0).

2.2- Identifying a compressed file

To identify a file, you can use the IdentifyMethod function. This function take only one argument: a file descriptor. This function returns the number of the method. Defines of standard methods are in the crunch.h file. The following defines are made at the moment:

• METHOD_UNKNOWN if the method number found is not correct.
• METHOD_HUFFMAN for the Huffman method.
• METHOD_AD_HUFF for the Adaptative Huffman method.
• METHOD_LZW for the LZW method.
• METHOD_LZSS for the LZSS method.

2.3- Checking a compressed file

To check a file, you can use the CheckFile function. This function take only one argument: a file descriptor. This function checks if a compressed file is correct. This includes checking the header's CRC, the signature, and the length of compressed file to determine if the file is correct. If it's correct, the method index is returned. If it's not correct, a negative number is returned, relative to the kind of error found. The following errors are reported:

• CHECK_ERROR for standard I/O error (cannot read, cannot do a seek on the file...).
• CHECK_INVALID if the signature wasn't found in the file (four bytes at the beginning of compressed file).
• CHECK_CORRUPT if the compressed file is truncated, or if it's length was modified.
• CHECK_CRC if the CRC on the header or the file was found incorrect.
• CHECK_VERSION if the version found is greater than the version the library was compiled with. Note that this value is returned only if major or minor version values are different. If the revision numbers are different, this error is not emitted.

2.4- Using CRC functions

There are two functions in the library that provide a simple way to compute a CRC value (on 16 or 32 bits). These functions are CRC16 and CRC32 functions. They take two arguments:

1. A pointer on unsigned char for the memory zone
2. A long indicating the length of the memory zone

2.5- Using Hash functions

There are two functions in the library that provide hash values for memory zones. They take also two arguments (see the CRC functions). The first hash function was taken from the "Ansi C" from B. Kernighan and D. Ritchie. It returns a value between 0 and 100 (thus you should have a lot of collisions, as it is not very efficient) and is called SimpleHash. The other one is based on Weinberger Hash function and return an unsigned long. It is not so far from a CRC computing (though it can't detect errors). The simplest way to use this hash value is to use a modulo on your array size. Here's an example.

III- Block compression and decompression

3.1- Introduction

This part of the library provides compression/decompression of data blocks. This allows "on the fly" compression of data blocks, before sending them in a file or in a socket for example. The interface is a bit tricky, because of the number of parameters, as you will see.

Currently, only Lzw and Lzss provide block compression, because they don't need a header like Huffman for example.

3.2- Generic interface

All block compression/decompression functions share a generic interface. They take five parameters:

1. A pointer to a context
2. A pointer to byte for the input block
3. A long for the size of the input block
4. A pointer to byte for the output block
5. A long for the size of the output block

• The context must be allocated, and initialized.
• The output block must be allocated.
• The output block should have the same size as input block.
• The size of output block must indicate the size of the output block.

3.3- Context

A context is a container of variable used by the compression function. It contains data such as the dictionnary, the number of wrote bytes, an index table etc etc... It should be allocated at the beginning of the compression sequence, and freed at the end of it, because it's a big structure (it weights 53Kb in Lzw and 78Kb in Lzss. This doesn't take into account the dynamic memory allocation of Lzw). Before the beginning of an encoding sequence, it must be initialized (calling either LzwInitContext, or LzssInitContext). The structure can be used for more than one call to the compression function. But this means that the same must be done when uncompressing. Otherwise, the dictionnary won't be the same, and you will really strange results (not the original data by the way).

You have to keep track, for each compressed block, of the original block size (before compression). This is useful when you uncompress it. Though you can put a greater value for the out block size than original (the function will uncompress only what was compressed).

3.4- Example

Let's take an example of what to do when you want to compress data. Suppose you have a huge structure that we wil call Foo, that you want to compress in memory. The following sequence can be used:

byte *CompressData(Foo *f, dword *final_size)
{
    LzwContext context;
    byte *compressed;

    /* Allocate the block that will contain compressed data */
    compressed = Malloc(sizeof(Foo));
    /* Init the context and size */
    LzwInitContext(&context);
    (*final_size) = sizeof(Foo);
    /* Compression */
    if (LzwBlockEncode(&context, (byte*)f, sizeof(Foo), compressed, final_size) == -1) {
	/* Cannot compress: we free the allocated zone and return */
	free(compressed);
	return NULL;
    }
    /* Successful compression: we realloc the block */
    compressed = realloc(compressed, (*final_size));
    /* and we return a pointer to compressed data. */
    return compressed;
}