Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Overview

I am working on a client-server application. The client is written in C++ (working on Windows, planning to support Linux) and the server is a .NET RESTful service. I need to HTTP POST some data to the server. The data is a unicode string that I need to compress using any fast compression algorithm (needs to be light on CPU). On the server I need to decompress raw bytes and get a string.

Problem

I cannot decompress raw bytes and I end up with decompressor's output buffers stay intact.

I tried

I have tried using Google Snappy, there is only a C/C++ version and any .NET ports don't appear to be finished. I also checked LZ4, where compression/decompression works in a single language, but when I try to use both - I cannot decompress data correctly (the decompressed bytes are set to 0s).

Question

Has anyone tried using a fast C/C++ compression and C# decompression? Any recommendations?

Client application shall not use more than 5% of CPU - that's a vague requirement I have. I plan to try gzip and maybe deflate, but I am not sure if I can configure compression levels there or whether it will work in both C/C++ and C#. Client collects some near real time data into a small buffer, when that buffer gets full it shall be compressed and POSTed. The whole OS process shall take less than 5% of the machine's CPU. A buffer is roughly posted 2 times a second which creates a traffic of about 2Kb/sec.

share|improve this question
3  
@oleksii That requirement doesn't make much sense to me. Are you saying it's okay if the compression takes an hour, but during that hour uses only 5% CPU? Why exactly do you have such a requirement? –  svick Jan 7 '13 at 22:48
1  
Gzip is probably what you are after. However, the problems you mentioned are not problems of the compression algorithms but you implementation. You should ask about that specifically on stackoverflow. –  Tamás Szelei Jan 7 '13 at 22:49
1  
I really doubt you will have any CPU load issues using gzip if your data is in the order of 2Kb/sec. –  Carson63000 Jan 8 '13 at 0:38
1  
Your LZ4 issue sounds like maybe an endianess problem or a 64/32-bit mismatch. –  Karl Bielefeldt Jan 8 '13 at 6:42
1  
If you have problems to tranfer binary data over the network, you should work on that, this has nothing to do with compression. If you can't find a better solution, try Convert.ToBase64 / Convert.FromBase64 (see msdn.microsoft.com/en-us/library)/… –  Doc Brown Jan 8 '13 at 7:14

4 Answers 4

If anybody interested I ended up using gzip from zlib. Never figured out why LZ4 doesn't work, as suggested in the comments this could be an endianess problem or a 64/32-bit mismatch. However, I tested this on a single machine compressing and decompressing a local file. The same compilation settings worked for gzip.

C/C++ sample compressor code

int compress_one_file(char *infilename, char *outfilename)
 {
    FILE *infile = fopen(infilename, "rb");
    gzFile outfile = gzopen(outfilename, "wb");
    if (!infile || !outfile) return -1;

    char inbuffer[128];
    int num_read = 0;
    unsigned long total_read = 0, total_wrote = 0;
    while ((num_read = fread(inbuffer, 1, sizeof(inbuffer), infile)) > 0) {
       total_read += num_read;
       gzwrite(outfile, inbuffer, num_read);
    }
    fclose(infile);
    gzclose(outfile);
 }

C# sample decompressor code

public static void Decompress(FileInfo fileToDecompress)
{
    using (FileStream originalFileStream = fileToDecompress.OpenRead())
    {
        string currentFileName = fileToDecompress.FullName;
        string newFileName = currentFileName + ".decompressed";

        using (FileStream decompressedFileStream = File.Create(newFileName))
        {
            using (GZipStream decompressionStream = 
                new GZipStream(originalFileStream, CompressionMode.Decompress))
            {
                decompressionStream.CopyTo(decompressedFileStream);
            }
        }
    }
}
share|improve this answer
    
+1, I will have this in mind when I get a similar task to solve ;-) –  Doc Brown Jan 8 '13 at 16:07

Both LZ4 and Snappy has been ported to .NET recently.

You can find LZ4 .NET port here at http://lz4net.codeplex.com.
You can find Snappy .NET port (actually P/Invoke wrapper) here at http://snappy4net.codeplex.com.

You can also check performance comparison (of .NET ports) here.

To answer your question I did use both to test C++ compression and C# decompression and it works fine.

Disclaimer: yes, both LZ4 and Snappy has been ported/wrapped by me.

share|improve this answer
    
Since you only mention P/Invoke on Snappy, does that mean that the LZ4 port is a real re-implementation? –  Joachim Sauer Feb 21 '13 at 14:58
    
It's MixedMode, C++/CLI, Unsafe C# or Safe C#. You can choose which one you want or even let it choose the best one automatically. –  Milosz Krajewski Feb 21 '13 at 16:40
    
Thanks for your work. I think future generations may benefit from this. I decided to go with standard gzip at the end of the day though. +1. –  oleksii Feb 21 '13 at 18:30

Take a look at 7-zip's LZMA format. here

I have only used their compression utilities (but not their SDK) but they are excellent. The compression format is portable across language and platform and they claim support for both C# and C++

share|improve this answer
    
"zlma"? The compression algorithm is called "LZMA" (and so is one of the file formats). –  Jan Hudec Jan 8 '13 at 7:03
    
@jan -- the fumbling fingers strike again -- will fix. –  James Anderson Jan 8 '13 at 10:36

You must develop a program that will run in the console. Your program will be called "TP2.exe" and has three parameters:

  1. the name of an input file,
  2. a boolean (true or false) indicating whether you want to compress the input file or unpack.
  3. the name of the output file (the result will be written to this file.)

For example, if you run the program with the following command in the console: TP2.exe test.txt true test2.txt Your program will read the file test.txt as input and writes the compressed file test2.txt output. Another example: if you run the program with the following command in the console:

TP2.exe test2.txt false test.txt

Your program will read the file and write test2.txt the decompressed file test.txt output. Your program should use the method of compression / decompression simply discussed in the course is based on the use of a self-organized list.

An example file uncompressed and compressed file example is provided as an example of CLIC. For the work is simple to make, you can assume that the sentences are already cut into words and that there is a single word per line.

Finally, at the implementation level, it is recommended to use a linked list to list the self-organized because it would be more efficient. A good idea would be to use a linked list implementation of the STL library.

share|improve this answer
    
This is the start of a good answer. Please edit your answer and address more of the specifics within the question. –  GlenH7 Feb 5 '13 at 14:49
1  
Sorry, this doesn't answer my question –  oleksii Feb 5 '13 at 17:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.