Compression has never been an easy task for VB developers. Until Windows 8, Microsoft provided very little support for it, and while 3rd-party solutions have long been available, they tend to be either very expensive or very complicated.
For many years, VB6 developers fell back on the classic zLib compression library (http://zlib.net/). zLib is an open-source compression library with a very permissive license, and it is "good enough" for most tasks: decent compression ratios, but with relatively slow compression and decompression speeds.
But in recent years, even zLib has become problematic. The stdcall variant of zLib (usable from VB6) hasn't been updated in over a decade, meaning it contains serious known security bugs. You can always compile your own version of zLib from the latest source code, but the core library definitions are bugged, so this requires a fairly deep knowledge of C and a lot of patience. (Also, zLib's source code hasn't been updated in over three years, and there are a huge number of bug fixes that have yet to be incorporated.)
And even if you do manage to survive all this and successfully build an up-to-date version of zLib, you're still left with compression technology that is now 20+ years old. A great deal of compression research has been done since 1995 (when zLib first released), and there are now open-source libraries that are both much faster than zLib, and with even better compression ratios.
So here's what this small project does: it provides a "Compression" module that wraps four different open-source compression libraries: zLib, zstd, lz4, and lz4_hc. If you're on Windows 8 or newer, it also wraps the four compression algorithms provided by the Windows Compression API. All compression/decompression functions are unified so you simply call a function like "Compress", and pass a "compression library enum" that specifies which compression engine you want to use.
To simplify this demo, precompiled DLLs are provided for each 3rd-party library. Because the 3rd-party libraries are all open-source projects (links to code below), I believe these still meet the vbforums requirements for precompiled binaries. You are of course free to compile these yourself, from the latest source code, but you will need a modern copy of Visual Studio, some knowledge of compiling C code, and you must manually modify the project files to build stdcall variants. (They all default to cdecl, as-is.)
Note that the 3rd-party DLLs are all bare C libraries, so they do not need to be registered on target PCs. Simply ship them in a subfolder of your project - for example, this demo project uses a "\Plugins\" subfolder, and the DLLs are all loaded at run-time via LoadLibrary.
As for the Windows Compression APIs - they are not redistributable, so your clients can only use if they're on Windows 8, 8.1, or 10. I imagine this greatly limits their usefulness for most developers. Sorry. :(
Here is a brief overview of the supported compression libraries, all of which are 100% open-source and free to use in personal or commercial projects (with attribution - see the included license files for details).
- zLib is the classic library you know and love. I've freshly compiled the "newest" version (v1.2.8) for this demo. Despite its age, zLib remains a solid general-purpose compression library, with good compression ratios across a wide variety of data, but with slow compression speeds compared to the competition. zLib supports a "compression level" parameter that allows you to choose a trade-off between faster but worse compression, or slower but better compression. Generally speaking, there is no longer much reason to use zLib, unless you specifically need the DEFLATE algorithm it provides (e.g. to work with .gz files).
- zstd (or "zstandard") is a modern replacement for zLib. It was originally developed by Yann Collet, and its ongoing development is now sponsored by Facebook. It is 100% open-source and BSD licensed. zstd is significantly faster than zLib at both compression and decompression, and it also achieves better compression ratios. It provides a "compression level" parameter just like zLib, but with a much wider range, including extremely slow speeds but extremely good compression ratios if you need that sort of thing. For most users, zstd could replace zLib in their existing projects, and they'd immediately get a "free" performance boost from it.
- lz4 is a real-time compression engine that emphasizes performance above all else. It was also developed by Yann Collet, and it is also 100% open-source and BSD licensed. lz4 is so fast that it is now used for OS-level compression (Linux), file system compression (OpenZFS, SquashFS), database compression (MySQL), RAM caching (Emscripten, ZRam), and a whole bunch of video games (Battlefield 4, Black Ops 3, etc). LZ4's speed comes at a trade-off, however - it does not compress as well as zLib or zstd on most data. It also provides an adjustable "compression level" parameter, but instead of providing "slower but better" compression as you increase this value, lz4 provides "faster but worse" compression. It is the best solution when speed is paramount. (For example, lz4 is one of the few algorithms fast enough to provide a performance benefit vs raw uncompressed data when reading/writing to a hard drive.)
- lz4_hc comes "for free" with lz4. It is a "high-compression" variant of lz4, with much better compression ratios but much slower compression speeds. Decompression speed remains the same. It is a good solution if you have all the time in the world for compression, but you still require very fast decompression. (This is the version that video games use, for example.)
- The Windows Compression API available in Windows 8+ provides four different compression algorithms: MSZIP, XPRESS, XPRESS_HUFF, and LZMS. The pros and cons of each algorithm are described on MSDN. Unlike the 3rd-party libraries, these algorithms do not support variable compression levels, so you are always stuck with the default behavior.
The included demo project allows you to compare compression speed, decompression speed, and compression ratio across all libraries. A baseline comparison of "no compression" is also provided, which measures timing against bare RtlMoveMemory calls. I've included a few multilanguage XML files for comparison (because they're small enough to fit inside vbforum size limits), but for best results, you should test some of your own files. Just drag-and-drop a file onto the project window to run an automated test across all libraries.
![Name: Compression_results.jpg
Views: 212
Size: 49.9 KB]()
Checkboxes allow you to toggle various test settings. Note that by default, libraries are tested at their default compression level. Different libraries default to different settings - for example, zLib defaults to a "good but slow" setting, while zstd defaults to its "fastest possible" setting - making comparisons somewhat tricky. To help remedy this, I've provided a checkbox that automatically tests each library at its minimum, maximum, and "middle" settings. This gives a good overview of what each library is capable of.
At present, the Compression module operates entirely on byte arrays and/or bare pointers (passed using VarPtr()). This makes it trivial to compress source data of any size or type. Specialized functions for Strings or other data types could always be added, but for now, those are left as an exercise to the reader.
Bug reports and feedback welcome, of course. Thank you to everyone who has contributed feedback so far.
Updates:
Download here:
For many years, VB6 developers fell back on the classic zLib compression library (http://zlib.net/). zLib is an open-source compression library with a very permissive license, and it is "good enough" for most tasks: decent compression ratios, but with relatively slow compression and decompression speeds.
But in recent years, even zLib has become problematic. The stdcall variant of zLib (usable from VB6) hasn't been updated in over a decade, meaning it contains serious known security bugs. You can always compile your own version of zLib from the latest source code, but the core library definitions are bugged, so this requires a fairly deep knowledge of C and a lot of patience. (Also, zLib's source code hasn't been updated in over three years, and there are a huge number of bug fixes that have yet to be incorporated.)
And even if you do manage to survive all this and successfully build an up-to-date version of zLib, you're still left with compression technology that is now 20+ years old. A great deal of compression research has been done since 1995 (when zLib first released), and there are now open-source libraries that are both much faster than zLib, and with even better compression ratios.
So here's what this small project does: it provides a "Compression" module that wraps four different open-source compression libraries: zLib, zstd, lz4, and lz4_hc. If you're on Windows 8 or newer, it also wraps the four compression algorithms provided by the Windows Compression API. All compression/decompression functions are unified so you simply call a function like "Compress", and pass a "compression library enum" that specifies which compression engine you want to use.
To simplify this demo, precompiled DLLs are provided for each 3rd-party library. Because the 3rd-party libraries are all open-source projects (links to code below), I believe these still meet the vbforums requirements for precompiled binaries. You are of course free to compile these yourself, from the latest source code, but you will need a modern copy of Visual Studio, some knowledge of compiling C code, and you must manually modify the project files to build stdcall variants. (They all default to cdecl, as-is.)
Note that the 3rd-party DLLs are all bare C libraries, so they do not need to be registered on target PCs. Simply ship them in a subfolder of your project - for example, this demo project uses a "\Plugins\" subfolder, and the DLLs are all loaded at run-time via LoadLibrary.
As for the Windows Compression APIs - they are not redistributable, so your clients can only use if they're on Windows 8, 8.1, or 10. I imagine this greatly limits their usefulness for most developers. Sorry. :(
Here is a brief overview of the supported compression libraries, all of which are 100% open-source and free to use in personal or commercial projects (with attribution - see the included license files for details).
- zLib is the classic library you know and love. I've freshly compiled the "newest" version (v1.2.8) for this demo. Despite its age, zLib remains a solid general-purpose compression library, with good compression ratios across a wide variety of data, but with slow compression speeds compared to the competition. zLib supports a "compression level" parameter that allows you to choose a trade-off between faster but worse compression, or slower but better compression. Generally speaking, there is no longer much reason to use zLib, unless you specifically need the DEFLATE algorithm it provides (e.g. to work with .gz files).
- zstd (or "zstandard") is a modern replacement for zLib. It was originally developed by Yann Collet, and its ongoing development is now sponsored by Facebook. It is 100% open-source and BSD licensed. zstd is significantly faster than zLib at both compression and decompression, and it also achieves better compression ratios. It provides a "compression level" parameter just like zLib, but with a much wider range, including extremely slow speeds but extremely good compression ratios if you need that sort of thing. For most users, zstd could replace zLib in their existing projects, and they'd immediately get a "free" performance boost from it.
- lz4 is a real-time compression engine that emphasizes performance above all else. It was also developed by Yann Collet, and it is also 100% open-source and BSD licensed. lz4 is so fast that it is now used for OS-level compression (Linux), file system compression (OpenZFS, SquashFS), database compression (MySQL), RAM caching (Emscripten, ZRam), and a whole bunch of video games (Battlefield 4, Black Ops 3, etc). LZ4's speed comes at a trade-off, however - it does not compress as well as zLib or zstd on most data. It also provides an adjustable "compression level" parameter, but instead of providing "slower but better" compression as you increase this value, lz4 provides "faster but worse" compression. It is the best solution when speed is paramount. (For example, lz4 is one of the few algorithms fast enough to provide a performance benefit vs raw uncompressed data when reading/writing to a hard drive.)
- lz4_hc comes "for free" with lz4. It is a "high-compression" variant of lz4, with much better compression ratios but much slower compression speeds. Decompression speed remains the same. It is a good solution if you have all the time in the world for compression, but you still require very fast decompression. (This is the version that video games use, for example.)
- The Windows Compression API available in Windows 8+ provides four different compression algorithms: MSZIP, XPRESS, XPRESS_HUFF, and LZMS. The pros and cons of each algorithm are described on MSDN. Unlike the 3rd-party libraries, these algorithms do not support variable compression levels, so you are always stuck with the default behavior.
The included demo project allows you to compare compression speed, decompression speed, and compression ratio across all libraries. A baseline comparison of "no compression" is also provided, which measures timing against bare RtlMoveMemory calls. I've included a few multilanguage XML files for comparison (because they're small enough to fit inside vbforum size limits), but for best results, you should test some of your own files. Just drag-and-drop a file onto the project window to run an automated test across all libraries.
Checkboxes allow you to toggle various test settings. Note that by default, libraries are tested at their default compression level. Different libraries default to different settings - for example, zLib defaults to a "good but slow" setting, while zstd defaults to its "fastest possible" setting - making comparisons somewhat tricky. To help remedy this, I've provided a checkbox that automatically tests each library at its minimum, maximum, and "middle" settings. This gives a good overview of what each library is capable of.
At present, the Compression module operates entirely on byte arrays and/or bare pointers (passed using VarPtr()). This makes it trivial to compress source data of any size or type. Specialized functions for Strings or other data types could always be added, but for now, those are left as an exercise to the reader.
Bug reports and feedback welcome, of course. Thank you to everyone who has contributed feedback so far.
Updates:
Code:
12 December 2016: fixed text box scroll behavior in test program.
(Thank you to Steve Grant for reporting.)
Added functions to report each library's default, minimum, and maximum compression settings.
Also updated the test framework to let you test these settings.
(Thank you to Arnoutdv for the suggestion.)
Added support for the Windows Compression API, available on Win 8 or newer.
(Thank you to dilettante for the idea.)
11 December 2016: recompile liblz4.dll to fix issues on old OS versions.
(Thank you to Steve Grant for reporting.)
10 December 2016: initial release