Bzip3 emerges as high-ratio 'spiritual successor' to BZip2
The new compression tool targets text and code with a modernized Burrows-Wheeler approach, though critics question its benchmark methodology.
Developer Kamila Szewczyk, known online as iczelia, has released bzip3, a compression algorithm designed as a "spiritual successor" to the long-standing BZip2. The tool aims to reclaim competitiveness in high-ratio compression, specifically targeting text-heavy data and source code.
Technically, bzip3 achieves its results by combining several methods: a fast Burrows-Wheeler transform (BWT) powered by libsais, an order-0 context mixing entropy coder, and an RLE with a Lempel Ziv+Prediction pass. According to the project's documentation on GitHub, the tool is licensed under LGPLv3. Performance varies by environment; for instance, x64 Linux builds using clang13 can reach compression speeds of 17MiB/s and decompression speeds of 23MiB/s per thread.
The push for higher ratios
For decades, BZip2 was a staple of Linux environments, but it was eventually superseded by XZ (LZMA2) for its compression ratios and Zstandard (zstd) for its speed. Bzip3 attempts to modernize the BWT approach to challenge these incumbents in specific niches.
In a benchmark involving all versions of Perl5 source code, bzip3 (using the -b 511 setting) produced a compressed size of 546,456,978 bytes. This is significantly smaller than the results for LZMA, which reached 2,056,645,240 bytes, and Zstandard, which reached 3,076,143,660 bytes. Szewczyk notes that "BZip3 excels at compressing text or code."
Industry implications and controversy
If bzip3 can consistently deliver superior ratios with viable decompression speeds, it could become the preferred standard for archiving massive logs and source code repositories. However, the tool's adoption may be hindered by significant resource demands. Some benchmarks indicate memory overhead as high as 18GB, suggesting it may remain a niche tool rather than a general-purpose replacement for zstd.
Furthermore, the tool's claims have met with skepticism from the developer community. On Hacker News, users have described the author's benchmarks as "disingenuous, to the point of looking cherry-picked." Critics argue that bzip3's advantage in the Perl5 test stems from using a massive 512MB block size while comparing it against the default, smaller window sizes of zstd—a configuration that inherently favors BWT-based compressors when handling repetitive data.
What to watch
As the project evolves, the community will be looking for more transparent, standardized benchmarks that use equivalent window sizes across different algorithms. Whether bzip3 can maintain its lead over LZMA and zstd without relying on extreme memory allocations remains to be seen. For now, the tool serves as a potent reminder of the continued utility of the Burrows-Wheeler transform in the era of modern big data.