一个并行的重复数据删除和压缩实用程序

Pcompress is a high-performance utility to do compression/decompression and deduplication in parallel by splitting input data into chunks and providing strong integrity checking. Chunk splitting can happen at fixed points or variable content-defined boundaries based on a rolling hash. It has a modular structure and includes support for multiple algorithms like LZMA, Bzip2, PPMD, LZ4, LZFX etc., with KECCAK, BLAKE2, SHA-256/512 chunk checksums. SSE optimizations for the bundled LZMA are included. Performance scales with the number of cores. Depending on the size of the data and the number of chunks it can scale to hundreds of cores. It also implements chunk-level Content-Aware Deduplication and Delta Compression features with variable block splitting based on a polynomial rolling hash. It has low metadata overhead and overlaps I/O and compression to achieve maximum parallelism. It support

pcompress-2.4
none_compress.c
allocator.h
crypto
ppmd_compress.c
main.c
test
Changelog
compressed_file_format.txt
COPYING
zlib_compress.c
lzfx
utils
config
delta2
main.o
adaptive_compress.c
pcompress.c
README.md
transpose
allocator.c
lzma_compress.c
lz4
rabin
pcompress.h
lzma
bsdiff
bzip2_compress.c
lzp
Makefile.in
lzfx_compress.c
libbsc_compress.c
COPYING.LESSER
INSTALL
lz4_compress.c