uc2_sha256: pure-C FIPS 180-4 implementation, one-shot and incremental
API, validated against published vectors (empty, abc, 56-byte,
1M 'a', byte-by-byte, every-split-point boundary).
uc2_ots: parser, serializer, and walker for the standard .ots binary
format. Strict canonical varint with 64-bit overflow check, depth-
bounded recursion, varbytes cap, max-digest cap. Walker supports
the calendar-path subset (APPEND, PREPEND, SHA256); proofs that
include other crypto ops (SHA1, RIPEMD160, KECCAK256) are accepted
as structurally valid but flagged for follow-up via the standard
'ots verify'.
UC2-OTS trailer: magic-bracketed sidecar appended after the recorded
archive bytes. Reverse-scan-safe; original UC2 Pro reader ignores
trailing bytes past its recorded length so backward compatibility is
preserved. Layout (all integers little-endian uint32):
front-magic + version + archive_len + proof_len + proof
+ proof_len + back-magic.
CLI: --ots-attach validates that the proof's leaf digest equals
SHA-256(archive[0..archive_len)) before appending and refuses to
overwrite an existing trailer unless -f is given. --ots-extract
writes the proof verbatim, byte-compatible with the standard
'ots verify'. --ots-info parses and prints the leaf, archive-match
status, and attestation list. uc2 -t recomputes the archive
SHA-256 and walks the proof.
Tests: 17 OTS unit tests (varint round-trip, canonical/overflow
rejection, file-envelope round-trip, walker on append/sha256/
sibling/unsupported-op/truncated/trailing-garbage, attest_name,
trailer round-trip + corruption rejection in 4 scenarios).
Plus an optional ctest target ots_cross_check that round-trips
the .ots through python-opentimestamps when the package is
installed; skipped (return code 77) otherwise.
rANS entropy coding is now a usable compression option:
uc2 -w -L 8 archive.uc2 files... # rANS Tight
uc2 archive.uc2 # decompresses (auto-detects method)
Block format for method 10:
[block-present:1] [nsyms:16] [rans_len:16]
[freq_table:344x12bits] [rans_data] [extra_bits]
Symbol IDs (0-343) encoded with rANS for near-optimal entropy.
Extra bits (distance/length parameters) stored separately in the
bitstream, preserving the existing variable-length encoding.
Integration:
- Compressor: flush_block_rans() dispatched when level >= 6
- Decompressor: decompressor_rans() dispatched for method 10
- CLI: levels 6-9 map to rANS Fast/Normal/Tight/Ultra
- COMPRESS records store method=10 for rANS files/cdir
- End-to-end round-trip verified (create/list/extract/verify)
Levels 2-5 (Huffman) remain the default for backward compatibility
with the original UC2 Pro.
UC2 now talks during operations, continuing the original's tradition:
$ uc2 -w archive.uc2 files...
UC2 compression level: Tight
Created archive.uc2 (3 files, 0 dirs, 1 master, 4096 bytes)
Everything went OK
$ uc2 -t archive.uc2
Testing archive integrity...
Everything went OK
$ uc2 -h
UC2 3.0.0 (UltraCompressor II)
"Fast, reliable and superior compression."
Messages are warm, confident, and slightly quirky — not a fun flag,
just how UC2 talks. Suppressed by -q for scripting:
$ uc2 -qt archive.uc2 # silent, exit code only
Compression level names: Fast (2), Normal (3), Tight (4), Ultra (5).
"Everything went OK" directly from the original (MAIN.CPP:918).
Completes Phase 2 (Original Compression Engine).
Replace the fixed first-4KB FNV-1a prefix matching with content-defined
chunking. Files are now split into ~4KB CDC chunks (Gear rolling hash),
each chunk hashed with FNV-1a. Files sharing any chunk hash are grouped
for master-block deduplication.
This detects shared content at ANY position in the files, not just
identical file prefixes. The improvement matters for:
- Patched executables (same code, different version strings)
- Edited documents (same body, different headers)
- Similar data files (shared structures at varying offsets)
The fallback master assignment (for backward compat with original UC2
Pro) still applies to ungrouped files, ensuring all files use custom
master indices (>= 2).
All 7 tests pass including master dedup round-trip and CDC unit tests.
Always assign custom master indices (>= FIRSTMASTER=2) to all files,
never SuperMaster (index 0). The original's ExtractFiles() routes
SuperMaster files through a code path that hangs. The original itself
never uses SuperMaster in file COMPRESS records — it always creates
at least one custom master, even for archives without dedup groups.
For ungrouped files, a default custom master is built from the largest
file's first 64KB. All files reference this master, matching the
original's archive structure.
The automated DOSBox-X test now validates multi-file round-trip in
both directions: 4 files UC2 v3 -> original, 5 files original -> UC2 v3.
All content verified byte-for-byte.
Investigation of multi-file extraction hang:
- LocMacNtx(0) returns VNULL for SuperMaster, but V(VNULL) is safe
in the original's virtual memory system (not a null dereference)
- 0xDEDEDEDE is a legacy sentinel for old archives, not the standard
value (original uses masterPrefix=0)
- The hang is in the file data decompression phase, not the cdir
parsing (listing of multi-file archives works correctly)
Multi-file backward compat remains under investigation.
The original UC2 Pro handles archives differently based on versionMadeBy:
300 causes multi-file listing to fail, while 203 (the original's own
version) works correctly. Also write method=compression_level in file
COMPRESS records instead of hardcoded 1.
Combined with the earlier csize=0 and default-tree fixes, single-file
UC2 v3 archives are now fully backward compatible with the original
UC2 Pro (listing + extraction verified). Multi-file archives can be
listed but extraction still hangs — under investigation.
Root cause: the original UC2 Pro expects csize=0 in the cdir COMPRESS
record (it ignores the field entirely). UC2 v3 was writing the actual
compressed size, which confused the original's archive reader.
Additional changes:
- Use default Huffman tree for all blocks (ensures tree encoding compat)
- Write method=compression_level in cdir COMPRESS (was hardcoded to 1)
- Add tests/scripts/bitdump.py for bit-level bitstream analysis
Single-file UC2 v3 archives are now fully readable by the original UC2
Pro (listing and extraction verified in DOSBox-X). Multi-file archives
still hang — the cdir bitstream decodes correctly in our Python analyzer
but fails in the original's ASM decompressor kernel. Investigation
continues; the bitdump.py tool enables targeted comparison.
Recursive directory scanning with parent/child ID tracking, directory
entries in the central directory (OSMETA + DIRMETA + EXTMETA long name
tags), and a CLI round-trip test verifying nested directory hierarchies.
Content-fingerprint grouping via FNV-1a hash of file headers: files
sharing identical first 4096 bytes are assigned a custom master block
built from the largest file in the group. Masters are compressed with
SuperMaster and written as MASMETA records in the central directory.
Files below 1 KB or without a group continue using the SuperMaster.
Includes CLI integration test and documentation updates (format spec,
usage, roadmap).
Restructure compat layer: #include_next headers moved to posix/ for
MinGW, new standalone headers in msvc/ for MSVC (unistd.h, utime.h,
getopt.h). Add getopt() implementation, chmod/unlink/chdir compat
functions, MSVC CRT initializer for UTF-8 console, _pgmptr fix.
Add a DJGPP CMake toolchain file and DOS compatibility layer (err.h,
fnmatch, getprogname/setprogname) so UC2 builds as a native DOS
executable via cross-compilation from Linux. The toolchain works
around a baked-in /usr/include in the DJGPP GCC binary by using -I
instead of -isystem to ensure DJGPP headers take precedence.
Add GitHub Actions CI workflow that builds and smoke-tests on both
ubuntu-latest and macos-latest.
Decompression MVP based on Jan Bobrowski's portable unuc2/libunuc2.
CMake build system targeting Linux (GCC/Clang) with MSVC fallback.
Includes original UC2 source by Nico de Vries and unuc2-0.6 for reference.
Eremey Valetov