Windows MSVC test_delta failed with STATUS_HEAP_CORRUPTION (0xc0000374).
ASan/UBSan on Linux finds nothing; the most likely Windows-specific
issue is malloc(0) in uc2_delta_apply when the target is empty
(test_empty_target). Bump to malloc(1) to get a canonical
free()-safe pointer.
Add fflush(stdout) between tests so the next CI run shows which
test (if any) still fails on Windows.
Round 2 of c67b631 cleanup. After the dirent + utime fixes, the
MSVC link surface still had:
- LNK2005 'fopen already defined': dropped g_fopen so we no longer
override the SDK's fopen. UTF-8 paths still work on Windows 10
with the active-codepage manifest; non-Unicode codepages will see
ANSI translation. This is good enough for the public release; a
full UTF-8 fopen wrapper can be added later if needed.
- LNK2019 'unresolved S_ISDIR / S_ISREG': MSVC's <sys/stat.h> defines
_S_IFDIR / _S_IFREG but not the POSIX S_IS* macros. Add them in
the unistd.h shim (which main.c already pulls).
- LNK1181 'cannot open input file m.lib': test_merkle and test_rans
linked libm unconditionally. Math is in the default CRT on MSVC;
link 'm' only on non-Windows.
- 'unistd.h' not found in test_blockstore.c: it actually only needs
getpid(). Use <process.h> + #define getpid _getpid on MSVC, keep
<unistd.h> elsewhere.
Win10 SDK 26100's <sys/utime.h> provides an inline utime() wrapper
that forwards to _utime32 (ANSI-codepath, not UTF-8). Defining our
own utime() collided with the inline (C2084: function already has a
body).
Rename the compat function to compat__utime and have the utime.h
shim translate utime -> compat__utime via #define so UC2's UTF-8
paths still go through compat__wpath at the call site.
Two pre-existing issues that have failed every Windows CI run since
2026-03-12 (when archive creation added <dirent.h> via 9525a81):
1. cli/src/main.c:33 includes <dirent.h>, which MSVC does not
provide. Add a minimal shim under cli/src/compat/include/msvc/
exposing DIR / struct dirent / opendir / readdir / closedir.
The implementation in compat_win32.c uses FindFirstFileW /
FindNextFileW and round-trips filenames through UTF-8 to match
the rest of the compat layer.
2. cli/src/compat/compat_win32.c:314 redefined struct utimbuf, which
collides with the Win10 SDK 10.0.26100+ <sys/utime.h>. The local
utime.h shim now pulls <sys/utime.h> directly so utimbuf comes
from the system, and compat_win32.c stops redefining it. An
opt-in _COMPAT_UTIMBUF_FALLBACK is provided for older SDKs that
hide utimbuf behind _CRT_DECLARE_NONSTDC_NAMES.
Linux and macOS builds continue to pass; this commit only touches
the MSVC compat path. Closes git-bug c67b631.
contrib/libarchive/ contains a design doc, an annotated skeleton of
archive_read_support_format_uc2(), and a CMake target that gates the
build on -DUC2_BUILD_LIBARCHIVE_PLUGIN=ON plus find_package(LibArchive).
The skeleton has the five required callbacks (bid, read_header,
read_data, read_data_skip, cleanup) with TODO markers at each
implementation point. The bid function has the magic-byte check
ready; the rest call into libuc2 for parsing and decompression.
libarchive's read-format API is internal; an out-of-tree .so cannot
be loaded into an unmodified libarchive. The integration plan in
contrib/libarchive/README.md is to upstream the file as a PR against
libarchive/libarchive. Full implementation is tracked as
git-bug b0b06a5.
The 2015 LGPL re-release was initiated by Vladislav Sagunov's request
to Nico de Vries; de Vries personally re-released the source. Earlier
CREDITS.md (and the inherited license-audit.md) misattributed this to
Danny Bezemer. Bezemer was a co-developer of UC2 during the original
1992-1996 work, per de Vries's release notes.
Add docs/blog/uc2-revival-writeup.md (1200 words) drafted from the
codex-reviewed revision: tightened scoping language ('byte-bitstream-
compatible' rather than ambiguous 'byte-compatible'), removed
overclaims about borg/restic/Kopia equivalence, dropped speculative
PQ/Filecoin/ZK details, and trimmed the demo to one compatibility
example + one OTS example.
Closes git-bug 98904d0.
Add SPDX-License-Identifier to every source file in lib/ and cli/.
Files derived from Bobrowski's libunuc2 retain LGPL-3.0-only;
cli/src/main.c (derived from his GPL-licensed unuc2 tool) and all
new Phase 2-7 work by Valetov are GPL-3.0-or-later. No silent
LGPL-to-GPL upgrade has been applied.
CREDITS.md now lists each Bobrowski-derived file specifically rather
than crediting libunuc2 as generic 'inspiration'.
docs/license-audit.md records the full per-file provenance table,
the LGPL-3.0 -> GPL-3.0 chain rationale (LGPL sec. 4 Combined Works
is the operative clause; LGPL sec. 3 single-direction upgrade is
documented but not exercised), and confirms that:
- the 2015 LGPL-3.0 release in original/UC2_source/ is preserved
unchanged;
- the 2020-2021 LGPL/GPL releases in original/unuc2-0.6/ are preserved
unchanged;
- lib/src/super.bin is bit-identical to upstream and to de Vries's
1992 distribution data.
uc2_sha256: pure-C FIPS 180-4 implementation, one-shot and incremental
API, validated against published vectors (empty, abc, 56-byte,
1M 'a', byte-by-byte, every-split-point boundary).
uc2_ots: parser, serializer, and walker for the standard .ots binary
format. Strict canonical varint with 64-bit overflow check, depth-
bounded recursion, varbytes cap, max-digest cap. Walker supports
the calendar-path subset (APPEND, PREPEND, SHA256); proofs that
include other crypto ops (SHA1, RIPEMD160, KECCAK256) are accepted
as structurally valid but flagged for follow-up via the standard
'ots verify'.
UC2-OTS trailer: magic-bracketed sidecar appended after the recorded
archive bytes. Reverse-scan-safe; original UC2 Pro reader ignores
trailing bytes past its recorded length so backward compatibility is
preserved. Layout (all integers little-endian uint32):
front-magic + version + archive_len + proof_len + proof
+ proof_len + back-magic.
CLI: --ots-attach validates that the proof's leaf digest equals
SHA-256(archive[0..archive_len)) before appending and refuses to
overwrite an existing trailer unless -f is given. --ots-extract
writes the proof verbatim, byte-compatible with the standard
'ots verify'. --ots-info parses and prints the leaf, archive-match
status, and attestation list. uc2 -t recomputes the archive
SHA-256 and walks the proof.
Tests: 17 OTS unit tests (varint round-trip, canonical/overflow
rejection, file-envelope round-trip, walker on append/sha256/
sibling/unsupported-op/truncated/trailing-garbage, attest_name,
trailer round-trip + corruption rejection in 4 scenarios).
Plus an optional ctest target ots_cross_check that round-trips
the .ots through python-opentimestamps when the package is
installed; skipped (return code 77) otherwise.
uc2_merkle_root() and uc2_dict_id() return uint64_t; the int _r
temporaries from 97e05ad's NDEBUG fix truncated the high 32 bits.
Under Release the assertion was stripped, hiding the bug; under
Debug the truncated _r never matched the second uint64_t call.
- Mark test corpus/archives as binary in .gitattributes to prevent
line ending conversion on CI (fixes extract test size mismatch)
- Fix alignment-unsafe struct cast in uc2_dict.c serialize/deserialize
(use memcpy-based byte access instead; fixes SEGFAULT on CI)
- Fix formatting issues in docs
README: rewritten to reflect current state (Phases 1-4 complete),
with feature list, compression levels table, full usage examples,
and project structure overview.
CREDITS: expanded with detailed attribution for all contributions
to the UC2 v3 revival (compression engine, dedup libraries, backward
compat, testing infrastructure).
New library (uc2_blake3.h / uc2_blake3.c) for Phase 7:
- Pure C BLAKE3 implementation (~300 lines)
- 256-bit (32-byte) digests using BLAKE2s round function
- Bao tree hashing structure for inputs > 1024 bytes
- Incremental API (init/update/final) and one-shot helper
- Constant-time hash comparison (timing-attack resistant)
Suitable for content verification, block integrity checking,
and content-addressable storage (replacing or supplementing
the 64-bit FNV-1a hashes used in Merkle DAG and block store).
7 unit tests:
- Empty input, determinism, collision avoidance
- Incremental vs one-shot consistency
- Single-byte-at-a-time update consistency
- Avalanche effect (1-bit change → ~50% output bits flip)
- Constant-time comparison
New library (uc2_preprocess.h / uc2_preprocess.c) for Phase 4:
BCJ (Branch/Call/Jump) filter:
- E8/E9 x86 address normalization (relative → absolute)
- Makes calls to the same function from different locations produce
identical byte sequences, improving LZ77 matching
- Round-trip verified; address normalization confirmed
BWT (Burrows-Wheeler Transform):
- Suffix-array-based forward transform
- LF-mapping inverse with reverse reconstruction
- Groups similar contexts for better entropy coding
- Round-trip verified for text ("banana") and binary data
Delta filter:
- Byte-wise delta encoding with configurable stride
- Stride 1 for sequential data, stride 2+ for interleaved channels
- Constant-delta sequences (arithmetic progressions) reduce to
repeated single values
Content detection:
- Automatic content type identification (text/x86/structured/binary)
- MZ/PE and ELF header recognition for x86
- Printable ASCII ratio for text detection
11 unit tests covering all filters and detection.
New library (uc2_dict.h / uc2_dict.c) formalizes master blocks as
proper dictionaries with:
- 64-bit content hash ID (FNV-1a) for cross-archive sharing
- 32-bit integrity checksum with verification
- Portable serialization format (24-byte header + data)
- Deserialization with magic number and size validation
Combined with the block store (uc2_blockstore.h), this enables
distributed dedup: archives in different locations can reference
shared dictionaries by content hash, with integrity verification
before decompression.
6 unit tests including serialization round-trip, corruption
detection, and bad-magic rejection.
Also added plausible deniability (multi-archive with separate
passwords) to Phase 5 roadmap.
rANS entropy coding is now a usable compression option:
uc2 -w -L 8 archive.uc2 files... # rANS Tight
uc2 archive.uc2 # decompresses (auto-detects method)
Block format for method 10:
[block-present:1] [nsyms:16] [rans_len:16]
[freq_table:344x12bits] [rans_data] [extra_bits]
Symbol IDs (0-343) encoded with rANS for near-optimal entropy.
Extra bits (distance/length parameters) stored separately in the
bitstream, preserving the existing variable-length encoding.
Integration:
- Compressor: flush_block_rans() dispatched when level >= 6
- Decompressor: decompressor_rans() dispatched for method 10
- CLI: levels 6-9 map to rANS Fast/Normal/Tight/Ultra
- COMPRESS records store method=10 for rANS files/cdir
- End-to-end round-trip verified (create/list/extract/verify)
Levels 2-5 (Huffman) remain the default for backward compatibility
with the original UC2 Pro.
New library (uc2_rans.h / uc2_rans.c) — table-based range Asymmetric
Numeral Systems (rANS) entropy coder:
- 32-bit state with 12-bit probability precision
- Supports up to 344 symbols (matching UC2's LZ77 alphabet)
- Frequency table normalization with minimum-frequency guarantee
- Reverse-order encoding with automatic renormalization
- Fast O(1) decoding via cumulative frequency lookup table
Performance: <5% overhead vs Shannon entropy on tested distributions.
Single-symbol streams compress to ~4 bytes (near-zero information).
Skewed distributions (90% one symbol) achieve sub-bit-per-symbol rates.
6 unit tests:
- Table construction with frequency normalization
- Round-trip: uniform, skewed, 344-symbol alphabet, single-symbol
- Comparison vs Shannon entropy (verified <5% overhead)
UC2 now talks during operations, continuing the original's tradition:
$ uc2 -w archive.uc2 files...
UC2 compression level: Tight
Created archive.uc2 (3 files, 0 dirs, 1 master, 4096 bytes)
Everything went OK
$ uc2 -t archive.uc2
Testing archive integrity...
Everything went OK
$ uc2 -h
UC2 3.0.0 (UltraCompressor II)
"Fast, reliable and superior compression."
Messages are warm, confident, and slightly quirky — not a fun flag,
just how UC2 talks. Suppressed by -q for scripting:
$ uc2 -qt archive.uc2 # silent, exit code only
Compression level names: Fast (2), Normal (3), Tight (4), Ultra (5).
"Everything went OK" directly from the original (MAIN.CPP:918).
Completes Phase 2 (Original Compression Engine).
Replace the fixed first-4KB FNV-1a prefix matching with content-defined
chunking. Files are now split into ~4KB CDC chunks (Gear rolling hash),
each chunk hashed with FNV-1a. Files sharing any chunk hash are grouped
for master-block deduplication.
This detects shared content at ANY position in the files, not just
identical file prefixes. The improvement matters for:
- Patched executables (same code, different version strings)
- Edited documents (same body, different headers)
- Similar data files (shared structures at varying offsets)
The fallback master assignment (for backward compat with original UC2
Pro) still applies to ungrouped files, ensuring all files use custom
master indices (>= 2).
All 7 tests pass including master dedup round-trip and CDC unit tests.
New library (uc2_cdc.h / uc2_cdc.c) for Phase 3 deduplication:
- Gear rolling hash: O(1) per-byte update, uniform distribution,
content-aware boundary detection via mask-based matching
- Configurable chunker: min/max/target chunk sizes (default avg 8KB),
streaming API with reset support
- FNV-1a content hash for chunk dedup addressing
- 256-entry random lookup table for Gear hash distribution
8 unit tests covering:
- Hash determinism and collision avoidance
- Complete data coverage (no bytes lost)
- Min/max chunk size enforcement
- Content-defined boundary alignment across shifted data
- Cross-file dedup detection (shared 256KB block found between
two files with different unique prefixes/suffixes)
Use the default tree for the first block when ibuf < 256 entries,
matching the original's bFlag logic (ULTRACMP.CPP:1105). For larger
blocks, generate custom Huffman trees from actual symbol frequencies.
Compression improvement on text data (textfile.txt, 1719 bytes):
Before (default-only): 1688 bytes compressed
After (custom trees): 1066 bytes compressed (37% smaller)
All tests pass including the bidirectional DOSBox-X round-trip.
Tested custom trees with the custom master fix in place — the original
still hangs. The tree incompatibility is a separate issue from the
SuperMaster path hang (both are real problems, both are now understood).
Custom trees give ~40% better compression for text data (1066 vs 1688
bytes for textfile.txt) but are incompatible with nuke1's assumptions.
Default tree retained for backward compatibility.
Updated roadmap to separate the backward compat (done) from the tree
optimization (remaining).
Always assign custom master indices (>= FIRSTMASTER=2) to all files,
never SuperMaster (index 0). The original's ExtractFiles() routes
SuperMaster files through a code path that hangs. The original itself
never uses SuperMaster in file COMPRESS records — it always creates
at least one custom master, even for archives without dedup groups.
For ungrouped files, a default custom master is built from the largest
file's first 64KB. All files reference this master, matching the
original's archive structure.
The automated DOSBox-X test now validates multi-file round-trip in
both directions: 4 files UC2 v3 -> original, 5 files original -> UC2 v3.
All content verified byte-for-byte.
Compared raw cdir bytes between original UC2 Pro and UC2 v3 archives
(using fixed bitdump.py match decoder). Key finding: the original
NEVER uses SuperMaster (masterPrefix=0) in file COMPRESS records —
it always assigns custom master indices (>= 2). UC2 v3 uses
SuperMaster for ungrouped files.
The original's ExtractFiles() processes SuperMaster files through
ToToWalk(TTefl, SUPERMASTER, ...) which hangs. Custom master files
go through the masroot chain walk, which works correctly.
Fix: always assign custom master indices to all files, generating at
least one custom master block even for archives without dedup groups.
This matches the original's behavior.
Other differences found: original uses hidden=0xDE (sentinel), tag=0
(no EXTMETA), method=3, and csize=8 (much smaller due to custom
master compression).
Single-file UC2 v3 archives are now fully backward compatible with the
original UC2 Pro — listing and extraction verified in automated DOSBox-X
test. SFX extraction timeout increased to 600s with 22-file completeness
check (incomplete extraction caused false test results throughout the
earlier investigation). Direction 1 (UC2 v3 -> original) test added.
Located the nuke1 ASM decompressor source at NUKE.CPP (305 lines of
C++ with inline x86 assembly). Analysis of the half-buffer flush logic
in FlushIt (ULTRACMP.CPP) reveals the extraction hang root cause:
For a 49152-byte SuperMaster, the original adjusts the output position:
wTOE = 49152 - 32768 = 16384 (lower buffer half)
wComp = 0 (flush when output enters lower half)
FlushIt triggers immediately after the first decompressed byte (because
wTOE is already in the lower half with wComp=0). It reads decompressed
output from position 32768 + wSkip = 49152, but the actual output was
written at position 16384. This produces garbage and triggers early
termination.
UC2 v3's compressor starts the LZ77 window at position 49152 (raw
SuperMaster size), while the original's decompressor expects the window
at position 16384 (49152 mod 32768). Fixing this requires aligning
the compressor's starting position to match the decompressor's adjusted
wTOE.
Also corrected: the earlier "single-file extraction works" finding was
a false positive from incomplete UC2DIST extraction (15/22 files).
With complete UC2DIST, listing works but extraction hangs for all files.
Testing revealed that custom Huffman trees from our treegen cause the
original UC2 Pro to hang even for single-file archives. The original's
ASM decompressor kernel (nuke1) has undocumented assumptions about tree
shapes that our treegen doesn't match.
Default tree is the only working option for backward compatibility.
Multi-file extraction remains a separate open issue (hangs even with
default tree, while listing works).
Investigation of multi-file extraction hang:
- LocMacNtx(0) returns VNULL for SuperMaster, but V(VNULL) is safe
in the original's virtual memory system (not a null dereference)
- 0xDEDEDEDE is a legacy sentinel for old archives, not the standard
value (original uses masterPrefix=0)
- The hang is in the file data decompression phase, not the cdir
parsing (listing of multi-file archives works correctly)
Multi-file backward compat remains under investigation.
The original UC2 Pro handles archives differently based on versionMadeBy:
300 causes multi-file listing to fail, while 203 (the original's own
version) works correctly. Also write method=compression_level in file
COMPRESS records instead of hardcoded 1.
Combined with the earlier csize=0 and default-tree fixes, single-file
UC2 v3 archives are now fully backward compatible with the original
UC2 Pro (listing + extraction verified). Multi-file archives can be
listed but extraction still hangs — under investigation.
Root cause: the original UC2 Pro expects csize=0 in the cdir COMPRESS
record (it ignores the field entirely). UC2 v3 was writing the actual
compressed size, which confused the original's archive reader.
Additional changes:
- Use default Huffman tree for all blocks (ensures tree encoding compat)
- Write method=compression_level in cdir COMPRESS (was hardcoded to 1)
- Add tests/scripts/bitdump.py for bit-level bitstream analysis
Single-file UC2 v3 archives are now fully readable by the original UC2
Pro (listing and extraction verified in DOSBox-X). Multi-file archives
still hang — the cdir bitstream decodes correctly in our Python analyzer
but fails in the original's ASM decompressor kernel. Investigation
continues; the bitdump.py tool enables targeted comparison.
Port the original TreeGen/RepairLengths/CodeGen algorithms faithfully
from TREEGEN.CPP for bitstream compatibility with the 1992 UC2 Pro:
- treegen() now accepts max_code_bits parameter (13 for main trees,
7 for tree-encoding meta-tree)
- Heap uses >= for child comparison (prefer right child on ties),
matching original Reheap()
- BuildCodeTree uses extract-one-then-combine pattern
- RepairLengths uses sorted linked lists with cascading space-fill
- Single/zero symbol cases assign length 1 to two symbols
- tree_enc RLE: trigger at run > 6 (not >= 6), max 20 per chunk,
single RepeatCode per run
- First block uses default tree (tree-changed=0) matching original
behavior for small blocks
Full backward compatibility with original UC2 Pro archives (Direction 2)
is maintained. Forward compatibility (UC2 v3 -> original, Direction 1)
remains in progress — the original still hangs, likely due to residual
bitstream-level differences in the ASM decompressor kernel.
Automated test that runs the original 1992 UC2 Pro (UC.EXE) in DOSBox-X
headlessly to create archives from the test corpus, then extracts with
UC2 v3 and verifies byte-for-byte file identity.
Key findings during development:
- uc2pro.exe is a UCEXE self-extracting archive, not the tool itself;
the actual archiver is UC.EXE inside the distribution
- UC.EXE must be run from its own directory (needs DOS.SEA overlay)
- DOSBox-X flatpak requires work dirs under $HOME (filesystem=home)
- The reverse direction (UC2 v3 → original) does not work: the original
UC2 Pro hangs reading UC2 v3 archives due to compression bitstream
differences (added as a roadmap item)
Also fixes create_archives.sh to use the same two-session DOSBox pattern
(extract SFX first, then use UC.EXE).
Recursive directory scanning with parent/child ID tracking, directory
entries in the central directory (OSMETA + DIRMETA + EXTMETA long name
tags), and a CLI round-trip test verifying nested directory hierarchies.
Content-fingerprint grouping via FNV-1a hash of file headers: files
sharing identical first 4096 bytes are assigned a custom master block
built from the largest file in the group. Masters are compressed with
SuperMaster and written as MASMETA records in the central directory.
Files below 1 KB or without a group continue using the SuperMaster.
Includes CLI integration test and documentation updates (format spec,
usage, roadmap).
Sphinx docs with Furo theme covering CLI usage, library API reference,
archive format specification, build instructions, history, and roadmap.
GitHub Actions workflow deploys to Pages on push to main.
Implement a compressor that produces bitstreams compatible with the
existing Bobrowski decompressor. The engine uses LZ77 sliding-window
match finding with hash chains, Huffman entropy coding, and delta-coded
tree serialization matching the original UC2 format exactly.
New files:
- lib/src/compress.c: LZ77+Huffman compressor (~950 lines)
- lib/src/uc2_internal.h: shared constants, types, checksums
- lib/src/uc2_tables.c: vval/ivval delta tables, default Huffman tree
- tests/src/test_roundtrip.c: compress→archive→decompress→verify tests
Key details:
- 4 compression levels (Fast/Normal/Tight/Ultra) with tunable search
- Lazy evaluation for better match selection at higher levels
- Delta-coded Huffman tree serialization with RLE
- Fletcher/XOR checksum computation
- Round-trip test covers 8 patterns × 4 levels (32 test cases)
Fixed 28 errors in the hand-computed ivval inverse delta table (rows
9-13) that caused the decompressor to reconstruct wrong Huffman trees
from compressor output.
Test corpus (empty, text, binary, compressible, incompressible) with
reference archives created by original UC2 v2.3 in DOSBox. Two CTest
tests: test_identify (magic detection) and test_extract (full
extraction pipeline verified byte-for-byte against corpus).
Restructure compat layer: #include_next headers moved to posix/ for
MinGW, new standalone headers in msvc/ for MSVC (unistd.h, utime.h,
getopt.h). Add getopt() implementation, chmod/unlink/chdir compat
functions, MSVC CRT initializer for UTF-8 console, _pgmptr fix.
Add a DJGPP CMake toolchain file and DOS compatibility layer (err.h,
fnmatch, getprogname/setprogname) so UC2 builds as a native DOS
executable via cross-compilation from Linux. The toolchain works
around a baked-in /usr/include in the DJGPP GCC binary by using -I
instead of -isystem to ensure DJGPP headers take precedence.
Add GitHub Actions CI workflow that builds and smoke-tests on both
ubuntu-latest and macos-latest.
UC2 should have warm, quirky status messages as its default voice,
continuing the original's tradition. Not a --fun flag, just how
the tool talks. Quiet mode for scripting.
Eremey Valetov