A new Linux job installs the andrewwutw DJGPP v3.4 cross-toolchain
(gcc 12.2.0, sha256-pinned), cross-compiles uc2.exe with
cmake/djgpp.cmake, and verifies the result is a DJGPP go32 DOS
executable. The DOS build had no CI coverage and could regress
silently.
The repo carried two diverged DJGPP toolchain files. djgpp.cmake
(referenced by the build docs) forces -nostdinc with explicit DJGPP
include paths, so it builds cleanly even on hosts where /usr/include
would otherwise leak past the cross-compiler. djgpp-toolchain.cmake
(previously referenced by the README) relied on the cross-gcc finding
its own headers and broke in that case. Keep djgpp.cmake as the single
toolchain file, point the README and roadmap at it, and drop
djgpp-toolchain.cmake.
Extraction of level 6-9 archives crashed (first seen on NetBSD/sdf.org,
reproducible everywhere), and files larger than the 64KB sliding window
silently corrupted at every level. Four causes:
- cli: master COMPRESS records hardcoded method 1 while master data was
compressed at opt.level, so rANS masters were fed to the Huffman
decoder. Records now carry method 10 at levels 6-9; levels 2-5 keep
method 1 for original UC2 Pro compatibility.
- decompress: decompressor_rans stopped at remaining == 0 without
consuming the end-of-block pair and its 12 extra bits, leaving the
bit cursor desynchronized; the next block-present read landed inside
the EOB extras and parsed a phantom block. The loop now decodes all
nsyms symbols and guards output writes instead.
- decompress: a refill read returning a single byte into an empty
buffer let head overtake tail in bits_feed; the unsigned difference
wrapped and head walked off the 4KB buffer (the actual segfault).
The refill now loops until a full byte pair is available, and a
sticky error flag stops the decoder treating negative bits_get
returns as data.
- compress/decompress: chunk loads wrote linearly past the circular
window edge, and the rANS decoder flushed output in one linear write
that cannot express ring wrap. Loads are now capped at the edge and
the decoder flushes incrementally in ring order.
Also: BCJ E8/E9 byte assembly no longer shifts promoted ints into the
sign bit, and the libarchive plugin uses timegm on NetBSD/OpenBSD/
DragonFly so DOS timestamps are not offset by the local timezone.
New cli_bigfile regression test (>128KB round-trip at L5 and L6); it
fails against the previous binary and passes now. Verified: 22/22
ctest including the DOSBox-X round-trip against original uc2pro.exe,
ASan/UBSan clean, and the full matrix on NetBSD 10 (sdf.org).
Scope shift from the original "make output a real UC2 v3 archive"
issue: that requires a new entry type or compress.c refactor (UC2
archives have one master per file, not a chain). This commit ships
the closest-in-spirit upgrade -- a self-contained format that solves
v1's main UX wart, the sidecar <archive>.blocks/ directory.
Format v2:
+0 8B magic "UC2INGST"
+8 1B version (2)
+9 1B cdc_bits
+10 2B reserved
+12 4B chunk_count
+16 ... chunk_count * 16B: 8B hash, 4B length, 4B offset
... chunk pool: unique chunks back-to-back at recorded offsets
The dedup map has a small implementation note: cap must be a power
of two for the mask-based linear probe to terminate. Caught when
test_ingest hung at 25 chunks -- initial_cap=50 is not power-of-two,
so probing wrapped to a non-empty slot indefinitely. Now rounded up
in dedup_map_init.
Trade-off: cross-archive dedup is not preserved (each --ingest call
overwrites the archive). v1 archives remain restorable through the
sidecar blockstore; the writer defaults to v2.
Tests: 6 cases (was 5). test_intra_call_dedup verifies that
identical chunks within a single ingest dedup correctly
(buffer-twice produces > 0 saved bytes). test_v2_self_contained
asserts the .blocks/ directory is NOT created for v2 archives.
Closes 96ef9b8. v3 (real UC2 v3 archive output) is filed at 59bec0d.
tests/scripts/dos_smoke.sh runs the DJGPP-built uc2 inside DOSBox-X
via the flatpak and asserts:
- uc2 -h loads under a real DPMI host and prints the banner
- uc2 -l <archive> opens an existing UC2 archive and produces output
Skips cleanly when any of uc2.exe, CWSDPMI.EXE, or DOSBox-X are
missing. CWSDPMI.EXE is the standard DJGPP DPMI extender from
csdpmi7b.zip; fetch recipe added to cmake/README-djgpp.md.
Verified locally against build-djgpp/cli/uc2.exe +
tests/archives/basic.uc2.
Closes 20019aa. CI matrix entry (9379647) remains a separate
follow-up.
read_header() slurps the archive on first call (using
__archive_read_ahead + __archive_read_consume), opens libuc2 against
the slurped buffer, walks uc2_read_cdir to cache every entry, and
yields one per call mapped onto archive_entry's pathname / size /
mtime / mode. Tagged entries are resolved via uc2_get_tag. Memory
scales with archive size in v1; seekable adapter via
__archive_read_seek is a future revision.
read_data() runs uc2_extract through a buffering write callback, then
yields the decompressed entry as a single slice (libarchive's pull
API permits this). read_data_skip and cleanup are correct.
Build verified clean against libarchive 3.7.7. End-to-end runtime
test via bsdtar requires a custom libarchive build that links the
plugin (the read-format API is internal). Integration recipe added
to contrib/libarchive/README.md.
Closes 591db60. M4 (master-block dep tracking regression test) and
M7 (bsdtar round-trip) tracked separately.
Replaces the skeleton with a real implementation of the bid callback,
self-registration, and graceful-EOF stubs for the rest of the
read-format vtable. Builds against a libarchive source tree
(LIBARCHIVE_SOURCE_DIR option) because the read-format API is
internal -- the public -devel package only ships archive.h and
archive_entry.h, not archive_read_private.h.
Key changes:
- __archive_read_ahead reads the first 4 bytes; magic check returns
bid 64 on 0x55 0x43 0x32 0x1A.
- __archive_read_register_format wired with the correct 12-argument
signature against libarchive 3.7.7.
- archive_platform_config.uc2.h.in stands in for the generated
config.h, satisfying archive_platform.h's include-or-error gate
without us needing to run libarchive's own configure.
Resulting libuc2_libarchive.a exports archive_read_support_format_uc2
with three undefined references (__archive_check_magic,
__archive_read_ahead, __archive_read_register_format) that resolve
when linked into a libarchive tree.
Read_header / read_data / cleanup are EOF stubs. Wiring to libuc2
is milestone 2+.
Closes b0b06a5; M2-3 tracked at 591db60.
cmake/djgpp-toolchain.cmake builds uc2.exe against the andrewwutw/
build-djgpp prebuilt cross toolchain. Verified with gcc 7.2.0 and
12.2.0; output is a 359 KB MZ + COFF + go32 DOS executable.
libuc2 (14 source files including the new uc2_ingest.c) compiles
unmodified. CLI uses the existing cli/src/compat/compat_dos.c shim
for BSD err.h and POSIX fnmatch -- already in tree, just needed
the toolchain file to set DJGPP=TRUE so cli/CMakeLists.txt picks it
up.
Documented gotcha: GCC honours CPATH and CPLUS_INCLUDE_PATH from
the build shell regardless of -nostdinc. On hosts that export them
(e.g. via Intel oneAPI's setvars.sh), host glibc headers end up first
in the cross-compiler's search path and the build fails on stdint.h.
The README walks through 'unset CPATH' as the remediation.
DOSBox-X end-to-end smoke test and CI matrix entry tracked as P2
follow-ups. Closes 195be9a.
Reads stdin, splits via CDC, deduplicates chunks against a sidecar
block store at <archive>.blocks/, writes a chunk-hash manifest at
<archive>. The reverse operation reads the manifest and reassembles
the byte stream from the block store.
Manifest format (magic UC2INGST) is a standalone container, not yet
unified with the master-block archive layout. Tar boundaries are not
preserved; the input is treated as an opaque byte stream. Follow-ups
filed for both.
Builds entirely on existing CDC + blockstore + merkle infrastructure.
No new compression or hashing primitives.
Tests cover small + 200 KB multichunk round-trip, idempotent dedup
(repeat ingest of the same data reports zero new chunks and exact
bytes_saved), empty stream, bad-magic rejection. Lint gate stays
green.
Closes fa0c7d4.
uc2_sha256: pure-C FIPS 180-4 implementation, one-shot and incremental
API, validated against published vectors (empty, abc, 56-byte,
1M 'a', byte-by-byte, every-split-point boundary).
uc2_ots: parser, serializer, and walker for the standard .ots binary
format. Strict canonical varint with 64-bit overflow check, depth-
bounded recursion, varbytes cap, max-digest cap. Walker supports
the calendar-path subset (APPEND, PREPEND, SHA256); proofs that
include other crypto ops (SHA1, RIPEMD160, KECCAK256) are accepted
as structurally valid but flagged for follow-up via the standard
'ots verify'.
UC2-OTS trailer: magic-bracketed sidecar appended after the recorded
archive bytes. Reverse-scan-safe; original UC2 Pro reader ignores
trailing bytes past its recorded length so backward compatibility is
preserved. Layout (all integers little-endian uint32):
front-magic + version + archive_len + proof_len + proof
+ proof_len + back-magic.
CLI: --ots-attach validates that the proof's leaf digest equals
SHA-256(archive[0..archive_len)) before appending and refuses to
overwrite an existing trailer unless -f is given. --ots-extract
writes the proof verbatim, byte-compatible with the standard
'ots verify'. --ots-info parses and prints the leaf, archive-match
status, and attestation list. uc2 -t recomputes the archive
SHA-256 and walks the proof.
Tests: 17 OTS unit tests (varint round-trip, canonical/overflow
rejection, file-envelope round-trip, walker on append/sha256/
sibling/unsupported-op/truncated/trailing-garbage, attest_name,
trailer round-trip + corruption rejection in 4 scenarios).
Plus an optional ctest target ots_cross_check that round-trips
the .ots through python-opentimestamps when the package is
installed; skipped (return code 77) otherwise.
- Mark test corpus/archives as binary in .gitattributes to prevent
line ending conversion on CI (fixes extract test size mismatch)
- Fix alignment-unsafe struct cast in uc2_dict.c serialize/deserialize
(use memcpy-based byte access instead; fixes SEGFAULT on CI)
- Fix formatting issues in docs
New library (uc2_blake3.h / uc2_blake3.c) for Phase 7:
- Pure C BLAKE3 implementation (~300 lines)
- 256-bit (32-byte) digests using BLAKE2s round function
- Bao tree hashing structure for inputs > 1024 bytes
- Incremental API (init/update/final) and one-shot helper
- Constant-time hash comparison (timing-attack resistant)
Suitable for content verification, block integrity checking,
and content-addressable storage (replacing or supplementing
the 64-bit FNV-1a hashes used in Merkle DAG and block store).
7 unit tests:
- Empty input, determinism, collision avoidance
- Incremental vs one-shot consistency
- Single-byte-at-a-time update consistency
- Avalanche effect (1-bit change → ~50% output bits flip)
- Constant-time comparison
New library (uc2_preprocess.h / uc2_preprocess.c) for Phase 4:
BCJ (Branch/Call/Jump) filter:
- E8/E9 x86 address normalization (relative → absolute)
- Makes calls to the same function from different locations produce
identical byte sequences, improving LZ77 matching
- Round-trip verified; address normalization confirmed
BWT (Burrows-Wheeler Transform):
- Suffix-array-based forward transform
- LF-mapping inverse with reverse reconstruction
- Groups similar contexts for better entropy coding
- Round-trip verified for text ("banana") and binary data
Delta filter:
- Byte-wise delta encoding with configurable stride
- Stride 1 for sequential data, stride 2+ for interleaved channels
- Constant-delta sequences (arithmetic progressions) reduce to
repeated single values
Content detection:
- Automatic content type identification (text/x86/structured/binary)
- MZ/PE and ELF header recognition for x86
- Printable ASCII ratio for text detection
11 unit tests covering all filters and detection.
New library (uc2_dict.h / uc2_dict.c) formalizes master blocks as
proper dictionaries with:
- 64-bit content hash ID (FNV-1a) for cross-archive sharing
- 32-bit integrity checksum with verification
- Portable serialization format (24-byte header + data)
- Deserialization with magic number and size validation
Combined with the block store (uc2_blockstore.h), this enables
distributed dedup: archives in different locations can reference
shared dictionaries by content hash, with integrity verification
before decompression.
6 unit tests including serialization round-trip, corruption
detection, and bad-magic rejection.
Also added plausible deniability (multi-archive with separate
passwords) to Phase 5 roadmap.
rANS entropy coding is now a usable compression option:
uc2 -w -L 8 archive.uc2 files... # rANS Tight
uc2 archive.uc2 # decompresses (auto-detects method)
Block format for method 10:
[block-present:1] [nsyms:16] [rans_len:16]
[freq_table:344x12bits] [rans_data] [extra_bits]
Symbol IDs (0-343) encoded with rANS for near-optimal entropy.
Extra bits (distance/length parameters) stored separately in the
bitstream, preserving the existing variable-length encoding.
Integration:
- Compressor: flush_block_rans() dispatched when level >= 6
- Decompressor: decompressor_rans() dispatched for method 10
- CLI: levels 6-9 map to rANS Fast/Normal/Tight/Ultra
- COMPRESS records store method=10 for rANS files/cdir
- End-to-end round-trip verified (create/list/extract/verify)
Levels 2-5 (Huffman) remain the default for backward compatibility
with the original UC2 Pro.
New library (uc2_rans.h / uc2_rans.c) — table-based range Asymmetric
Numeral Systems (rANS) entropy coder:
- 32-bit state with 12-bit probability precision
- Supports up to 344 symbols (matching UC2's LZ77 alphabet)
- Frequency table normalization with minimum-frequency guarantee
- Reverse-order encoding with automatic renormalization
- Fast O(1) decoding via cumulative frequency lookup table
Performance: <5% overhead vs Shannon entropy on tested distributions.
Single-symbol streams compress to ~4 bytes (near-zero information).
Skewed distributions (90% one symbol) achieve sub-bit-per-symbol rates.
6 unit tests:
- Table construction with frequency normalization
- Round-trip: uniform, skewed, 344-symbol alphabet, single-symbol
- Comparison vs Shannon entropy (verified <5% overhead)
UC2 now talks during operations, continuing the original's tradition:
$ uc2 -w archive.uc2 files...
UC2 compression level: Tight
Created archive.uc2 (3 files, 0 dirs, 1 master, 4096 bytes)
Everything went OK
$ uc2 -t archive.uc2
Testing archive integrity...
Everything went OK
$ uc2 -h
UC2 3.0.0 (UltraCompressor II)
"Fast, reliable and superior compression."
Messages are warm, confident, and slightly quirky — not a fun flag,
just how UC2 talks. Suppressed by -q for scripting:
$ uc2 -qt archive.uc2 # silent, exit code only
Compression level names: Fast (2), Normal (3), Tight (4), Ultra (5).
"Everything went OK" directly from the original (MAIN.CPP:918).
Completes Phase 2 (Original Compression Engine).
Replace the fixed first-4KB FNV-1a prefix matching with content-defined
chunking. Files are now split into ~4KB CDC chunks (Gear rolling hash),
each chunk hashed with FNV-1a. Files sharing any chunk hash are grouped
for master-block deduplication.
This detects shared content at ANY position in the files, not just
identical file prefixes. The improvement matters for:
- Patched executables (same code, different version strings)
- Edited documents (same body, different headers)
- Similar data files (shared structures at varying offsets)
The fallback master assignment (for backward compat with original UC2
Pro) still applies to ungrouped files, ensuring all files use custom
master indices (>= 2).
All 7 tests pass including master dedup round-trip and CDC unit tests.
New library (uc2_cdc.h / uc2_cdc.c) for Phase 3 deduplication:
- Gear rolling hash: O(1) per-byte update, uniform distribution,
content-aware boundary detection via mask-based matching
- Configurable chunker: min/max/target chunk sizes (default avg 8KB),
streaming API with reset support
- FNV-1a content hash for chunk dedup addressing
- 256-entry random lookup table for Gear hash distribution
8 unit tests covering:
- Hash determinism and collision avoidance
- Complete data coverage (no bytes lost)
- Min/max chunk size enforcement
- Content-defined boundary alignment across shifted data
- Cross-file dedup detection (shared 256KB block found between
two files with different unique prefixes/suffixes)
Use the default tree for the first block when ibuf < 256 entries,
matching the original's bFlag logic (ULTRACMP.CPP:1105). For larger
blocks, generate custom Huffman trees from actual symbol frequencies.
Compression improvement on text data (textfile.txt, 1719 bytes):
Before (default-only): 1688 bytes compressed
After (custom trees): 1066 bytes compressed (37% smaller)
All tests pass including the bidirectional DOSBox-X round-trip.
Tested custom trees with the custom master fix in place — the original
still hangs. The tree incompatibility is a separate issue from the
SuperMaster path hang (both are real problems, both are now understood).
Custom trees give ~40% better compression for text data (1066 vs 1688
bytes for textfile.txt) but are incompatible with nuke1's assumptions.
Default tree retained for backward compatibility.
Updated roadmap to separate the backward compat (done) from the tree
optimization (remaining).
Always assign custom master indices (>= FIRSTMASTER=2) to all files,
never SuperMaster (index 0). The original's ExtractFiles() routes
SuperMaster files through a code path that hangs. The original itself
never uses SuperMaster in file COMPRESS records — it always creates
at least one custom master, even for archives without dedup groups.
For ungrouped files, a default custom master is built from the largest
file's first 64KB. All files reference this master, matching the
original's archive structure.
The automated DOSBox-X test now validates multi-file round-trip in
both directions: 4 files UC2 v3 -> original, 5 files original -> UC2 v3.
All content verified byte-for-byte.
Compared raw cdir bytes between original UC2 Pro and UC2 v3 archives
(using fixed bitdump.py match decoder). Key finding: the original
NEVER uses SuperMaster (masterPrefix=0) in file COMPRESS records —
it always assigns custom master indices (>= 2). UC2 v3 uses
SuperMaster for ungrouped files.
The original's ExtractFiles() processes SuperMaster files through
ToToWalk(TTefl, SUPERMASTER, ...) which hangs. Custom master files
go through the masroot chain walk, which works correctly.
Fix: always assign custom master indices to all files, generating at
least one custom master block even for archives without dedup groups.
This matches the original's behavior.
Other differences found: original uses hidden=0xDE (sentinel), tag=0
(no EXTMETA), method=3, and csize=8 (much smaller due to custom
master compression).
Single-file UC2 v3 archives are now fully backward compatible with the
original UC2 Pro — listing and extraction verified in automated DOSBox-X
test. SFX extraction timeout increased to 600s with 22-file completeness
check (incomplete extraction caused false test results throughout the
earlier investigation). Direction 1 (UC2 v3 -> original) test added.
Located the nuke1 ASM decompressor source at NUKE.CPP (305 lines of
C++ with inline x86 assembly). Analysis of the half-buffer flush logic
in FlushIt (ULTRACMP.CPP) reveals the extraction hang root cause:
For a 49152-byte SuperMaster, the original adjusts the output position:
wTOE = 49152 - 32768 = 16384 (lower buffer half)
wComp = 0 (flush when output enters lower half)
FlushIt triggers immediately after the first decompressed byte (because
wTOE is already in the lower half with wComp=0). It reads decompressed
output from position 32768 + wSkip = 49152, but the actual output was
written at position 16384. This produces garbage and triggers early
termination.
UC2 v3's compressor starts the LZ77 window at position 49152 (raw
SuperMaster size), while the original's decompressor expects the window
at position 16384 (49152 mod 32768). Fixing this requires aligning
the compressor's starting position to match the decompressor's adjusted
wTOE.
Also corrected: the earlier "single-file extraction works" finding was
a false positive from incomplete UC2DIST extraction (15/22 files).
With complete UC2DIST, listing works but extraction hangs for all files.
Testing revealed that custom Huffman trees from our treegen cause the
original UC2 Pro to hang even for single-file archives. The original's
ASM decompressor kernel (nuke1) has undocumented assumptions about tree
shapes that our treegen doesn't match.
Default tree is the only working option for backward compatibility.
Multi-file extraction remains a separate open issue (hangs even with
default tree, while listing works).
Investigation of multi-file extraction hang:
- LocMacNtx(0) returns VNULL for SuperMaster, but V(VNULL) is safe
in the original's virtual memory system (not a null dereference)
- 0xDEDEDEDE is a legacy sentinel for old archives, not the standard
value (original uses masterPrefix=0)
- The hang is in the file data decompression phase, not the cdir
parsing (listing of multi-file archives works correctly)
Multi-file backward compat remains under investigation.
The original UC2 Pro handles archives differently based on versionMadeBy:
300 causes multi-file listing to fail, while 203 (the original's own
version) works correctly. Also write method=compression_level in file
COMPRESS records instead of hardcoded 1.
Combined with the earlier csize=0 and default-tree fixes, single-file
UC2 v3 archives are now fully backward compatible with the original
UC2 Pro (listing + extraction verified). Multi-file archives can be
listed but extraction still hangs — under investigation.
Root cause: the original UC2 Pro expects csize=0 in the cdir COMPRESS
record (it ignores the field entirely). UC2 v3 was writing the actual
compressed size, which confused the original's archive reader.
Additional changes:
- Use default Huffman tree for all blocks (ensures tree encoding compat)
- Write method=compression_level in cdir COMPRESS (was hardcoded to 1)
- Add tests/scripts/bitdump.py for bit-level bitstream analysis
Single-file UC2 v3 archives are now fully readable by the original UC2
Pro (listing and extraction verified in DOSBox-X). Multi-file archives
still hang — the cdir bitstream decodes correctly in our Python analyzer
but fails in the original's ASM decompressor kernel. Investigation
continues; the bitdump.py tool enables targeted comparison.
Port the original TreeGen/RepairLengths/CodeGen algorithms faithfully
from TREEGEN.CPP for bitstream compatibility with the 1992 UC2 Pro:
- treegen() now accepts max_code_bits parameter (13 for main trees,
7 for tree-encoding meta-tree)
- Heap uses >= for child comparison (prefer right child on ties),
matching original Reheap()
- BuildCodeTree uses extract-one-then-combine pattern
- RepairLengths uses sorted linked lists with cascading space-fill
- Single/zero symbol cases assign length 1 to two symbols
- tree_enc RLE: trigger at run > 6 (not >= 6), max 20 per chunk,
single RepeatCode per run
- First block uses default tree (tree-changed=0) matching original
behavior for small blocks
Full backward compatibility with original UC2 Pro archives (Direction 2)
is maintained. Forward compatibility (UC2 v3 -> original, Direction 1)
remains in progress — the original still hangs, likely due to residual
bitstream-level differences in the ASM decompressor kernel.
Automated test that runs the original 1992 UC2 Pro (UC.EXE) in DOSBox-X
headlessly to create archives from the test corpus, then extracts with
UC2 v3 and verifies byte-for-byte file identity.
Key findings during development:
- uc2pro.exe is a UCEXE self-extracting archive, not the tool itself;
the actual archiver is UC.EXE inside the distribution
- UC.EXE must be run from its own directory (needs DOS.SEA overlay)
- DOSBox-X flatpak requires work dirs under $HOME (filesystem=home)
- The reverse direction (UC2 v3 → original) does not work: the original
UC2 Pro hangs reading UC2 v3 archives due to compression bitstream
differences (added as a roadmap item)
Also fixes create_archives.sh to use the same two-session DOSBox pattern
(extract SFX first, then use UC.EXE).
Content-fingerprint grouping via FNV-1a hash of file headers: files
sharing identical first 4096 bytes are assigned a custom master block
built from the largest file in the group. Masters are compressed with
SuperMaster and written as MASMETA records in the central directory.
Files below 1 KB or without a group continue using the SuperMaster.
Includes CLI integration test and documentation updates (format spec,
usage, roadmap).
UC2 should have warm, quirky status messages as its default voice,
continuing the original's tradition. Not a --fun flag, just how
the tool talks. Quiet mode for scripting.
Eremey Valetov