Controlled input
Sequential reads and bounded channels limit memory pressure.
Back to projects
Compression and archiving engine
GGCompress
Go engine for processing large files in parallel while keeping memory controlled and integrity verifiable.
Technical vision
Parallel archiving with controlled memory
GGCompress processes large files with sequential reads, parallel processing, ordered writing, and a verifiable .ggc format.
Impact.ggc
versioned format with deterministic manifest and chunk table
Parallel processing
Worker pools process chunks in parallel with buffer reuse.
Safe publication
Extraction uses temporary staging and publishes the result only after full validation.
Execution flow
Main flow
Parallel archiving with controlled memory
GGCompress processes large files with sequential reads, parallel processing, ordered writing, and a verifiable .ggc format.
LARGE FILE -> BOUNDED CHUNKS -> WORKERS -> ORDERED OUTPUT -> CHECKSUM
- 01
Read and split files
Files, directories, and symbolic links enter the deterministic manifest of the .ggc format.
- 02
Process chunks in parallel
Workers process chunks with bounded channels and ordered writing.
- 03
Validate before publishing
Per-chunk checksums and global SHA-256 validate the result before final publication.
Engineering decisions
Architecture
Versioned .ggc format
The format stores a deterministic manifest, chunk table, files, directories, and symbolic links.
Parallelism with backpressure
Worker pools, bounded channels, and buffer reuse control memory during parallel processing.
Extraction through staging
The result is published only after full validation in temporary staging.
Risk controls
Reliability
Archive corruption
Per-chunk checksums and global SHA-256 verify integrity.
Partial publication
Temporary staging avoids publishing a result before full validation.
Stack
Runtime
- Go
- worker pools
- bounded channels
Format
- .ggc
- deterministic manifest
- chunk table
Integrity
- per-chunk checksums
- global SHA-256
- temporary staging
Outcome
Measurable parallel processing
The project includes measurements for workers, chunk size, throughput, compression ratio, speedup, and peak heap without exposing values not provided.
- versioned format with deterministic manifest and chunk table
- .ggc