NCF: Compressed Binary Serialization with HMAC Signing

What is NCF?

NCF (Nexus Compact Format) is NSL's binary serialization format. It handles everything JSON can't: compression, random access, integrity checking, signing, streaming, and even RPC.

File extension: .ncf

Two-Pass Compression

Pass 1 -- Structural compression:

• Inline small ints (0-127) as a single byte
• Zigzag + varint for larger integers
• IEEE 754 half-precision (float16) when lossless
• Float unit encoding (0.0-1.0 quantized to 1 byte)
• Frequency-sorted string interning
• Dict shape dedup (repeated key patterns stored once)
• Typed lists (skip per-element type tags)
• Boolean bit-packing (8 bools per byte)
• Value dedup (repeated subtrees stored once)
• Tensor encoding with auto dtype selection and XOR delta encoding

Pass 2 -- Byte-level compression (for payloads > 64 bytes):

• LZ77 hash matching (4-byte hash, 64KB window)
• Canonical Huffman coding (max 15-bit codes)
• Only applied when compressed < original

Basic Usage

# Encode and decode
let bytes = ncf.encode({model: weights, config: params})
let original = ncf.decode(bytes)

# File I/O
ncf.save("data.ncf", my_data)
let loaded = ncf.load("data.ncf")

# Inspect without decoding
let meta = ncf.inspect(bytes)
print(meta)  # format version, compression, size info

# Size estimation
let size = ncf.size(my_data)  # encoded byte count

Indexed Random Access

Load only the fields you need -- without decoding everything:

# Save with index
ncf.save_indexed("model.ncf", {
    layer1: w1,
    layer2: w2,
    layer3: w3,
    config: {lr: 0.001, epochs: 10}
})

# Load just one field
let w1 = ncf.load_field("model.ncf", "layer1")

# List available fields
let fields = ncf.list_fields(bytes)  # ["layer1", "layer2", ...]

# Lazy access (decode single field, skip rest)
let cfg = ncf.lazy_get(bytes, "config")
let keys = ncf.lazy_keys(bytes)  # keys without decoding values

Memory-Mapped I/O

For large files, memory-map instead of loading into RAM:

let handle = ncf.mmap_open("huge_model.ncf")
let fields = ncf.mmap_fields(handle)
let w1 = ncf.mmap_field(handle, "layer1")  # only decodes this field
let raw = ncf.mmap_raw(handle, "layer2")    # raw bytes, no decode
ncf.mmap_close(handle)

Streaming

Process sequences of values without loading all at once:

# Encode a stream
let bytes = ncf.encode_stream([batch1, batch2, batch3, batch4])

# Iterate
let iter = ncf.stream_iter(bytes)
let val = ncf.stream_next(iter)  # returns null when exhausted
while val != null
    process(val)
    val = ncf.stream_next(iter)
ncf.stream_close(iter)

# Count chunks
let n = ncf.stream_count(bytes)

Integrity and Signing

# CRC32 integrity
let valid = ncf.verify(bytes)  # checks CRC32

# HMAC-SHA256 signing
let signed = ncf.encode_signed(data, "secret_key")
let ok = ncf.verify_signed(signed, "secret_key")  # true if untampered

RPC Over NCF

NCF includes a built-in RPC system:

# Server
let server = ncf.rpc_serve(9000, {
    add: fn(a, b) => a + b,
    multiply: fn(a, b) => a * b
})

# Client
let client = ncf.rpc_connect("localhost", 9000)
let result = ncf.rpc_call(client, "add", {a: 3, b: 4})  # 7
ncf.rpc_notify(client, "log", {msg: "hello"})  # fire-and-forget
ncf.rpc_close(client)
ncf.rpc_stop(server)

Utilities

# Diff two NCF payloads
let changes = ncf.diff(bytes1, bytes2)  # added/removed/changed keys

# Merge
let merged = ncf.merge(bytes1, bytes2)

# Convert to/from JSON
let json_str = ncf.to_json(bytes)
let ncf_bytes = ncf.from_json(json_string)

# Schema validation
let valid = ncf.validate(data, {name: "string", age: "number"})

# Versioning
let versioned = ncf.versioned(data, 3)
let ver = ncf.check_version(versioned)  # 3

# Statistics
let stats = ncf.stats(bytes)  # type counts, depth, sizes

Safety

NCF enforces strict limits against malicious input: 64MB max decompressed size, 1024:1 max compression ratio, 4M max collection items, 128 max recursion depth, CRC32 on every payload, and Huffman code length validation (prevents CVE-2023-4863 class bugs).