Concepts

Content addressing

How PotDrive turns bytes into IPFS CIDs that Portaldot can serve, and how content-defined chunking makes new versions cheap.

CIDs that Portaldot understands

TransactionStorage indexes data under blake2_256(data), and the node's bitswap RPC accepts any CIDv1 whose multihash carries that digest. So every PotDrive block gets a CIDv1 with the blake2b-256 multihash (0xb220):

Block	Codec	Prefix
File leaf	`raw` (0x55)	`bafk2bzace…`
File root, directory	`dag-pb` (0x70)	`bafykbzace…`

sdk/src/cid.ts

export function cidFor(bytes: Uint8Array, codec = RAW): CID {
  return CID.createV1(codec, Digest.create(BLAKE2B_256, blake2AsU8a(bytes, 256)));
}

These are standard IPFS CIDs. Any IPFS tool can parse them, and a Portaldot node started with --ipfs-server serves them to IPFS peers over libp2p bitswap.

Files as UnixFS DAGs

A file that fits in one chunk is a single raw block. Larger files get a UnixFS dag-pb root that links every leaf in order and records each leaf's size in blockSizes. That is the same structure ipfs add --raw-leaves builds, with blake2b instead of sha2-256.

A file: raw leaves cut by FastCDC, linked in order by a UnixFS dag-pb root. A folder adds directory nodes above file roots.

Because the root records leaf sizes, a reader can map any byte offset to the leaf that holds it. That is how the gateway answers HTTP Range requests by fetching only the leaves it needs (see Gateway).

Content-defined chunking

Fixed-size chunking cuts every 1 MiB. Insert one byte near the start of a file and every later boundary shifts, so every chunk gets a new hash and has to be stored and paid for again. PotDrive uses FastCDC instead (Xia et al., USENIX ATC 2016). A cut point is where a rolling gear hash of the last ~32 bytes matches a mask. Boundaries follow the content, so an edit only changes the chunks around it.

Min / avg / max	256 KiB / 512 KiB / 1 MiB. The 1 MiB cap stays under bitswap's soft block limit.
Normalization	Harder mask (2^-21) before the average size, easier mask (2^-17) after it.
Gear table	256 values from a fixed splitmix32 seed. The table is part of the format: changing it changes every CID.
Speed	Chunks and hashes 16 MiB in about 360 ms in Bun on an M-series Mac.

Measured on a 16 MiB file

New bytes that had to be stored after each edit (run bun scripts/bench-cdc.ts to reproduce):

Edit	Fixed 1 MiB	FastCDC
Insert 1 KiB at 300 KiB	16,385 KiB (100%)	609 KiB (3.7%)
Delete 10 KiB at 5 MiB	11,254 KiB (68.7%)	343 KiB (2.1%)
Overwrite 4 KiB mid-file	1,024 KiB (6.3%)	898 KiB (5.5%)
Append 2 MiB	2,048 KiB (11.1%)	2,513 KiB (13.6%)

At about 105 tPOT per MiB, re-uploading that file after a 1 KiB insertion costs roughly 1,680 tPOT with fixed chunks and 62 tPOT with FastCDC. Fixed chunking only wins slightly on pure appends.

Deduplication against the chain

Before uploading, PotDrive reads TransactionStorage.Transactions once and compares content hashes. Any block Portaldot already holds, from an earlier version, another file or another user, is linked instead of stored. The cost sheet shows reused and new bytes separately, and the upload key only pays for the new ones.

sdk/src/upload.ts

const held = new Set((await listStored(api)).map((e) => e.contentHash));
const reused = file.blocks.filter((b) => held.has(contentHash(b.cid)));
const fresh  = file.blocks.filter((b) => !held.has(contentHash(b.cid)));   // only these are stored

Directories

A folder becomes a UnixFS directory: a dag-pb node whose links are named entries, sorted by name. Building the same folder in any order gives the same root CID. Identical files inside it share blocks. Paths resolve one verified block at a time:

resolve a path

const s = await resolvePath("bafykbzace…", "assets/logo.svg");
// root (directory) → "assets" (directory) → "logo.svg" (file root or raw leaf)

Edit this page on GitHub