Synchronisation

Synchronisation is a recursive comparison of two Merkle trees built from file metadata. The client only ever sends the differences.

1 · A tree built from metadata, not contents

Each file's hash is sha256(size : mtime-seconds), derived from stat() alone, so building the tree reads no file data. A directory's hash is the hash of its sorted child name:hash lines. This makes the whole tree a Merkle hash: two directories share a hash iff every file beneath them has identical size and timestamp. Empty directories and dotfiles are omitted, so the two sides always describe the same thing.

2 · Compare the root, then descend only where it differs

1

Ask for the root hash

GET /api/v1/tree?path=&depth=0, if it equals the local root hash, nothing changed: done in a single request.

2

Descend the differences

GET /api/v1/tree?path=<dir>&depth=1, for each directory whose hash differs, fetch one level and compare child by child. Identical subtrees are skipped.

3

Push new & changed files

PUT /api/v1/file?path=<rel>&mtime=<ms>, streamed with backpressure, so a multi-GB file stays near-constant in memory. The server applies the sent mtime so the hashes match next time.

4

Remove what's gone

DELETE /api/v1/file?path=<rel>, anything on the server but absent locally is deleted; a directory delete removes its whole subtree and prunes empty parents.

3 · Why it stays correct and cheap

4 · Access logs, mirrored incrementally

Caddy writes one rotating access log. The client mirrors it by timestamp, GET /api/v1/logs lists sizes and rotated-file timestamps; GET /api/v1/logs/file?which=active fetches only the newly-appended tail via an HTTP Range. A new rotated file signals the active log restarted. Filenames never cross the wire, so the server can't influence where the client writes.

5 · Server statistics, sampled and mirrored

The server samples its own health (per-core CPU, memory, disk usage, and cumulative disk & network byte counters) on a fixed interval, appending one JSON line per sample to per-type, date-stamped day-files. GET /api/v1/stats lists those files (type, date, size) plus the sample interval; GET /api/v1/stats/file?name=<file> streams one, using an HTTP Range to fetch only the growing tail of the current day. Past days are immutable, so the client mirrors each once and keeps it after the server prunes its own retention window. GET /api/v1/stats/now returns a single live snapshot for debugging.

Rates are not stored: the byte counters are cumulative, and the reader divides the delta between two samples by the elapsed time between them, so a gap after downtime is averaged over its true length instead of reading as a spike, and a counter reset (reboot) is dropped. As with the logs, filenames are validated against a fixed pattern and never influence where the client writes. Metrics a host can't report (e.g. disk or network IO off Linux) are simply absent rather than an error.

The endpoints

CallPurpose
GET /api/v1/treeMerkle node at a path/depth
PUT /api/v1/fileUpload a file + apply its mtime
DELETE /api/v1/fileDelete a file or a whole site
GET /api/v1/logsList active size + rotated timestamps
GET /api/v1/logs/fileStream a log (Range supported)
GET /api/v1/statsList stat day-files (type, date, size)
GET /api/v1/stats/fileStream a stat file (Range supported)
GET /api/v1/stats/nowLive server-health snapshot
GET /api/versionServer's API version (skew check)