--- title: "Mounting Remote Disks Over Plain HTTP: Contributing to fs-relay" description: How Muayyad Alsadi's fs-relay mounts remote folders over HTTP 10x faster than s3fs, and the upstream work that made SQLite locks solid for v0.2.0. summary: A reflection on 20 years of Linux network mounts, why Muayyad Alsadi's fs-relay beats s3fs and NFS by 10x, and the upstream work on SQLite locking and v0.2.0. date: 2026-11-08T00:00:00.000Z heroImage: src: /_astro/hero.DPw8SuwK.webp width: 1360 height: 768 format: webp tags: - linux - fuse - go - networking - sqlite - open-source categories: - The Linux Lounge - Technology Bites published: true featured: true draft: false author: Jad coAuthor: name: Muayyad Alsadi url: https://x.com/muayyadalsadi href: /posts/mounting-remote-directories-over-http/ slug: mounting-remote-directories-over-http --- In 2004, setting up NFS across two beige desktop boxes in my room felt like inviting a thunderstorm indoors. You edited `/etc/exports`, restarted `portmap` and `rpcbind`, crossed your fingers, and waited for the whole desktop to freeze the moment someone tripped over an Ethernet cable. Anyone who grew up in [those early Linux days](/posts/open-source-ai-rebellion/) knows the specific dread of a dead network mount. A terminal tab running `ls` would enter an unkillable kernel sleep (`D` state). Even `kill -9` bounced off it. You either lived with a haunted directory tree or reached for the physical reset button. Two decades later, our workloads moved to cloud instances and container sandboxes, yet the friction of touching files across a wire never really disappeared. We traded manual NFS exports for `rsync` loops, background sync daemons, or `sshfs`. When you work from the Middle East, where [latency is a daily tax](/posts/latency-is-political-when-you-live-far-from-the-center/) on every round trip, chatty remote filesystems punish every command you type. At the end of October, [Muayyad Alsadi](https://github.com/muayyad-alsadi/) ([@muayyadalsadi](https://x.com/muayyadalsadi)) started building a clean-sheet answer to this problem called [fs-relay](/project/fs-relay/). Over the past week, I had the pleasure of contributing upstream fixes across locking, startup safety, shutdown, and self-update for today's `v0.2.0` release. This is the story of how Muayyad's design makes `fs-relay` ten times faster than the usual suspects, and what went into making multi-mount SQLite and atomic self-updates solid. What's in this post: - [Why traditional remote mounts struggle](#why-traditional-remote-mounts-struggle) - [How Muayyad made fs-relay 10x faster](#how-muayyad-made-fs-relay-10x-faster) - [Upstream work toward v0.2.0](#upstream-work-toward-v020) - [Try fs-relay v0.2.0](#try-fs-relay-v020) ## Why Traditional Remote Mounts Struggle When you want multiple machines or [coding agents](/posts/my-mcp-stack/) to share a live workspace without copying gigabytes back and forth, you hit five walls almost immediately: - **The classical user-space copy loop**: Moving a file across a network the textbook way repeats the same five steps for every chunk. With a 4 KiB buffer and a 1 GiB file, that loop runs over 250,000 times, paying 250,000 syscalls, 250,000 user/kernel context switches, and a trail of scattered, fragmented memory allocations: 1. Allocate memory. 2. Receive from the network into memory. 3. Write from memory to disk. 4. Repeat steps 2 and 3 in either direction (*net -> disk* or *disk -> net*). 5. Free memory. - **Round-trip chatty metadata**: A simple `tar -xf` or project build issues thousands of `lookup`, `getattr`, `open`, `read`, and `close` calls. Sending each call as a separate network round trip turns a fast link into molasses. - **Root privileges and firewall ports**: Classic NFS and SMB expect root daemons, kernel modules, and open ports that are painful to expose across networks. - **Broken advisory locks**: Most network mounts either ignore POSIX byte-range locks or fake them locally. Point two processes at the same SQLite database and you get silent corruption. - **No fast-lane for large payloads**: Other protocols wrap every small chunk of a large file inside its own message frame, so framing overhead grows linearly with file size. `fs-relay` has a dedicated `raw` path for large files that pays a constant overhead regardless of size. ## How Muayyad Made fs-relay 10x Faster Muayyad designed `fs-relay` around a strict constraint: five Go packages, zero CGO, rootless Linux FUSE v3 (`fusermount3`) on the client, and standard HTTP and WebSockets on the wire. Instead of trying to micro-optimize the classical copy loop, `fs-relay` skips both the user-space copy cycle and the per-file round trips: - **Zero-copy kernel transfers for large files**: Large reads and writes bypass framing overhead on a raw socket and use Linux kernel zero-copy primitives (`splice(2)` and `copy_file_range(2)`, as Muayyad detailed in [his article on zero-copy single context switching](https://medium.com/@alsadi/secret-sauce-is-zero-copy-single-context-switching-b749bd614dca)). Bytes move between socket and disk inside the kernel without user-space buffer allocation, copying, or a context switch on every 4 KiB block. - **1 MiB write-behind buffering**: Contiguous client writes coalesce up to 1 MiB on the mount before flushing across the wire. - **Local read-only opens**: Opening a file read-only takes no server file handle. The client resolves `open`, `flush`, and `release` locally and reads by inode. - **Batch directory and small-file prefetching**: Directory listings (`readdir`) return entry attributes inline, then batch-prefetch up to 32 small regular files (`<= 64 KiB`, up to 1 MiB total) via `reads` and up to 32 child directories via `readdirs` in one trip. - **Server-pushed invalidation**: Instead of polling for stale attributes, the mount listens on one WebSocket or chunked `join` stream for server-pushed cache evictions and 2-second heartbeats. The framing overhead of reading a 1 GiB file should not be ten times that of a 100 MiB file; the goal is constant cost. Once those bottlenecks (the 250,000-syscall copy loop and the per-file round trips) are out of the way, `fs-relay` also keeps tiny files and chatty operations lean by encoding frames with FlatBuffers for in-place reads with no new allocations (a recurring lesson for anyone who has watched [serialization overhead dominate Go hot paths](/posts/go-json-v2-explained/)). That is why the repository benchmarks test both scenarios, and the numbers speak for themselves. On small files, unpacking `ansible.tar` (27 MiB across 5,868 files) takes 203.56 seconds on `s3fs` (mounted over `rustfs`) and 107.89 seconds on `fuse-nfs`, while `fs-relay` finishes in 15.47 seconds. On large files, unpacking `Antigravity.tar.gz` (500 MiB) takes 36.13 seconds on `fuse-nfs` and 21.01 seconds on `s3fs`, compared to 9.78 seconds on `fs-relay`. Even more surprising, running DuckDB queries over Parquet files mounted through `fs-relay` beat DuckDB reading the exact same files directly from `rustfs` over native S3. ## Upstream Work Toward v0.2.0 Muayyad's code and `SPEC.md` are clear enough that jumping in was easy. The first contribution ([`9071dbd`](https://github.com/muayyad-alsadi/fs-relay/commit/9071dbd)) proposed dual licensing under MIT and the Waqf Digital Public License (`WaqfDPL-1.0`), which Muayyad adopted so the project stays free and open in perpetuity. From there, the work covered four practical engineering problems. ### 1. Server-Authoritative POSIX Locks and SQLite Coherence To run SQLite across two remote mounts without corruption, both mounts must agree on byte-range locks (`getlk`, `setlk`, `setlkw`) and `flock`. [PR #3](https://github.com/muayyad-alsadi/fs-relay/pull/3) added the server-side lock table in `internal/local/lock.go`. A classic POSIX lock belongs to a `(PID, inode)` pair, which breaks across machines when two hosts reuse the same PID. In `fs-relay`, a lock holder is the mount's HMAC-signed client ID paired with the kernel lock owner. Because SQLite's rollback journal hammers three exact byte offsets at `0x40000000`, the lock table tracks dedicated fast-path counters for the pending byte, the reserved byte, and the 510-byte shared range, spilling arbitrary ranges onto an overflow list only when needed. Getting the lock table working was only half the battle. Enabling a dedicated `go test -race` job in CI ([PR #25](https://github.com/muayyad-alsadi/fs-relay/pull/25)) and running multi-writer SQLite stress tests exposed subtle distributed races, fixed across [PR #24](https://github.com/muayyad-alsadi/fs-relay/pull/24), [PR #30](https://github.com/muayyad-alsadi/fs-relay/pull/30), [PR #33](https://github.com/muayyad-alsadi/fs-relay/pull/33), and [PR #34](https://github.com/muayyad-alsadi/fs-relay/pull/34): - **Draining write-behind before unlock**: If mount A writes a transaction and unlocks before its buffered bytes reach the server, mount B acquires the lock and reads stale bytes. `fs-relay-bind` now drains dirty buffers before sending `F_UNLCK` and drops cached attributes and prefetched file bodies the moment a lock grant arrives. - **Closing wake-up races**: When a writer downgrades to a shared read lock or a waiter receives `EAGAIN`, registering the waiter under the lock mutex ensures coalesced broadcast wake-ups never miss a blocked reader. - **Guarding late grants**: If a file handle closes while a lock request is still in flight, the client immediately releases the late grant so orphaned locks never wedge the namespace. We verified this on Linux 6.12 with SQLite 3.46.1 in rollback journal mode: four writers split across two mounts committed 200 rows concurrently, and `PRAGMA integrity_check` returned `ok` on both mounts. (`WAL` mode stays out of scope because it relies on an `-shm` file mapped into shared memory via `mmap` on a single kernel, which cannot be shared across separate machines.) ### 2. Startup Validation and Single-Server Data Locks Early on, a typo on the command line or a second `fs-relay` process pointing at the same data directory could cause confusing state. Across [PR #16](https://github.com/muayyad-alsadi/fs-relay/pull/16), [PR #17](https://github.com/muayyad-alsadi/fs-relay/pull/17), [PR #18](https://github.com/muayyad-alsadi/fs-relay/pull/18), [PR #20](https://github.com/muayyad-alsadi/fs-relay/pull/20), [PR #29](https://github.com/muayyad-alsadi/fs-relay/pull/29), and [PR #35](https://github.com/muayyad-alsadi/fs-relay/pull/35), we added non-blocking `-token-file` secret loading, default `http://` URL schemes, pre-mount directory validation (`-force-existing-mnt`), and an exclusive `.fs-relay.lock` file lock on the data root hardened against symlink traversal. ### 3. Bounded Unmount on Interrupts Anyone who has pressed `Ctrl+C` on a FUSE daemon while a shell is still sitting inside the mounted folder knows how easily shutdown can hang. In [PR #19](https://github.com/muayyad-alsadi/fs-relay/pull/19), [PR #37](https://github.com/muayyad-alsadi/fs-relay/pull/37), [PR #38](https://github.com/muayyad-alsadi/fs-relay/pull/38), and [PR #39](https://github.com/muayyad-alsadi/fs-relay/pull/39), we tightened CLI argument validation and reworked signal shutdown in `fs-relay-bind`. On `SIGINT` or `SIGTERM`, `fs-relay-bind` attempts a clean unmount bounded by a 2-second timeout. If open files or a stalled network flush keep the mount busy, it falls back to an interruptible lazy detach (`fusermount3 -uz` or `MNT_DETACH`) and exits non-zero with a clear message instead of hanging the terminal. ### 4. Two-Phase Atomic Self-Update in v0.2.0 For `v0.2.0` ([PR #40](https://github.com/muayyad-alsadi/fs-relay/pull/40)), we added `update`, `version`, and `help` subcommands to both `fs-relay` and `fs-relay-bind`, alongside a 12-hour cached background release check (`FS_RELAY_NO_UPDATE_CHECK` disables it). Updating two companion binaries (`fs-relay` and `fs-relay-bind`) in place has sharp edges. If the first binary swaps and the second fails, the machine is left running mismatched versions. Our updater verifies the release tarball against the GitHub SHA-256 digest (falling back to `checksums.txt`), preserves existing file permissions (such as `0700`), stages both replacement binaries in exclusive temporary files in the target directory, and rolls back the primary binary automatically if the companion commit fails. ## Try fs-relay v0.2.0 There is genuine satisfaction in contributing to software that refuses unnecessary layers. You do not need root daemons, kernel modules, or heavyweight cluster coordinators to mount a directory across machines. Standard HTTP, careful FUSE caching, and server-side POSIX locks get the job done in five small packages. Starting a share, mounting it on a client, and updating both binaries in place takes three commands: ```bash mkdir -p ./data/demo /mnt/demo fs-relay -listen 127.0.0.1:8080 -data ./data -token-file ./token.txt fs-relay-bind -url 127.0.0.1:8080 -ns demo -token-file ./token.txt /mnt/demo fs-relay update ``` You can grab the `v0.2.0` binaries or read the design contract in [Muayyad Alsadi's fs-relay repository](https://github.com/muayyad-alsadi/fs-relay), or check the summary card on the [project page](/project/fs-relay/). Have you tried replacing `sshfs` or `rsync` loops with user-space FUSE over HTTP in your own workflows, or are you still nursing old NFS mounts?