A working notebook on systems, databases and the web platform. Long-form teardowns of real production problems — the profiling output, the wrong turns, and the numbers that finally settled it.
It was not the database, the GC, or the network. It was a debug log statement and a mutex on the hot path — and it only showed up at p99.9.
Every post has a reproduction, a measurement, or a rollback.
Row estimates off by 4000×, a seq scan that should have been an index-only scan, and the four lines of the plan that actually matter.
Hindley–Milner in an afternoon: unification, substitutions, occurs checks, and the one place where the naive algorithm blows the stack.
Zero-copy is a syscall-level promise, not a free lunch. Benchmarks on Linux 6.8 — and why your TLS layer copies everything anyway.
At-least-once delivery is the default. Designing a key store that holds under duplicate in-flight requests, mid-write crashes and clock skew.
One component contract instead of four media query blocks. What broke, what got faster, and the Safari bug we shipped a workaround for.
DNS, TCP, TLS 1.3, HTTP/2 prioritisation — measured on a real connection from Lisbon to Frankfurt, with a waterfall you can reproduce.
Coordinated omission, missing warm-up, and the p-value nobody wanted. A review checklist for measurements that survive a skeptical reader.
Six indexes on a 200M-row table: modelling the write amplification, the cache footprint, and the point where the planner stops caring.
Submission queue back-pressure, kernel workers that ignore your nice level, and why the 5.10 LTS baseline matters more than the benchmark.
A toy storage engine with a WAL, an LRU cache and a B-tree you can actually debug.
sendfile, epoll, io_uring — what the syscalls promise and what they really cost.
From DNS to first paint: every hop that costs you a millisecond.
Notes on papers that changed production code — and the ones that didn't.
EXPLAIN (ANALYZE, BUFFERS) is the only plan worth trusting. Estimated rows are a guess; buffers tell you where the I/O actually happened.
Found it: an Arc<Mutex<…>> clone held across an .await. 2% of requests were queueing on a lock nobody needed to hold.
Rust tip: #[inline(never)] on a suspicious function makes its frames far easier to read in perf. Costs nothing, saves an hour.
curl -w "%{time_connect} %{time_appconnect}" beats any RUM dashboard for a first triage. Ten seconds, no instrumentation.
Half of the “the database is slow” tickets end the same way: someone added an index and skipped ANALYZE.
No growth hacks, no sponsored tool round-ups. Just the write-up of whatever broke in production that fortnight, usually with the flame graph attached.
I work on storage and query engines. Before that, six years of backend infrastructure and a long stretch of writing parsers for a living. This page is where the debugging sessions get written down — usually three weeks after I stop being annoyed about them. Corrections welcome, especially the ones with a benchmark.