ch4-odirect

ch4 - O_DIRECT: the contract (not written yet)

The question this chapter will answer: when does bypassing the page cache win, when does it lose, and what exactly do you have to rebuild yourself to make it win?

Nothing here runs yet. make ch4 says No rule to make target 'ch4-odirect/directread', because the alias is wired and the source file is not written. Drop a directread.c in this directory and it builds with no Makefile edit.

The plan

directread <path> <offset> <length> reads a byte range with O_DIRECT, obeying whatever ch0's dioalign reported for that file.

There are three independent alignment rules, not one: the address of your buffer in memory, the file offset, and the length. Confirming that each is enforced separately, by misaligning exactly one at a time and cataloguing the errno, is the first experiment.

The rest of the chapter is a number, not a program. Sequential scan of a large file at queue depth 1: buffered read() in 64 KiB chunks against O_DIRECT in 4 KiB chunks. Direct I/O should lose badly, because you gave up the readahead that ch1 measured at up to 32 pages. Then run O_DIRECT again with 1 to 4 MiB chunks and watch it come back. The kernel's prefetch is gone, so you are the prefetch now. That is the contract.

The proof that the bypass is real reuses ch1's instrument: drop caches, O_DIRECT-read the whole file, point resident at it, and see all dots. A buffered read of the same file leaves all hashes.

Traps already known

• O_DIRECT is not supported on tmpfs. /tmp may well be tmpfs, and you would get a permanent EINVAL and conclude your alignment math is wrong. Work on a real block-backed filesystem.
• posix_memalign returns the errno as its return value and does not set errno. Checking errno afterwards reads stale garbage.
• Drop caches between the buffered and direct timing runs, or the buffered arm wins trivially by reading from RAM the previous run warmed.
• Time with CLOCK_MONOTONIC, never wall clock.
• Build the timing runs without sanitizers. ASan distorts I/O timings badly.

Read

• man 2 open, the O_DIRECT section and its NOTES. It is the best short essay in the man pages on why a flag ended up the way it is, and it states the alignment rules outright.
• man 3 posix_memalign, and note what it returns.
• Linus's 2007 lkml message calling O_DIRECT "deranged", for the counterposition to every database engineer who relies on it.