"Remounting filesystem read-only": your site goes down with a 500, and the disk isn't full

The symptom: an instant 500, and SSH that stops answering

The store, a PrestaShop instance on a virtual private server, went down all at once: blank page, 500 error, and the first thing that stands out is how fast the response comes back, 0.02 seconds. A "real" PHP 500, caused by an application error, usually takes a few tenths of a second, because it has to load the framework first. A response that fast is almost always a sign the error happens much earlier, at a layer that has nothing to do with the store's code.

The second clue: SSH into the container was timing out. Not "connection refused", an actual timeout, as if something had gotten stuck halfway through. Inside the container, a plain touch test returned:

touch: cannot touch 'test': Read-only file system

The filesystem had remounted itself read-only. The physical node hosting the container, checked right away, was fine: normal load, no hardware errors reported.

What it looked like, and wasn't

A filesystem that flips to read-only points, in order, to three things. None of the three was the right one, but ruling them out one at a time is what led to the real cause.

• "The disk is full." The first guess, and also the easiest to check: df -h inside the container showed 186 GB used out of 1.5 TB available, 14%. Not it. A disk that's genuinely out of space gives a different and much clearer error ("No space left on device"), not a read-only remount.
• "The physical disk died." A read-only remount is also the typical ext4 behavior when it hits I/O errors from a failing drive. But the check on the physical node showed no hardware errors, no bad sectors, no array warnings. The disk was healthy.
• "MySQL got corrupted." One of the configuration tables did come back marked as crashed once the site was up again. But that was a consequence of the read-only filesystem hitting mid-write, not the cause of the incident: MySQL cannot have broken a filesystem it doesn't manage.

The physical node's kernel log, checked after ruling out the first two guesses, told a precise story:

Buffer I/O error on device dm-4
EXT4-fs warning: I/O error 3 writing to inode ...
Aborting journal on device dm-4-8
EXT4-fs error (device dm-4):
ext4_journal_check_start: comm mysqld:
Detected aborted journal
EXT4-fs (dm-4): Remounting filesystem read-only

The real cause: the pool underneath the disk was full, not the disk itself

The container's volume isn't a dedicated physical disk: it's an LVM volume with thin provisioning, sitting on a pool shared with other virtual machines on the same node (the mechanism is identical on any hypervisor using thin-provisioned LVM storage, Proxmox in this case, but it isn't specific to that product). With thin provisioning, the volume "sees" its declared size, 1.5 TB, but physical blocks only get allocated from the pool when they're actually needed. If the underlying pool fills up, writes fail at the block level, even though the container's own volume still reports free space from the inside: the 14% shown by df was entirely true, and entirely beside the point.

Checking the pool confirmed it: Data% at 100.00%, with the status attribute reporting the letter D, out of data space. Pool full, writes rejected at the block level, ext4 seeing I/O errors, aborting its journal to protect data consistency, and remounting read-only. From that point on PHP and MySQL can't write anything, and the store answers 500 to every request.

The pool hadn't filled from a single event, but from three problems stacking on top of each other over time:

1. Old snapshots that were never cleaned up. An automated snapshot job had been disabled in the past, but without deleting the snapshots it had already created. They had been sitting there for weeks, holding on to blocks that would otherwise have been freed.
2. The pool's autoextend threshold was set to 100%. A pool that only autoextends once it's already full effectively never autoextends in time: there was no margin to react before it saturated, and no early warning.
3. No periodic discard/TRIM. When a file is deleted inside the container's filesystem, the blocks it occupied stay allocated in the pool until something runs a TRIM that hands them back. The math, done after the incident, was stark: 1.40 TB allocated in the pool against 186 GB actually in use inside the container, over a terabyte of "phantom" space that was never returned.

The final trigger was a routine backup: its snapshot got created on a pool that was already at its limit, and within minutes the pool saturated completely.

The fix

The intervention went in this order, most urgent first, most permanent last:

1. Emergency pool extension (non-destructive, as long as there's headroom in the underlying volume group): the pool drops back under 100% and writes start working again.
2. Removing the failed backup's snapshot, safe because the backup system recreates it on the next run anyway.
3. Removing the old, unused snapshots, with the client's explicit approval before touching them: on their own, they accounted for most of the space recovered. The container's live volume was never touched, verified both before and after.
4. e2fsck with the container stopped, to replay the journal and fix the inodes left orphaned by the interrupted write.
5. Database integrity check: MySQL's automatic repair had already kicked in on its own for the table flagged as crashed; an explicit check across every table at risk confirmed the data was intact and identical to before the incident.
6. Lowering the pool's autoextend threshold, so next time the system extends itself with margin, instead of waiting until it's already full.

Recovering the "phantom" space with TRIM is a separate job: an unprivileged container can't run it from the inside, and with the container running it often can't be run from the outside either. It needs a short maintenance window with the container stopped, dedicated to that alone.

How to verify it holds

• Track 5xx errors over time, not just whether the site answers again: in this case, every error was concentrated in the single minute of the restart (container booting, MySQL mid-repair), and from there on, zero errors across thousands of subsequent requests. An isolated error at restart is normal; one that keeps happening isn't.
• Watch the pool's fill percentage over time, not just right after the fix: a pool that starts filling up again at the same rate as before means the underlying causes (snapshots piling up, an overly permissive autoextend setting) were postponed, not actually removed.
• Run an explicit check across the database tables after any incident like this, even if the site already looks fine: a read-only remount hitting mid-write can leave a table in an inconsistent state without any error showing up on the page right away.
• Schedule a maintenance window for TRIM if the volume has built up a lot of phantom space: without it, the real margin stays lower than what the pool's numbers suggest.

The store this happened on stays anonymous on purpose: what matters here is the mechanism. If a site goes down with an instant 500, SSH times out, and the filesystem turns out to be read-only, the first thing to check isn't "how much disk is left inside the container": it's how much space is actually left in the pool underneath it, because those are two different numbers, and only one of them, when it hits zero, takes the store down.