There’s a special kind of dread when your NAS tells you files are damaged. It’s worse when the drives say they’re perfectly healthy. This is the story of a week spent chasing corrupted films across my brand-new ZFS pools, and the culprit that turned out to be hiding in plain sight.
The setup
My homelab runs on one box: an older Intel desktop, running Proxmox. TrueNAS SCALE runs inside it as a virtual machine and looks after a set of NAS drives holding media, downloads, photos and family files.
Nothing exotic. Plenty of people run exactly this kind of setup.
Act one: 10,000 errors and a bad design decision
It started when TrueNAS flagged one of the pools with over 10,000 checksum errors. The drive’s SMART data was spotless. No bad sectors, no warnings, nothing.
The cause was how I’d built it. Instead of handing TrueNAS the real disks, I’d given it virtual disks sitting on LVM inside Proxmox. ZFS is designed to talk to real hardware. It checks every block it reads against a checksum, and it trusts that what it wrote is what the disk stored. Put a layer of virtualisation in between, and that trust breaks down.
So I did the migration properly:
- Copied all the data off, one drive at a time.
- Rebuilt each drive as a whole-disk passthrough, so TrueNAS sees the actual drive.
- Copied everything back.
Even this wasn’t smooth. zfs send | zfs recv crashed with core dumps partway through some of the big datasets. The workaround was a small script that sends one dataset at a time, logs any failure and moves on to the next, with rsync as the fallback for anything that wouldn’t send.
Problem solved, I thought. Errors went from over 10,000 to… five.
Act two: five is not zero
Five checksum errors on a brand-new pool, on a healthy disk, is still five too many. And they weren’t random noise: they were real, unreadable blocks in real files. Three films on the first drive came back with “Input/output error” when I tried to copy them.
Then the second drive, freshly rebuilt, on its own passthrough, with a completely clean SMART report, finished its first scrub with three damaged files of its own.
This is where it got interesting. One of those files was a 4K movie. I’d copied it from the first drive with rsync, which checks the data as it reads. So it left the first drive intact. Somewhere between reading it and it landing on the second drive, it got damaged.
Two different drives, each healthy by every measure the drives themselves could report, both picking up the odd corrupted block. Plus those zfs recv crashes. When separate healthy disks show the same kind of random corruption, you stop looking at the disks and start looking at what they have in common.
What every single byte passes through on its way in and out: the RAM.
Act three: the RAM
I checked what was actually installed. It was a textbook recipe for trouble:
- Two different memory kits mixed together. Two pairs of different-sized sticks: same brand, different kits, different builds.
- All four slots filled, which puts more strain on the CPU’s memory controller.
- XMP switched on, running the memory above its standard speed. XMP is an overclock. It’s tested for each kit on its own, not for two kits mixed together in the same machine.
Any one of those on its own is usually fine. All three together is exactly the kind of setup that works perfectly… apart from the occasional flipped bit. On a gaming PC you might get a rare crash and shrug. On a NAS, a flipped bit in the wrong place becomes a permanently damaged file. That’s exactly what I was seeing.
The twist
The plan was simple: borrow a matched kit of bigger sticks from another PC, put it in the server with XMP off, run Memtest86+, and enjoy twice the memory, all of it stable.
Memtest passed with zero errors. But Proxmox only showed half of it.
My server’s CPU is several generations old, and its memory controller has a maximum size per stick. Put in a stick bigger than that and it can only use part of it. On top of that, both sticks had gone into the same memory channel, so it was running single-channel as well. It passed its test; it just wasn’t the upgrade I’d planned.
The fix that stuck: a single matched pair of supported sticks, one in each channel (proper dual-channel), with XMP off so it runs at standard speed. Memtest passed again. Less memory than I’d hoped for, but fully supported and fully stable. The bigger upgrade will have to wait for a second matched kit.
The proof
A clean Memtest is encouraging, but it isn’t proof. Bad RAM can pass a test and still misbehave under real load. The real test was a full scrub of both pools, where ZFS reads every single block and checks it.
- Drive two: 6 hours 18 minutes. It found only the two damaged films we already knew about.
- Drive one: 13 hours 36 minutes. It found ten damaged files.
Ten sounds bad, until you look at when they were damaged. Every one of them also appeared in snapshots taken before the RAM swap. They’d been damaged back then, and this was simply the first time anything had read those blocks. Everything written after the swap came back clean: several hundred gigabytes of downloads and file moves.
That was the result I wanted. The RAM was the cause, and it’s fixed.
Cleaning up
- The two damaged films on drive two: I still had good copies from the migration, so I restored them and checked them against the originals, byte for byte.
- Six damaged TV episodes: simply deleted and re-downloaded.
- Old migration snapshots: they were still holding the damaged blocks, so they went.
- Error counters: cleared. Both pools now report zero errors.
Total loss: a handful of films and episodes, all replaceable. None of my photos or family files were touched, and I’ll be honest, that’s partly luck.
What I learned
- Never mix RAM kits in a server. Buy one matched kit for the slots you need.
- Turn XMP off on anything that stores data you care about. The speed difference on a NAS is unnoticeable; the stability difference isn’t.
- Check your CPU’s memory limits before buying. Older CPUs often cap how big each stick can be.
- Give ZFS the real disks. Don’t hide them behind LVM or virtual disks.
- ZFS saved me. Without checksums, none of this would have shown up. Files would simply have been quietly broken, and I’d have found out years later when a film wouldn’t play or a photo wouldn’t open.
- “The drive says it’s healthy” doesn’t mean your data is. When healthy drives show corruption, look at what they share.
- Scrub regularly, and keep backups. Since this happened, my important files are snapshotted and backed up every night.
If your NAS ever shows checksum errors on a drive that looks perfectly healthy, don’t just replace the drive. Check your RAM first.
This was a long week wondering if my drives were toast, and it turns out to be the memory not faulty but not matched.