kellyfelkins.org: hard disk failure

hard disk failure

The Story of the Dying Disk

I just replaced the primary hard disk in this system. Here’s the story:

I began to notice weeks ago that my system seemed a bit sluggish. This server is a few years old and probably under configured for what it is being asked to do. I’m constantly piling more stuff on it and for the most part it hums along without complaint. I attributed the periodic sluggishness to insufficient memory and associated swapping.

A few days BC (before crash) I decided there was something more to it than swapping. I’m not sure exactly why. I think it had to do with some specific operation that I was doing over and over again. Thinking that little else was going on on the system, it didn’t make sense to call the slowness “swapping”—I mean, there was no time or reason for this to be swapped out. Also I think I noticed that some operations would slow down in the middle – perhaps something was loading and it would load part of it fast, then get real slow.

I started looking for a problem. Nothing new in the system logs that I noticed. I realize this is all very vague, but I decided it was the disk and that the disk was having trouble. I started trying to monitor the disk health.

I installed S.M.A.R.T. Monitoring Tools. SMART is a standard for reporting the health of a disk. Disks are pretty, ah, smart these days, and they track lots of things that are related to their health. SMARTMON interacts with the disk to report and control this monitoring.

The instant smartmon tests reported some problems, but concluded that the disk was OK, I think. I asked the disk to perform the more comprehensive tests—tests which apparently take 30 minutes or more. It never finished. The test was aborted. I tried this a few times and never got a complete test. The system reported that the test was canceled on request. I did not cancel the tests and I haven’t figured out what did cancel the test. Some have reported that a system going to sleep would cancel a test though my system is not configured to sleep.

I spent days, perhaps a week, messing with this. I thought I had time. At some point I decided that my disk was hurting and that I needed to replace it, so I bought a new drive and brought it home.

It wasn’t till then that I decided to make a backup.

This was a crucial mistake. What I perceived to be a very gradual slide into a non-optimal but still working state was really a drive plunging off the cliff of its life.

An aside: I may have contributed. Did I accelerate the destruction of this drive? Often you (read “I”) aggravate the situation more than you (I) help. I’m an amateur administrator of these systems and I don’t see failures like this very often, so I’m always learning or relearning, since the last failure happened so long ago. When it was not working well I tried to “tune it up” with hdparms. I tried several settings but none moved the drive performance above “abysmal”.

Of course the backup failed. Now the drive was reporting consistency problems. Files that appeared to be there were unreadable. I made several attempts to salvage pieces and had some success. Unfortunately there were recent files that were not readable.

I’ve put the system back together via a combination of a 6 month old full backup, fragments that I was able to salvage from the failing drive, and various copies from other systems.

Lessons Learned

Take Your Backups Seriously: See my recent article about backups for suggestions.
Aggressively Fix Failing Disks: I’m convinced that I could have saved everything had I moved more quickly.

Share this article on

* * *