OWC Mercury Elite Pro Qx2: Data Recovery

OWC Mercury Elite Pro Qx2
OWC Mercury Elite Pro Qx2

This OWC external enclosure is a common sight on the desks of Mac users with big storage needs. It’s a pretty standard 4-bay box, styled somewhat like a cousin of a PowerMac G5 or 1st generation Mac Pro. Inside are the usual options of RAID 0 to RAID 5 with a few additions like JBOD & RAID 10 thrown in for good measure. There are a few variations of this device but the back panels commonly have USB, Firewire, and eSATA ports for direct connection to a PC or Mac. There is no ethernet port on these drives which makes the Qx2 a DAS (Direct Attached Storage) rather than NAS (Network Attached Storage).

Aside from massive name, the OWC Mercury Elite Pro Qx2 also comes with a potentially huge amount of storage. Currently up to 32TB on the OWC store, but also available diskless or BYOD (Bring your own disks). With so much storage space, these drives often become the one and only repository for vast lumps of important data. The benefits of RAID give a false sense of security that the data is safe from drive failures. Unfortunately, there are a number of reasons why the RAID array alone will not protect from certain failures. Most of  these failures can be overcome by us in our workshop, but they are not one-button fixes. It is helpful to understand why a seemingly rock-solid platform can be even more risky than a simple external USB drive.

Redundancy

Under common settings, the Qx2 will use RAID 5 for the array. With four 2TB drives this gives you a 6TB volume on a Mac or ~5.5TB on a PC[1], and can cope with a single disk failure. There is a lot of debate about how good RAID 5 really is for such large drives[2]. In our example this means that if a single disk fails, it will need to be replaced, and then the new disk rebuilt with 2TB of data calculated from the other disks. This will take many hours, even under optimal conditions, but if anything goes wrong before it completes the array could stop showing up all together. At this stage, the data is probably recoverable but don’t panic. One wrong move and the data could be gone for good.

If the data is crucial then get assistance from a RAID recovery service now and you should get back all or most of the data.

If any disks are removed or replaced at this point the array could get reinitialised and either make the recovery more complicated or wipe the data completely.

Other Failures

Aside from all the problems with a RAID setup, the volume could also fail in the same ways that a standard hard drive can. There could be deleted files, a reformatted or corrupt partition, or even the RAID controller failure. RAID cannot protect against those types of failure at all.

Recovery

Our first step would be making read-only copies of each disk in the array. This protects against further disks failing, and also allows us to work from copies without risking the original disks. In fact, once the disks are copied, we put the originals to one side and don’t touch them again until all the data is recovered and supplied back to the user.

Once we have our copies, they are loaded into our own hardware where we recreate the RAID in a virtual environment. Again, we don’t use the original hardware, as that may have been the root cause of the problem.

When the virtual RAID has been loaded and all the data extracted, the files are supplied back on whatever alternative storage is suitable, (not the original device!) Once the data has been delivered to the user, and backups made, the old unit can then be destroyed, or returned and reused.

Avoidance

Anyone using RAID on a regular basis should know that RAID is not a replacement for backups. If anything, the increased number of disks makes failure more likely. This needs to be addressed by either making backups to another device, or an online service (preferably both). You ideally want backups that keep historic versions of the files, so that inadvertently deleting a file or changing a file by mistake will not also replace the backup version.

If you are having problems with an OWC Mercury Elite Pro Qx2, give us a call or send a message via the form on this page. We give free advice and could help you avoid permanent data loss.

1. Macs now use 1000 bytes for 1KB but PCs use 1024 bytes.

2. Even RAID 6 does not solve the long time required to rebuild a disk, even though it allows for two disk failures.

Data Recovery from Fire Damaged Hard Drives

Data Recovery from Fire Damaged Hard Drives
Fire Damaged Hard Drives

Following a fire at a science lab, five hard drives had sustained damage. Although hard drives often survive a fire, they can sometimes be damaged more by the water used to bring the blaze under control. Fortunately for the lab involved, somebody had managed to extract the hard disks from the scene and quickly bag them. It is critical to work fast with fire and water damage, before corrosion takes hold.

The lab have been using our data recovery services since 2005 so the technical department knew exactly where to send the disks.

Cleaning and Decontamination

Data Recovery from Fire Damaged Hard Drives
Data Recovery from Fire Damaged Hard Drives

To maximise the chances of recovery, we have a strict procedure for fire and water damaged disks. First the outside covers of the drives are cleaned of any loose soot and all electronic components are labelled and removed. The hard drive carcases then get thoroughly cleaned and inspected for signs of water ingress or damage to the protective seals. If the damage has transferred inside the drive then it will be taken into our cleanroom for internal decontamination and cleaning. The electronic parts of the disks are dealt with separately. First they are dusted of loose debris, and then immersed in a chemical bath. This removes contaminants from any connectors or contact surfaces, and also helps remove anything that could cause the electronics to malfunction when powered on. The circuit boards are then dried and tested for faults before being reattached to the hard drives.

Recovery

Once cleaning and decontamination is complete, the drives are reassembled and attached to an imaging machine. The drives are copied as fast as possible, as they may have been exposed to temperatures outside of their specified design. This process means that each sector is only read a single time and then the disk is powered off and returned to storage. We are then free to work on the copies. It is part of our standard data recovery procedure, but all the more important in this case.

Success

Our strict and thorough process for fire and water damaged drives meant that we had a 100% success rate from these drives. Failure to follow any part of the process could have meant the difference between the data being recovered or not.

Fire Damage Data Recovery

Battle Cards

For a limited time only we have produced collectible battle cards, complete with hard drive portraits and battle stats. There are 10 to collect. Some are based on common hard drives, but others are a bit more obscure.

Classic Hard Drive Battle Cards
Classic Hard Drive Battle Cards

We’ve also got plans for a few new ranges, so get these new cards while you can. These cards are not for sale so contact us if you want some.

Death Star Battle Card
Death Star Battle Card
Classic Hard Drive Battle Card
Classic Hard Drive Battle Card

iOS vs Android Secure Erasure

I’ve previously written about the difficulties recovering data from modern iOS devices. This new post by Avast(!?) shows it from another angle.  What happens when you sell an old device?

TL;DR

Because iOS uses hardware encryption on the main storage, recovery is almost impossible without the passcode. In contrast, Android phones (usually, by default) don’t encrypt the main storage. Also they allow external SD cards which may not be encrypted either. This means if an iPhone and an Android phone are sold or lost without being carefully erased, the iPhone will not easily give up the data but an android phone will. It also means if you forget your iPhone passcode you are unlikely to ever get the data back.

 

How To Ruin Your Chances Of Data Recovery

No Chance Data Recovery

Maybe the title is a bit harsh, but in the past week I have seen an unusually high number of hard drives ruined by avoidable problems. There is nothing more frustrating than knowing the data would have been recoverable if  the hard drive hadn’t been tampered with first.

Manufacturing

Hard drives are manufactured in a controlled environment. Staff wearing white overalls,  gloves, and masks, control machines which are carefully organised to prevent contamination getting inside the hard drive.

Clean Room

To prevent contamination when repairing the inside of a hard drive, it is necessary to use a cleanroom. This is a specially designed system that filters the air and keeps airborne particles to a minimum. A cleanroom is the only safe way to open a hard drive. We have one of these, which allows us to carry out the most intricate repairs without any risk of particle damage.

Damage

Before opening a hard drive for internal rework, it is crucial to confirm that there is an internal mechanical problem in the first place. For instance, our data recovery process has numerous tests we can carry out before we confirm that the problem is under the hood. Only then will we open the top cover and check for damage inside. Unlike CDs or DVDs, the internal disks of a hard drive are not designed to come into contact with normal air. Even a fingerprint could mean the difference between a successful or unsuccessful recovery. If you slip with a screwdriver then forget it.

The numbers

On a percentage basis of all the failed drives we receive (hundreds per year), we only need to go to the cleanroom with around 10%. There are likely to be far less intrusive ways into the data without ever taking off the top cover.

Even if…

I’ve lost count of the number of times I’ve been told, “It’s an internal fault. The heads are clicking.” Although some drives will click as a symptom of failed heads, there are so many other reasons to cause a drive to click that it is not a reliable way to diagnose problems.

Reasons a hard drive will click

  • Electronic fault
  • Firmware fault
  • Bad sectors
  •  Weak (but not completely failed) heads
  • Problems reading partition info

All of the problems above can be overcome without ever unscrewing the top cover. In fact, removing the top cover will only introduce more doubt to the diagnosis. If the top cover is removed outside of a cleanroom, then not only do the above problems need to be solved, but also new problems of contamination and possibly damage by tools.

(Don’t) Get the discs out

Another common wrong diagnosis is to take the disc pack from one drive and place it into a donor drive. Although this is a correct course of action for some drives, there are some serious implications. First, when the disc pack is built it is clamped together onto the spindle motor. This alignment is so crucial that if you remove one disc at a time, you will never be able to regain this alignment. This was true 20 years ago when the magnetic data was nowhere near as densely packed. Rotate the discs a fraction of a millimetre and you can wave goodbye to all the files.

It is also the case that most of the hard drive firmware is now stored on the discs, so moving them to a new donor will not help if the fault is firmware based.

Advice

If you are even considering a destructive course of action, at least get some professional advice first. You don’t need to follow the advice, but at least it gives somebody (maybe me) the option to give a warning. You don’t want to find out later that the data would have been simple to recover, if only…

2TB Western Digital 2nd Opinion

Hard Drive Data Recovery

Case Study

Drive: WD20EARS-00MVWB0
Problem: Clicking. Previously diagnosed by a third party as unrecoverable due to media damage.

We are always keen to test our services against our competitors, just to make sure we’re still up there with the best of them. When we heard from one of our service partners (ABC Rawpaw) that one of the top ranking data recovery companies on Google had given up on a customer’s drive, we were curious to take a look. It’s also worth mentioning that the client had paid £234 to the other company, despite them recovering no data.

Third party report

We received a copy of the original diagnosis report with the hard drive. The report mentioned incorrect head alignment and also damage to the head assembly preventing the drive from functioning correctly. It also stated that they had subsequently replaced the old heads with new ones but due to suspected disc damage, recovery was unsuccessful. A plausible enough report, but was it accurate?

Booking in

The drive was booked into our process. We noted that the warranty seals had been removed and the top cover showed signs of being previously opened. Whenever we receive a drive in this state, we always want to make sure the drive has been rebuilt correctly. We need to do this in our cleanroom to prevent any contamination getting in from the air. Our office is clean, but it’s no place to open a hard drive.

Findings

The inside of the drive was examined in our cleanroom and found to be spotlessly clean. There were no signs of disc damage or particles, but we did notice that the screws were not factory-tight, suggesting previous work had taken place. Happy that there was nothing untoward inside the disk, it was rebuilt for further diagnosis. The drive was powered on and initially failed to reach a ready state. This is common for Western Digital drives as they often have firmware corruption. We used our proprietary firmware tools (and John’s keen knowledge) to repair some of the firmware area of the drive which then allowed the drive to reach a ready state and allow access by our imaging tools.

Imaging

We did find that one of the heads was not performing within spec, but we were able to work around this, again using specialist tools. After imaging the majority of the drive we replaced the heads to allow better access to the missing areas. This helped improve the amount of successfully recovered data.

Result

We ended up with over 800GB of data recovered from the drive in good condition, even though one of the biggest companies (according to Google) was unable to get anything! This is not the first time we’ve recovered data where others have failed, but we always wonder how many people give up on ever seeing their data again, even though it may be recoverable with the right knowledge.

In the end we had an extremely happy customer (and service partner), and were able to test ourselves against one of the biggest data recovery firms in the UK. A great result for us that shows the importance of getting a second opinion.

Mavericks (10.9) Western Digital Data Loss (UPDATED)

Portsmouth & Southsea Mac Support

We’ve been hearing reports of data loss with certain external hard drives after an upgrade to the latest Mac operating system Mavericks. The blame seems to fall to the disk management software that Western Digital bundle with their external hard drives. In response WD has removed WD Drive Manager, WD Raid Manager, and WD SmartWare from their website until they figure out the problem.

We have already had first-hand data recovery enquiries for such disks, so it remains to be seen if these issues can be resolved, or if the data is permanently damaged. One notable case involved a 6TB RAID with two 3TB drives in a mirrored array. After the update to Mavericks, the volume appeared as an empty 3TB drive.

It’s worth noting that even if you use a Western Digital RAID drive set as a mirror, this problem can still cause data loss. Both disks in the mirror are susceptible to the same problem. Remember that RAIDs need to be backed up even more than single disks!

More info and links at MacRumours

Update – 8-11-2013

We’ve now completed the recovery process for one of these hard drives. It appears that the drive we received had been formatted to an empty MyBook volume. The way the Mac filesystem stores data means although most of the files can be recovered, it is not possible to restore the original file and folder structure. This causes issues for files like InDesign which link to external files by name when you add graphics to the document.

We’ve not yet seen enough of these disks to know if this is a common outcome for all affected disks.

What To Do When You Run Out Of Space

It’s a common problem that as we generate more data each year we start running out of space to put it. This is now even more of an issue in the smartphone market, where built-in cameras are generating increasingly large photos and videos, without providing much in the way of additional storage. The most common iPhones are still 16 & 32GB but the photos they now produce can be megabytes in size, with videos easily reaching 1GB.

What To Do When You Run Out Of Space
What To Do When You Run Out Of Space

It’s tempting to take that data and put it somewhere else, so either a laptop or external hard drive. Then once you’ve copied it all you delete it from the phone and gain back all that space. Problem solved.

Not So Fast…
If that copy on your laptop is now the only copy, then you could be one spilt coffee from disaster. If the laptop goes up in smoke, gets stolen, dropped or any of the myriad other ways of failing then it’s bye bye data.

The Fix

The key to making backups is redundancy. The key to making backups is redundancy. The key to making backups is redundancy.

You need to make extra copies of your data to different types of storage. This could be an external hard drive, NAS, USB Pen, SD card, anything. But don’t just pick one of those. Make a few backups. Put one in a locked safe somewhere. Send photos off to the cloud. Store a copy of your music at your nan’s house. If any of those copies gets lost or broken you can just replace it with another copy.

So let’s run through an example. All those photos on your iPhone have filled it up. Here’s what I would do:

  1. Copy the photos to my computer. Check them.
  2. Backup the computer as usual. (You’re already doing that, right?)
  3. Make another backup, or copy the photos to an online storage service like Dropbox.
  4. Now it is safe to delete the photos from the iPhone and revel in all that fresh space.

Note: Deleted photo recovery is virtually impossible for all modern iPhone versions due to encryption. 

Here’s another example for when your computer runs out of space instead:

  1. Is it possible to upgrade the internal storage? If it is then you should do that.
  2. If this is not possible, or too expensive then you will have to get creative. It will be more fiddly but copy all data to two external hard drives.
  3. You always want to avoid just leaving your data in one place. All electronic devices can (and will) fail, and they have a terrible habit of doing so at the worst possible moment.

So, just remember that no single copy of your files are safe. Making extra copies is cheaper and easier than waiting until something fails.

Thanks to Alexander Armstrong for inspiring this post.

Time Machine Must Create A New Backup For You

Time Machine must create new backup for you
Time Machine must create new backup for you

The other day I got an error message from Time Machine that it wanted to start from scratch with the backups on my Time Capsule. As I only use the Time Capsule as one part of my backup routine that was no big deal. I made sure my other non Time Capsule backups were up to date and then clicked Start New Backup. But what if the Time Capsule had been my only backup? I’d have been running without a backup for as long as Time Machine takes to backup my machine from scratch. Somehow I’ve trimmed my work laptop down to 78GB but it still estimated 6-14 hours until completion. (Yes, over wireless. No, I can’t relocate my office for the day.) What if my laptop died in that time? I’d have a dead laptop, a deleted Time Machine backup and an unfinished Time Machine Backup. In other words I’d be stuffed!

Now don’t get me wrong, I think that Time Machine has been brilliant in getting people making regular backups and I applaud the ease of use. The problem I have is that the language they use in their warnings is not strong enough. People should be more scared of losing their precious family photos or business files. There’s no need to sugar coat it. Also instead of deleting the old backup first, Time Machine should start a new backup and then delete the old one after the new one is verified. As long as there is enough free space of course. In my case there was plenty of free space.

I have included the original warning text below, along with my own version. I’ve not gone totally overboard, and sure it could use some work, but little details like this could really costs somebody some important data. It’s worth spending some time on.

Original Message Text

Transcript: Time Machine completed a verification of your backups on “Time Capsule”. To improve reliability, Time Machine must create a new backup for you.

Click Start New Backup to create a new backup. This will remove your existing backup history. This could take several hours.

Click Back Up Later to be reminded tomorrow. Time Machine won’t perform backups during this time.

My Version

Time Machine must create new backup for you. Fixed
Time Machine must create new backup for you. Fixed

Transcript: Time Machine has found a problem with your backups on “Time Capsule”. First backup any important files, then click Start New Backup. This will create a new backup and then delete the old one. This could take several hours and you will be at risk of data loss until it completes.

Click Back Up Later to be reminded tomorrow. Time Machine won’t perform backups during this time.

Notes

There are a few interesting things to point out here. As you are starting from scratch with the backup, you lose any old versions of files that Time Capsule had stored. It’s bad practice but some people will happily delete files from their computer, safe in the knowledge that Time Capsule has a copy. In that scenario, deleting the backup would be a disaster!

I still don’t know exactly what caused the problem with my Time Capsule backup, but since starting again I haven’t seen that message.