An important part of data recovery is staying ahead of new technology. It’s important that we are able to recover data from wherever people currently store their files. People once stored their most valuable data on a hard drive in their PC. Making hard drives is complicated and expensive, so only a handful of manufacturers had the resources to build them. The relatively small number of brands allowed us to became experts in the way those disks work & fail. Over time, some of those manufacturers merged or went bust. There are now just a few factories building disks. That’s not to say hard drive technology is standing still, it’s just a small enough target to keep an eye on. When Seagate release a new family of disks we scrabble around for a little while, and then ultimately find out how they work.
Enter the SSD
Solid State Drives have been on the horizon for a long time now. In fact, the three Macs within arms reach of me right now all have SSDs in them. The biggest problem for me, is that the barrier to entry to make SSDs is really low. There are thousands of factories in China alone with the capability to pump out millions of SSDs. Just buy some controllers and NAND chips, solder them to a circuit board — Instant SSD.
In 2018 there are only three hard drive manufacturers left. By comparison there are 35 SSD manufacturers listed — without including all the white-label, rebadged, refurbished, grey-market & clone drives in the market. If we estimate that each manufacturer produces three different product lines, you’ll get to 105 different types of SSD vs nine hard drive families. In fairness, I can think off the top of my head that some of these hard drive brands have more than three families of disk, but you get the idea. Staying on top of all those brands quickly goes from difficult to impossible.
The Future of Data Recovery
You’ve got to wonder if the future of data recovery will be choosing even smaller niches. Some companies may focus on recovering just one or two brands of SSD, and know that they at least have a chance of staying up to date with the latest technology.
SSDs present a number of new obstacles to data recovery. Some of these are not challenges as much as actual show-stopping dead ends.
Does your SSD controller use encryption, wear levelling & compression? The answer is yes for most SSDs. What happens to the data if that controller fails? In some cases this means the data is gone for good. In other cases it could mean weeks or months of manual work. You can’t just solder on a replacement controller as it won’t have the necessary encryption keys, nor will it have any idea of where the data has been stored across the multiple NAND chips.
In the previous part of this series, I explained why small files take longer to copy than large ones. This can make for unpredictable results when you copy a mixed batch of files. I decided to track the time estimates for such a copy and see how wrong the time estimates really are.
Let’s kick off with another animation. I took a series of screenshots when copying files to another disk. I then tracked the estimated time to complete vs the actual completion time. I have also included the raw numbers at the bottom of the page.
How wrong was the original estimate?
The original time estimate was off by almost two hours! 1:59 to be precise. After 9 minutes, the estimate was off by an hour. After around 50 minutes copying, the estimate was only off by 1 minute.
We can’t really use this data to extrapolate much, as it is specific to the data being copied. You’ll notice that the data toward the end of the copy was mostly media files. What the data does show quite clearly is that you cannot trust the time estimates.
Large media files – Fast copy, Accurate time estimate
Small office / text files – Slow copy, Accurate time estimate
Mixed data – Mixed copy time, Inaccurate time estimate
Over the past decade, hard drive companies have been endlessly bought-out and then re-sold. At this point I’ve pretty much lost track of who manufactures which brands now. Since all this restructuring, it’s quite common to see portable Seagate branded drives with Samsung disks inside & vice versa. There are also Maxtor branded versions of those same drives in some markets.
So what’s wrong with this disk?
Here’s a list of some of the problems with this disk:
Unfinished labelling (the white edges are usually peeled off)
Mismatched serial numbers
Wrong PCB for a Seagate / Samsung disk
Misspelled Regulatory as Reaularory (see image below)
I’ve never seen a disk quite like this. It’s from a Samsung external case with a Samsung logo on the front label. It also uses a Seagate model number ST1000LM024. Normal enough so far, however the label shows one serial number while the label opposite the SATA connector shows a different one. Also a third different serial number is reported electronically to the system when the disk is attached.
The label at the end of the disk is actually a clue to the true identity of this disk. It features the familiar Hitachi / IBM style with two separate stickers & barcodes. The disk is actually a Hitachi HTS5432L9 which suggests a much older 320GB disk that was likely destined for the scrapheap in a former life. Funnily enough, these Hitachi disks had their own strange history of mislabelling.
I originally thought this disk may have been a white-label or grey market disk. Some disks get refurbished and are then sold under different brand names in other markets. After a bit more investigation I think it may actually be more sinister than that. This is more like a fake or fraudulent disk, designed to dupe somebody into thinking it is a larger disk than it really is. It appears to the computer as 1TB however only contains 320GB of usable space. This is very similar to the fake flash drives we’ve seen before. The problem with fake capacity disks is that when you exceed the genuine size, the rest of the data usually becomes inaccessible. Also depending on how the disk handles the problem, it could damage the existing data when it fails.
Fortunately for the owner of this disk, they had not yet used up 320GB of the disk. In fact this disk failed when the USB connector fell off. Maybe another sign of the poor build-quality of this fake. Once we figured out what we were working with we were able to recover all data. It took a combination of Hitachi firmware repair, careful imaging, and then exFAT reconstruction.
Urgent Warning: Fusion Drive always consists of two separate disks. If you want your data back you must get both parts. We’ve heard a number of reports that users with failed Fusion Drives are only given the Hard Disk back when receiving Apple repairs. On its own, the hard drive is not enough to recover all data in original condition. This is especially true if FileVault encryption is used.
What is Fusion Drive?
Fusion Drive is Apple’s version of a hybrid solid state & mechanical disk. It combines a small fast SSD with a large slow hard drive to achieve a balance between cost & performance. Frequently used files are moved to the SSD, and old stale data is sent to the slow hard drive. This is all taken care of automatically behind the scenes. Unless you dig into the terminal, you wouldn’t even know you had two separate disks inside the Mac. Fusion Drive is part of Apple’s Core Storage system. It is somewhat similar to Linux LVM as a volume management system.
What Fusion Drive is not
Fusion Drive does not use the SSD as a cache for files but actually moves data from one disk to the other. This is important, as both disks are required for full recovery.
Why does Fusion Drive exist?
At launch, and even now, the cost for large capacity SSDs is way higher than the cost of an equivalent hard drive. The problem is that SSDs offer huge benefits to the user experience. When you use an SSD, you hardly ever have to wait for things to load. The computer boots up within seconds.
Hybrid drives aim to bridge the gap between solid state and mechanical disks. An iMac with a 3TB Fusion Drive comes with some of the benefits of SSDs, but much less cost. As the cost of SSDs fall, the need for Fusion Drive will eventually disappear. Apple has shown with their current lineup that they’d much rather go all-SSD where possible. Current iMac Pro & MacBook Pro both use 100% SSD internal storage.
We’ve had two recent cases where a user has brought a “Fusion Drive” to us for recovery, but actually only had the hard drive part. Apple had given the damaged hard drive back after replacement, but reused the SSD when creating a new Fusion Drive. This user only had a few GB of data so the Hard Drive hadn’t even been used yet. All the data was stored on the SSD which was now overwritten.
The majority of Fusion Drives we’ve seen have a Seagate ST3000DM001 3TB hard drive combined with a 128GB blade SSD.
I published a post back in 2016 highlighting the problem we were seeing with Sandforce controllers on single SSD drives. We have now also come across this PCIe 3.2TB SSD that uses a Marvel controller as well as Sandforce controllers to control banks of data chips. Like the others, this device has I/O errors which we believe is caused by the Sandforce controllers. This PCIe SSD device requires proprietary drivers for it to be recognised in the bios and OS, regardless of what platform it is run on. Although we can communicate with the device via an SSD Utility provided by Toshiba, there is no access to the user data.
Seagate has another batch of dubious drives in circulation at the moment. These slim disks are often used in external Ultra Slim Portable enclosures, but also appear in laptops. Visually these disks have a new-look design that seems a bit strange at first glance. For starters, instead of a solid metal top cover, these disks only have a partial top lid, sealed by nothing more than the printed label. If you decided to peel the sticker from one of these, you’d be unsealing the top cover and allow dust to fall straight onto the disks. This is BAD.
Aside from the visual differences, these disks also feature a number of new firmware changes that are barriers to recovery. The industry standard recovery tools only have limited support at this stage but the good news is that we can already recover from most common issues.
D.S.A.A (Dead Shortly After Arrival)
Another fun feature of these disks is the way that some of them just fail after a couple of weeks light use. As always, we only see faulty disks here, but it’s always a surprise to see a disk that’s only a few weeks or months old on our desks.
Something we’ve seen more than once is a customer that buys an external drive, copies a load of data onto it, wipes their computer, and then finds when trying to load the data back on that the backup disk has failed. Always remember to take two separate backups when erasing your main disk!
Another annoying trait with these disks is that they are manufactured from lots of different internal parts. This makes it a nightmare to locate suitably matched heads when we need to replace them.
I have good news for our customers, our developers have been working hard and have produced a solution to the firmware failures of this model hard drive. I posted a blog alerting our customers back in March 2015 when we started to see the problems arise. A solution was tested just before the end of last year and we now have a process that fixes these issues.
More than enough has been written about the Spectre & Meltdown CPU vulnerabilities so I won’t go over them directly here. If you’d like more info, I’ve included some links at the end.
My interest is more with the trouble caused by the intended fixes. Despite CPU manufacturers testing the fixes for some months, in certain circumstances they are causing random reboots, and other performance problems.
To “Safely Remove” a disk is standard practice to avoid filesystem corruption. In simple terms, the computer stops writing to the disk and then you’re safe to unplug it. In the case of a random reboot, the computer doesn’t get the chance to finish with the disk. A crash at a critical moment can damage the filesystem and render the machine un-bootable. Even if the system can recover itself, if you crash enough times your chances of trouble increase. It’s hard to trust a system that could randomly reboot at any moment.
Until we get fixes from the manufacturers, there’s not much else we can do. Now would be a really good time to make sure your backups are working and up-to-date. These fixes could take a while to stabilise, and in the meantime you could be one random reboot away from data loss.
If you need to add or change your Time Machine™️ backup drive, the process is pretty simple. Plug in a new disk and make sure there is nothing on it that you need. In most cases Time Machine will ERASE the disk before using it! You have been warned.
Now might be a good time to find the disk on your desktop and rename it to something obvious like “backups” for example.
Next, open Time Machine Preferences from the Time Machine menu. If you don’t have the Time Machine icon near the clock, you can also find the settings within “System Preferences”.
Click the “Use Disk” button, and Time Machine will start making a new full backup to this new disk. This may take a few hours.
Despite falling out of favour in giant tech companies, there are not many businesses in the UK that don’t use some sort of SQL database. Whether that’s a website running WordPress & MySQL or an intranet hooked up to an MSSQL DB, SQL is still everywhere. Even iPhones use SQLite to store Notes and other data inside apps.
On Disk Problems
Sometimes the fault lies with the underlying disk, Virtual Machine, or RAID array that the file is stored on. First we solve the disk problem and then extract a working copy of the database.
Another issue with database files is that they can go corrupt silently, and you might not notice until hours, weeks, or months later when a crucial process fails.
Our SQL recovery tools, alongside our hard disk & RAID recovery services can help get your lost databases up and running again with minimal fuss & downtime.
Peter MorganThese guys are amazing!! Really helped us out in an impossible situation with a corrupted SQL database which even the software provider couldn’t fix. Not only did they fix the corruption but they did it quickly and super professionally keeping us informed at every step of the way. I have used Dataquest many times and over the years and they always deliver.4th July, 2017