An important part of data recovery is staying ahead of new technology. It’s important that we are able to recover data from wherever people currently store their files. People once stored their most valuable data on a hard drive in their PC. Making hard drives is complicated and expensive, so only a handful of manufacturers had the resources to build them. The relatively small number of brands allowed us to became experts in the way those disks work & fail. Over time, some of those manufacturers merged or went bust. There are now just a few factories building disks. That’s not to say hard drive technology is standing still, it’s just a small enough target to keep an eye on. When Seagate release a new family of disks we scrabble around for a little while, and then ultimately find out how they work.
Enter the SSD
Solid State Drives have been on the horizon for a long time now. In fact, the three Macs within arms reach of me right now all have SSDs in them. The biggest problem for me, is that the barrier to entry to make SSDs is really low. There are thousands of factories in China alone with the capability to pump out millions of SSDs. Just buy some controllers and NAND chips, solder them to a circuit board — Instant SSD.
In 2018 there are only three hard drive manufacturers left. By comparison there are 35 SSD manufacturers listed — without including all the white-label, rebadged, refurbished, grey-market & clone drives in the market. If we estimate that each manufacturer produces three different product lines, you’ll get to 105 different types of SSD vs nine hard drive families. In fairness, I can think off the top of my head that some of these hard drive brands have more than three families of disk, but you get the idea. Staying on top of all those brands quickly goes from difficult to impossible.
The Future of Data Recovery
You’ve got to wonder if the future of data recovery will be choosing even smaller niches. Some companies may focus on recovering just one or two brands of SSD, and know that they at least have a chance of staying up to date with the latest technology.
SSDs present a number of new obstacles to data recovery. Some of these are not challenges as much as actual show-stopping dead ends.
Does your SSD controller use encryption, wear levelling & compression? The answer is yes for most SSDs. What happens to the data if that controller fails? In some cases this means the data is gone for good. In other cases it could mean weeks or months of manual work. You can’t just solder on a replacement controller as it won’t have the necessary encryption keys, nor will it have any idea of where the data has been stored across the multiple NAND chips.
In the previous part of this series, I explained why small files take longer to copy than large ones. This can make for unpredictable results when you copy a mixed batch of files. I decided to track the time estimates for such a copy and see how wrong the time estimates really are.
Let’s kick off with another animation. I took a series of screenshots when copying files to another disk. I then tracked the estimated time to complete vs the actual completion time. I have also included the raw numbers at the bottom of the page.
How wrong was the original estimate?
The original time estimate was off by almost two hours! 1:59 to be precise. After 9 minutes, the estimate was off by an hour. After around 50 minutes copying, the estimate was only off by 1 minute.
We can’t really use this data to extrapolate much, as it is specific to the data being copied. You’ll notice that the data toward the end of the copy was mostly media files. What the data does show quite clearly is that you cannot trust the time estimates.
Large media files – Fast copy, Accurate time estimate
Small office / text files – Slow copy, Accurate time estimate
Mixed data – Mixed copy time, Inaccurate time estimate
Have you ever noticed that it takes longer to copy 200MB of small text files than it does to copy a 200MB video file. This can seem a bit strange, but there is a simple reason for it.
Every time you copy a file, the system must also copy over some metadata. Things like the filename, creation date, modification date, filesize etc. When you copy a large file like a video, this information is copied once, and then all the data blocks are copied into place. With tiny text files, new metadata needs to be transferred for each and every file.
In real-world examples, I’ve seen disks capabale of copying at 100MB/s get as low as 1MB/s when copying small files.
This is easier to see in simple animations. In the images below, the metadata is represented by the blue container and red block. The data blocks are shown in green.
This behaviour also leads to one of the most annoying things about copying files. The crazy time-estimates! If you copy a mix of large and small files, the computer can’t figure out how long the copy will take, so just adjusts the estimate as it goes along.
This post assumes you’re copying the files in ideal conditions. In the real world you also need to account for slow networks, slow connections, failing disks and countless other things that can slow down a transfer.
This is the first post in a mini-series about copying files. More coming soon.
Apple’s PCIe SSDs have always proven challenging for data recovery. We have a good success rate with them, however many of our tools don’t even support working at PCI level. This particular SSD provided a new challenge though.
The SSD we received was initially showing up correctly in the system. Upon access, the disk would read a few bytes and then stop reading. The SSD would remain visible to the system but not respond to any further commands. The only way to bring it back was to cycle power. Yep, the IT classic — Turn it off and on again! The problem is, PCI cards don’t come with on-off switches so the solution needed to work within software.
First I thought a copy task with a scheduled reboot may get the device back online, however I quickly found out PCI slots stay powered on during a reboot 🤦♀️ This means once the device gets stuck, a reboot won’t bring it back on.
My second idea was to schedule shutdowns but the time taken to boot and then restart the copy process seemed like a nightmare.
In the end I found a series of commands which could keep a fairly steady copy process going. The main idea was:
Start a copy task
Pause on error
Toggle the PCI connection using low-level commands
Resume the copy task
Rinse & repeat
Although not the most elegant solution, this process cycled through 647 times, taking an average of 760MB per cycle. In less than 24 hours the whole 512GB device was cloned to another disk ready for recovery.
After this script finished I did find a slightly cleaner way to reset the PCI slot which will result in much faster recoveries in the future.
I was unable to find any mention of this type of recovery online, so if anyone else knows about it, they’re keeping it to themselves. If you know anyone with a failed Mac SSD, get in touch. This is just one of the many solutions we have for recovering them.
Over the past decade, hard drive companies have been endlessly bought-out and then re-sold. At this point I’ve pretty much lost track of who manufactures which brands now. Since all this restructuring, it’s quite common to see portable Seagate branded drives with Samsung disks inside & vice versa. There are also Maxtor branded versions of those same drives in some markets.
So what’s wrong with this disk?
Here’s a list of some of the problems with this disk:
Unfinished labelling (the white edges are usually peeled off)
Mismatched serial numbers
Wrong PCB for a Seagate / Samsung disk
Misspelled Regulatory as Reaularory (see image below)
I’ve never seen a disk quite like this. It’s from a Samsung external case with a Samsung logo on the front label. It also uses a Seagate model number ST1000LM024. Normal enough so far, however the label shows one serial number while the label opposite the SATA connector shows a different one. Also a third different serial number is reported electronically to the system when the disk is attached.
The label at the end of the disk is actually a clue to the true identity of this disk. It features the familiar Hitachi / IBM style with two separate stickers & barcodes. The disk is actually a Hitachi HTS5432L9 which suggests a much older 320GB disk that was likely destined for the scrapheap in a former life. Funnily enough, these Hitachi disks had their own strange history of mislabelling.
I originally thought this disk may have been a white-label or grey market disk. Some disks get refurbished and are then sold under different brand names in other markets. After a bit more investigation I think it may actually be more sinister than that. This is more like a fake or fraudulent disk, designed to dupe somebody into thinking it is a larger disk than it really is. It appears to the computer as 1TB however only contains 320GB of usable space. This is very similar to the fake flash drives we’ve seen before. The problem with fake capacity disks is that when you exceed the genuine size, the rest of the data usually becomes inaccessible. Also depending on how the disk handles the problem, it could damage the existing data when it fails.
Fortunately for the owner of this disk, they had not yet used up 320GB of the disk. In fact this disk failed when the USB connector fell off. Maybe another sign of the poor build-quality of this fake. Once we figured out what we were working with we were able to recover all data. It took a combination of Hitachi firmware repair, careful imaging, and then exFAT reconstruction.
Urgent Warning: Fusion Drive always consists of two separate disks. If you want your data back you must get both parts. We’ve heard a number of reports that users with failed Fusion Drives are only given the Hard Disk back when receiving Apple repairs. On its own, the hard drive is not enough to recover all data in original condition. This is especially true if FileVault encryption is used.
What is Fusion Drive?
Fusion Drive is Apple’s version of a hybrid solid state & mechanical disk. It combines a small fast SSD with a large slow hard drive to achieve a balance between cost & performance. Frequently used files are moved to the SSD, and old stale data is sent to the slow hard drive. This is all taken care of automatically behind the scenes. Unless you dig into the terminal, you wouldn’t even know you had two separate disks inside the Mac. Fusion Drive is part of Apple’s Core Storage system. It is somewhat similar to Linux LVM as a volume management system.
What Fusion Drive is not
Fusion Drive does not use the SSD as a cache for files but actually moves data from one disk to the other. This is important, as both disks are required for full recovery.
Why does Fusion Drive exist?
At launch, and even now, the cost for large capacity SSDs is way higher than the cost of an equivalent hard drive. The problem is that SSDs offer huge benefits to the user experience. When you use an SSD, you hardly ever have to wait for things to load. The computer boots up within seconds.
Hybrid drives aim to bridge the gap between solid state and mechanical disks. An iMac with a 3TB Fusion Drive comes with some of the benefits of SSDs, but much less cost. As the cost of SSDs fall, the need for Fusion Drive will eventually disappear. Apple has shown with their current lineup that they’d much rather go all-SSD where possible. Current iMac Pro & MacBook Pro both use 100% SSD internal storage.
We’ve had two recent cases where a user has brought a “Fusion Drive” to us for recovery, but actually only had the hard drive part. Apple had given the damaged hard drive back after replacement, but reused the SSD when creating a new Fusion Drive. This user only had a few GB of data so the Hard Drive hadn’t even been used yet. All the data was stored on the SSD which was now overwritten.
The majority of Fusion Drives we’ve seen have a Seagate ST3000DM001 3TB hard drive combined with a 128GB blade SSD.
I published a post back in 2016 highlighting the problem we were seeing with Sandforce controllers on single SSD drives. We have now also come across this PCIe 3.2TB SSD that uses a Marvel controller as well as Sandforce controllers to control banks of data chips. Like the others, this device has I/O errors which we believe is caused by the Sandforce controllers. This PCIe SSD device requires proprietary drivers for it to be recognised in the bios and OS, regardless of what platform it is run on. Although we can communicate with the device via an SSD Utility provided by Toshiba, there is no access to the user data.
Seagate has another batch of dubious drives in circulation at the moment. These slim disks are often used in external Ultra Slim Portable enclosures, but also appear in laptops. Visually these disks have a new-look design that seems a bit strange at first glance. For starters, instead of a solid metal top cover, these disks only have a partial top lid, sealed by nothing more than the printed label. If you decided to peel the sticker from one of these, you’d be unsealing the top cover and allow dust to fall straight onto the disks. This is BAD.
Aside from the visual differences, these disks also feature a number of new firmware changes that are barriers to recovery. The industry standard recovery tools only have limited support at this stage but the good news is that we can already recover from most common issues.
D.S.A.A (Dead Shortly After Arrival)
Another fun feature of these disks is the way that some of them just fail after a couple of weeks light use. As always, we only see faulty disks here, but it’s always a surprise to see a disk that’s only a few weeks or months old on our desks.
Something we’ve seen more than once is a customer that buys an external drive, copies a load of data onto it, wipes their computer, and then finds when trying to load the data back on that the backup disk has failed. Always remember to take two separate backups when erasing your main disk!
Another annoying trait with these disks is that they are manufactured from lots of different internal parts. This makes it a nightmare to locate suitably matched heads when we need to replace them.
We’ve seen a few cases recently where a user has unexpected lost all data, and been left with a disk named OS X Base System. Inside the disk is a file named Install macOS Sierra or Install macOS High Sierra, and a few other system files.
The OS X Base System is usually part of a macOS installer or update, so it’s unclear how disks ended up getting replaced with this.
If this has happened to you, we’d love to hear more about how.
We can usually recover these disks, but it’s really important that you stop using the disk as soon as possible. If you install anything to the disk you could risk losing all the data permanently.
We’ve still got more of these to investigate so we’ll update the post when we’ve learned more.
Although most of these free services seemed harmless at first, we now live in a different time. Now,if we let them see your IP Address, and which page you are on, they can combine that with their vast pools of other data to target ads at you, and build profiles about your online behaviour and preferences. If you’ve ever seen an advert for a product you were recently researching that follows you around the internet for days, you’ll know what I mean.
You may wonder why anyone ever allowed such tracking, but these services crept up on us. Google Analytics genuinely helped website owners to easily see which pages were working well. We could use the information to make changes and see how they performed. Share buttons allowed an easy way to get content into valuable social networks. These things eventually felt normal and necessary, and were not really given a second though. Now, with advances in machine learning and AI, any crumb of information we give them can be processed with others into something much more potent.
Large (free) web services have proven that they don’t respect user privacy, so we’ve totally cut them off. We can’t stop them doing dubious things, but we can stop giving them our data. Hopefully as more companies implement GDPR, we’ll see a trend away from the tracking-by-default we see from the likes of Facebook & Google. Did you know for example that many of the “Like on Facebook” type buttons that appear websites, often leak information back to the other site even if you don’t click the button!
Although it sounds like we’ve just thrown away a bunch of useful services, we’ve actually made a few gains. Our page-load speeds should be a bit faster without the third-party scripts. We also found a replacement anti-spam tool that runs directly on our site, and doesn’t send any information to a third party service.