In the previous part of this series, I explained why small files take longer to copy than large ones. This can make for unpredictable results when you copy a mixed batch of files. I decided to track the time estimates for such a copy and see how wrong the time estimates really are.
Let’s kick off with another animation. I took a series of screenshots when copying files to another disk. I then tracked the estimated time to complete vs the actual completion time. I have also included the raw numbers at the bottom of the page.
How wrong was the original estimate?
The original time estimate was off by almost two hours! 1:59 to be precise. After 9 minutes, the estimate was off by an hour. After around 50 minutes copying, the estimate was only off by 1 minute.
We can’t really use this data to extrapolate much, as it is specific to the data being copied. You’ll notice that the data toward the end of the copy was mostly media files. What the data does show quite clearly is that you cannot trust the time estimates.
Large media files – Fast copy, Accurate time estimate
Small office / text files – Slow copy, Accurate time estimate
Mixed data – Mixed copy time, Inaccurate time estimate
Have you ever noticed that it takes longer to copy 200MB of small text files than it does to copy a 200MB video file. This can seem a bit strange, but there is a simple reason for it.
Every time you copy a file, the system must also copy over some metadata. Things like the filename, creation date, modification date, filesize etc. When you copy a large file like a video, this information is copied once, and then all the data blocks are copied into place. With tiny text files, new metadata needs to be transferred for each and every file.
In real-world examples, I’ve seen disks capabale of copying at 100MB/s get as low as 1MB/s when copying small files.
This is easier to see in simple animations. In the images below, the metadata is represented by the blue container and red block. The data blocks are shown in green.
This behaviour also leads to one of the most annoying things about copying files. The crazy time-estimates! If you copy a mix of large and small files, the computer can’t figure out how long the copy will take, so just adjusts the estimate as it goes along.
This post assumes you’re copying the files in ideal conditions. In the real world you also need to account for slow networks, slow connections, failing disks and countless other things that can slow down a transfer.
This is the first post in a mini-series about copying files. More coming soon.
I have a particularly nerdy Apple rabbit hole to share with you today involving the labels on Apple hard drives. At some point around 2007-2008 Apple started re-labelling the hard drives used in their computers. I’m pretty sure the hard drives in black & white MacBooks were standard white labels with an Apple part number (655-XXXX) printed on them. Now disks feature a black label with white text. Not particularly exciting, but maybe somebody (Jony?) wanted the disks to match the fancy black circuit boards now used in all Apple hardware. That’s the sort of attention to detail we’ve come to love from Apple devices. Whatever the reason, this has resulted in a batch of Toshiba drives in circulation with incorrect information printed on them. I assume these are the result of a simple ⌘+C, ⌘+V error. My speculation is that after printing the labels for the 320GB disk nobody remembered to change the text for the 160GB version.
For reference MK3253GSX is 320GB and MK1653GSX is 160GB. In Apple’s world, both disks use MK3253GSX on the label, even though the correct number is shown when you check Disk Utility or About This Mac > System Information.
This wonky number business all came to a head ? when we were looking for replacement parts for one of these disks. We needed the double headed 160GB drive, not the four headed 320GB.
When I was trying to research this, I was quite suprised to find an old blog post of ours that shows the same problem with Hitachi drives manufactured around the same time. I’ve not found any other mention of this labelling fault, so thought I’d post it up here for future Mac Archaeologists to find.
Despite some amount of publicity, the problem of fake capacity flash drives being sold online has still not gone away. Recently we did a bit of consumer investigation and bought a few to test. Surely after two or three years eBay will have sorted this problem out?
What Are Fake Capacity Flash Drives?
If you’ve not heard of these before, a fake capacity flash drive is simply a small low capacity flash drive pretending to have (much) more storage. The clever / devious part is that they may have only 16GB of real storage, but can appear to the computer as anything up to 2TB. You will be able to copy data to the device, but only the first 16GB may be readable later. Some devices constantly overwrite the same 16GB while other just dump the rest of the data into a back hole.
How Do I Know If My USB Drive is Fake?
There are not any strict rules, but you can start to build up a picture as you gather more information.
Price – These flash drives will be WAY cheaper than anything you can buy in a retail store. We just bought 512GB flash drives for £10 and 1TB drives for £15. Currently a 64GB SD card is £32.48 at Novatech, so you can see how expensive real flash drives should be.
Quality – Large capacity USB drives are relatively expensive. You’d expect them to be well made and probably well packaged. The fake drives usually come unbranded, in generic clear bags.
Wording – Since these problems have been reported, some sellers are putting some disclaimers on the listings to suggest you should only use the devices for small amounts of data etc. This is obviously nonsense. If you buy a real 1TB of storage, you can use all 1TB.
Testing – Although most people won’t be familiar with testing hardware, there are some pretty simple tools to test USB drives. They write patterns of data to the chosen device and then read it back again to make sure it was written correctly. USB Test Tool (It’s German, but also runs in English)
What To Do If You Have A Fake Capacity Flash Drive
Most online marketplaces like eBay have pretty robust buyer protection to allow you to claim a refund. The advice for making claims varies with each website so you may need to hunt around, or contact the site directly. Although eBay are more than happy to organise refunds, they show little interest in stopping the sale of these devices. We’ve spoken to eBay customer services a number of times and they said that their system will flag up if enough people make returns.
You Often Don’t Know Until It’s Too Late
It’s a bad idea to keep using one of these USBs. If you detect a fake capacity, you may think it’s OK to just use it for small amounts of data but it’s not a good idea. The problem is that as you use and delete files, you could gradually start edging towards the limit of the storage. The next file you write could just go into the black hole, and the data is lost. In fact, many people don’t realise their flash drive is a fake until they exceed the “genuine” part of the storage. If you were copying photographs to the USB drive, you might not notice that some of them are missing until you try to read them again much later.
Not Just Memory Sticks
These same types of fake flash drives also appear as SD cards so we’ve heard from people that were at weddings or on their honeymoon and lost all or most of the photos.
Update 31-10-2016: Since posting this, we were made aware of this report from the USA about eBay’s problem with fake flash storage.
This OWC external enclosure is a common sight on the desks of Mac users with big storage needs. It’s a pretty standard 4-bay box, styled somewhat like a cousin of a PowerMac G5 or 1st generation Mac Pro. Inside are the usual options of RAID 0 to RAID 5 with a few additions like JBOD & RAID 10 thrown in for good measure. There are a few variations of this device but the back panels commonly have USB, Firewire, and eSATA ports for direct connection to a PC or Mac. There is no ethernet port on these drives which makes the Qx2 a DAS (Direct Attached Storage) rather than NAS (Network Attached Storage).
Aside from massive name, the OWC Mercury Elite Pro Qx2 also comes with a potentially huge amount of storage. Currently up to 32TB on the OWC store, but also available diskless or BYOD (Bring your own disks). With so much storage space, these drives often become the one and only repository for vast lumps of important data. The benefits of RAID give a false sense of security that the data is safe from drive failures. Unfortunately, there are a number of reasons why the RAID array alone will not protect from certain failures. Most of these failures can be overcome by us in our workshop, but they are not one-button fixes. It is helpful to understand why a seemingly rock-solid platform can be even more risky than a simple external USB drive.
Under common settings, the Qx2 will use RAID 5 for the array. With four 2TB drives this gives you a 6TB volume on a Mac or ~5.5TB on a PC, and can cope with a single disk failure. There is a lot of debate about how good RAID 5 really is for such large drives. In our example this means that if a single disk fails, it will need to be replaced, and then the new disk rebuilt with 2TB of data calculated from the other disks. This will take many hours, even under optimal conditions, but if anything goes wrong before it completes the array could stop showing up all together. At this stage, the data is probably recoverable but don’t panic. One wrong move and the data could be gone for good.
If the data is crucial then get assistance from a RAID recovery service now and you should get back all or most of the data.
If any disks are removed or replaced at this point the array could get reinitialised and either make the recovery more complicated or wipe the data completely.
Aside from all the problems with a RAID setup, the volume could also fail in the same ways that a standard hard drive can. There could be deleted files, a reformatted or corrupt partition, or even the RAID controller failure. RAID cannot protect against those types of failure at all.
Our first step would be making read-only copies of each disk in the array. This protects against further disks failing, and also allows us to work from copies without risking the original disks. In fact, once the disks are copied, we put the originals to one side and don’t touch them again until all the data is recovered and supplied back to the user.
Once we have our copies, they are loaded into our own hardware where we recreate the RAID in a virtual environment. Again, we don’t use the original hardware, as that may have been the root cause of the problem.
When the virtual RAID has been loaded and all the data extracted, the files are supplied back on whatever alternative storage is suitable, (not the original device!) Once the data has been delivered to the user, and backups made, the old unit can then be destroyed, or returned and reused.
Anyone using RAID on a regular basis should know that RAID is not a replacement for backups. If anything, the increased number of disks makes failure more likely. This needs to be addressed by either making backups to another device, or an online service (preferably both). You ideally want backups that keep historic versions of the files, so that inadvertently deleting a file or changing a file by mistake will not also replace the backup version.
If you are having problems with an OWC Mercury Elite Pro Qx2, give us a call or send a message via the form on this page. We give free advice and could help you avoid permanent data loss.
1. Macs now use 1000 bytes for 1KB but PCs use 1024 bytes.
2. Even RAID 6 does not solve the long time required to rebuild a disk, even though it allows for two disk failures.
It’s a common problem that as we generate more data each year we start running out of space to put it. This is now even more of an issue in the smartphone market, where built-in cameras are generating increasingly large photos and videos, without providing much in the way of additional storage. The most common iPhones are still 16 & 32GB but the photos they now produce can be megabytes in size, with videos easily reaching 1GB.
It’s tempting to take that data and put it somewhere else, so either a laptop or external hard drive. Then once you’ve copied it all you delete it from the phone and gain back all that space. Problem solved.
Not So Fast…
If that copy on your laptop is now the only copy, then you could be one spilt coffee from disaster. If the laptop goes up in smoke, gets stolen, dropped or any of the myriad other ways of failing then it’s bye bye data.
The key to making backups is redundancy. The key to making backups is redundancy. The key to making backups is redundancy.
You need to make extra copies of your data to different types of storage. This could be an external hard drive, NAS, USB Pen, SD card, anything. But don’t just pick one of those. Make a few backups. Put one in a locked safe somewhere. Send photos off to the cloud. Store a copy of your music at your nan’s house. If any of those copies gets lost or broken you can just replace it with another copy.
So let’s run through an example. All those photos on your iPhone have filled it up. Here’s what I would do:
Copy the photos to my computer. Check them.
Backup the computer as usual. (You’re already doing that, right?)
Make another backup, or copy the photos to an online storage service like Dropbox.
Now it is safe to delete the photos from the iPhone and revel in all that fresh space.
Note: Deleted photo recovery is virtually impossible for all modern iPhone versions due to encryption.
Here’s another example for when your computer runs out of space instead:
Is it possible to upgrade the internal storage? If it is then you should do that.
If this is not possible, or too expensive then you will have to get creative. It will be more fiddly but copy all data to two external hard drives.
You always want to avoid just leaving your data in one place. All electronic devices can (and will) fail, and they have a terrible habit of doing so at the worst possible moment.
So, just remember that no single copy of your files are safe. Making extra copies is cheaper and easier than waiting until something fails.
When we recover iPhones, although we can usually provide the data in a computer-readable format, it is often easier to just load the data back onto another iPhone. Fortunately, as long as you restore the data in the correct order, you should get your iPhone up and running with your restored data.
Before you attach the iPhone you need to make sure iTunes is ready for it. You need to put the recovered iPhone backup into the correct place for iTunes to find. Quit iTunes first.
The iPhone backups go into:
C:\Documents and Settings\username\Application Data\Apple Computer\Mobile Sync\Backup
Now copy your iPhone backup folder from the recovered data to the backup folder on the computer. If you’ve ever made any iPhone backups they will be in this folder with long unreadable names of random numbers and letters. It doesn’t matter that you can’t read it, iTunes can. Don’t rename it!
So now your /MobileSync/Backup folder should look something like:
Remember that the string of numbers will not be the same.
Next you can launch iTunes and see if the iPhone backup is seen in the list. Within iTunes select Preferences and then the Devices tab (Image Below). If your backup file has been recognised then you should see the iPhone name in the list and the date of the backup. If not, go back and check that everything is in the right place.
Next, it’s a good idea to sign in to the iTunes Store. If you’ve not used this computer with this iPhone before you may also need to authorise the computer to access your purchased apps and media. Once signed-in to the store you should choose the Store menu and select “Authorise This Computer…” Be aware that you can only authorise a set number of computers. (The limit can be reset if needed.)
Now you can import the Music and apps into iTunes. Don’t worry if some of the apps don’t have icons yet.
Once all that is finished you are ready to connect the new iPhone. You will need to follow some on-screen prompts and settings, and then it will ask if you want to set it up as a new device, or restore it from a backup. This is where all that hard work pays off. Choose the backup, and let iTunes work its magic. It can take a while if you have lots to restore but when it’s finished your phone should boot into a familiar screen. Your Contacts, Messages, Calendars, Notes and lots of other data will be back in their respective apps.
If none of the other apps got transferred you can choose the Apps tab within iTunes and tick “Sync Apps.” You can then tick whichever apps you want to send to the iPhone.
Like most external hard drives, Buffalo external drives are simply a wrapper around a regular hard drive. Aside from the protective shell they also have some electronic parts to convert between the internal hard drive and the external USB, Firewire, eSATA or Thunderbolt connections.
If you have problems with an external drive, you can perform a relatively simple test to check where the fault lies. Be aware that opening the external drive case will probably void your warranty, and if there is crucial data on the drive you should seek professional data recovery. That’s the obligatory warning out the way, so lets have a look at some troubleshooting.
First check all cables are plugged in securely, and not damaged or frayed near the ends.. If you have an identical drive with spare cables try them, but make sure you don’t plug in a power supply with different voltage! Hard drives don’t handle extra voltage well so you’ll end up in a worse position than you started.
If you know how, you could remove the hard drive from the external case and attach it directly to a PC to see if that allows access to the data. If it does, you should copy the data off straight away. The drive could still be faulty & fail again soon.
Whatever you do, don’t dismantle the actual hard drive. Hard drives are built in controlled clean-air environments and even the smallest spec of dust can cause permanent damage to the drive.
Since the introduction of unique ROM chips on the hard drives, it is often no longer possible to exchange circuit boards with another hard drive to access the data. In our experience circuit board problems are far less common than they used to be.
A NAS (Network Attached Storage) puts storage onto your network, where it can be accessed by many computers. They often have more than one hard drive which can allow you to have automated copies of your data, which is known as RAID. A common type of RAID found on NAS devices is RAID 1, which will make two hard drives into a mirror copy of one another. Some manufacturers call RAID 1 Safe mode. If you have a NAS with two 1TB hard drives and set them to RAID 1 mirroring, instead of 2TB of storage (1TB x2) you only get 1TB. Everything you store to the NAS gets saved to both drives automatically. The theory is that if one of the drives fail, you can access all of the data from the other one. In practice that is not always the case. More on that later.
RAID 0 (Stripe)
Another common NAS option is RAID 0. The “R” in RAID stands for redundant, however there is no redundancy in RAID 0, so it’s not a real RAID type. If you setup the same two 1TB disks as RAID 0, you will get a 2TB volume to store your data on. The problem is that every single file you write to the NAS will be split into tiny pieces and distributed across both drives. If one drive fails, you not only lose the data from that failed drive, but also from the non-failed drive as it only contains half the pieces of each file. RAID 0 should never be used for long term storage, but can be fast so is often used for video editing.
So that’s the hardware taken care of. What other things should you look out for when choosing a NAS?
Another problem with most NAS devices is the non-standard filesystems they use to store the data on the disks. If the NAS itself fails, you cannot usually read the disks by attaching them to a standard PC. So even in RAID 1 mirror mode, you could end up with no usable copies of your data. Most NAS drives run a simplified version of Linux, but only some of them use standard Linux filesystems like ext2/3/4.
Backup my backup?
Some NAS drives have a USB port to allow you to backup the data to an external hard drive. This is great, as long as you can access the backup data on a regular PC, and it doesn’t need to go through the NAS. You can imagine why that would be a problem.
To summarise, NAS drives can be a great way to upgrade your home or small office storage. They can allow collaboration and sharing of files between users, and should simplify your backup process. Just remember that a NAS is a small server that needs to be backed up as a matter of urgency. As long as you have that covered then a NAS can be a smart addition to your network.
Plenty of shops will sell you a “Mac Hard Drive” but there is no reason why you cannot use a windows format drive on a Mac. You just need to format it first. There was once a time when a drive was specially formatted by Apple to use on their Macs, but these days Apple use the same hard drives as everyone else. To use with the latest versions of OS X I would recommend following the steps below.
NOTE: Formatting your drive will destroy all the data. Make sure there’s nothing on there you need.
1. First attach the drive to your Mac. The Mac will notify you with a small finder window to initialise the drive. See below.
2. Once you have clicked initialize you will see the Disk Utility Application window. See below.
3. You need to select the drive you want to format in the left hand window of the utility as highlighted in blue. Internal drives show as grey and external drives show as yellow. At this point make sure you choose the correct drive, the utility will not allow you to format the internal boot drive. See below.
4. Now choose the Partition Tab. See below.
5. Now click on the Partition Layout drop down bar and choose the first option “1 Partition”. Also to the right under Partition Information give your drive a name and below that choose the partition type you want which will be Mac OS Extended ( journaled ). We are nearly there. You now need to click on the options tab in the bottom left of the utility window and choose GUID Partition Table and click okay. As you will read in the text information, this allows the drive to be used with all current OS X Macs. See Below.
7. Now all you need to do is click the apply button as shown in red below.
8. Another window will appear asking for confirmation to partition the drive. Click partition. See below.
9. A formatting window with a progress bar will now appear and then disappear when done. You will now see your named drive in the left window, which means that your drive is now formatted. Close the disk utility and the hard drive is ready to use. See below.