How to Break the 2TB (2 TeraByte) File System Limit
Posted by Carlton Bale on May 28th, 2007
In the 1990's, breaking the 640 KB memory limit of DOS was a big deal. Now, with single hard drives reaching capacities of 1 TB (1,000 GB), it is very easy to create a RAID array larger than 2 TB, but your operating system may not be able to handle it. But if you don't plan ahead, your operating system will only be able to address the first 2 TB and all that extra storage beyond 2 TB will be unusable. Here is an overview of some of the methods you can use to get around the 2 TB limit.
Equipment Requirements:
- Hardware RAID controller capable of 64-bit LBA addressing (for volume sizes greater than 2 TB). For this example, I'll use an Areca ARC-1230 RAID card.
- Several hard drives to connect to the RAID controller to create a RAID array. For this example, I'll assume (five) 1000GB SATA drives.
- Drives must be configured in a RAID level 5 Volume Set. For the first two examples, I'll assume all five drives are members the same RAID level 5 volume set. RAID level 5 requires the space of 1 drive to be allocated for parity data, so total available storage space for 5 drives will be 4 drives x 1000GB = 4TB.
Required Background Information: I'm assuming you already have an understanding of RAID 5, its benefits, and requirements. If not, read this Wikipedia article. Now, let's discuss the difference between RAID sets, Volume Sets, and Operating System Partitions.
- RAID Sets are groups of drives that a RAID controller groups together to act as one single array. The individual disks are not visible to the operating system but rather are controlled by a hardware RAID controller.
- Volume Sets are create by the RAID controller and reside on top of RAID Sets. A Volume Set set is presented to the operating system as a single, virtual disk drive. This is a little confusing, but the RAID level (RAID level 5 in this example) is determined when the Volume Set is created (not when the RAID Set is created.) It is possible to have multiple Volume Sets residing on the same RAID set, and the Volume Sets may even use different RAID levels.
- Partitions are created by the Operating System and reside on top of Volume Sets. (Volume Sets appear as virtual disk drives to the operating system.) You can use have the Operating System create one or more formatted partitions on top of a volume set.
Breaking 2TB Option 1 - Use Windows with NTFS and GUID Partition Tables (GPT) partitions. It is possible for Windows to use NTFS partitions larger than 2TB as long as they are configured properly. Windows requires that the GUID Partition Tables be used in place of the standard Master Boot Record (MBR) partition tables. You will need Windows XP x64 Edition or Windows Server 2003 Service Pack 1, Windows Vista, or later for GPT support. (It is possible to mount and read existing GPT partitions under Windows XP and 2000 using GPT Mounter from four.; however, their MacDrive product does not support GPT partitions.) There are a couple of stipulations for GPT disks. First, the system drive on which Windows is installed can't be a GPT disk because it is not possible to boot to a GPT partition. Secondly, an existing MBR partition can't be converted to GPT unless it is completely empty; you must either delete everything and convert or create the partition as GPT. Read this Microsoft TechNet article for more details on GPT.
- To summarize: 1 RAID array of five 1TB Drives -> 1 RAID level 5 Volume Set that is 4TB -> 1 NTFS GUID Partition Table Windows partition that is 4TB.
Breaking 2TB Option 2 - Use Linux with CONFIG_LBD enabled. Most Linux file systems are capable of partitions larger than 2 TB, as long as the Linux kernel itself is. (See this comparison of Linux file systems.) Most Linux distributions now have kernels compiled with CONFIG_LBD enabled (Ubuntu 6.10 does, for example.) As long as the kernel is configured/compiled properly, it is straight-forward to create a single 4TB EXT3 (or similar) partition.
- To summarize: 1 RAID array of five 1TB Drives -> 1 RAID level 5 Volume Set that is 4TB -> 1 EXT3 (or similar) Linux partition that is 4TB.
Breaking 2TB Option 3 - Use Standard Partitions and Create Multiple Volume Sets within a RAID array. A RAID array itself can be larger than 2 TB without presenting a volume set larger than 2 TB to the operating system. This way, you can use older file systems (that support only 2TB) and still have RAID 5 protection and more than 2 TB of total storage. To do this, put all 5 drives into a RAID set and create a 2 TB RAID Level 5 volume set — this will leave 2TB of the RAID set unused. Then create a second 2 TB RAID level 5 volume set. Boot into your operating system, create a partition on each of the 2TB virtual drives, and format each of the two 2TB virtual drives. The disadvantage is that there is not one single, large 4TB partition. The advantage is that 1) backwards compatibility for the file system and partitions and 2) they are both part of a RAID 5 array and are protected from single drive failures and only 1 drives worth of storage is sacrificed for RAID parity data.
- To summarize: 1 RAID array of five 1TB Drives -> 2 RAID level 5 Volume Sets that are 2TB each -> 2 standard NTFS (or any other) partitions that are 2TB.
Note 1: RAID Capacity Expansion. If your RAID card supports online capacity expansion, it is possible to expand any of the configurations above. For options 1 and 2, expand the RAID Set, then Expand the Volume Set, then Expand the Operating System partition. For option 3, expand the Raid Set, Create a 3rd Raid level 5 Volume set, and then create a third operating system partition. To learn more about expanding a RAID array on an Areca controller running Windows, read this article.
Note 2: Software RAID. Software RAID adds an additional level of complexity to RAID. For that reason, I recommend using a Hardware RAID controller. Having said that, I think everything mentioned above is technically possible if you are using software RAID, but I've never messed with it some I'm not positive.
June 4th, 2007 at 2:07 am
Hi, nice article.
Have you tried any of the options in your list?
And have you looked into expanding any of these partitions?
June 4th, 2007 at 8:38 am
Yes, I did try all three. Unfortunately, I didn't plan ahead and had to use option 3 (create a second volume set) for my production data. It was a standard NTFS/MBR partition, so I couldn't exceed 2TB with it. I expanded it from 500GB->1TB->1.5TB->2TB, then added another 500GB volume set for 2.5TB total. I used the instructions I documented in this article about Windows RAID array capacity expansion and it was painless.
I really wanted to switch to Linux, but I had zero confidence around expanding partitions. GParted is easy to use, but it gave me errors viewing large partitions (2.0TB total, combination of NTFS and ext3), so I gave up on it. I looked into command line tools, but I felt certain I'd be more likely to erase everything than successfully expand the intended partition. Windows does have some ease-of-use advantages.
July 11th, 2007 at 2:44 am
XP Service Pack 2 don't support GPT Disk?
July 11th, 2007 at 8:33 am
Nanashi:
From the original post: "You will need Windows XP Service Pack 2 or later or Windows Server 2003 Service Pack 1 or later for GPT support."
August 7th, 2007 at 2:26 pm
Suggest you go back and double check that advice about 32-bit Windows XP Service Pack 2. It does NOT support GPT disks.
Microsoft explicitly said this when they announced the changing of the limits for Windows 2003 Server and Windows XP 64-bit.
August 7th, 2007 at 3:02 pm
Richard: Thanks for pointing that out. I've corrected the post to say Windows XP x64 Edition in place of Windows XP SP2. I'd like to say it was my original intention to say x64 instead of SP2, but to tell you the truth I can't remember what I was thinking.
For additional reference, here a Microsoft GPT FAQ document that clearly states XP x64 is the only version of XP that supports GUID Partition Tables.
December 16th, 2007 at 6:04 pm
Hi, I could not make this work under linux. I have 4×1TB with Adpatec Raid 5. I created a 200GB RAID volume and left the rest alone and installed CentOS 5 x64 and although it installs, GRUB panics after initial reboot and cannot load the OS. Jonathan
December 17th, 2007 at 5:16 am
I think the problem is with GRUB; I've had similar problems. I was viewing a 2 TB partition in GRUB and it would not display correctly, much less resize. I'm far from a Linux expert but I do know that large volumes are better managed from a terminal window. I spent some time investigating this but can't remember the exact command to do so. Personally, I'd much rather have a well-designed GUI that minimizes my chances of doing something really stupid - like deleting all my partitions just because I typed a "b" instead of a "d". Unfortunately, I don't think that is an option.
December 18th, 2007 at 4:40 pm
Thanks for this information; I think you summed everything up nicely. I've been meeting a few customers here and there who have hit the 2TB limit, but I imagine we'll be seeing more.
December 19th, 2007 at 4:40 am
Hi and thanks for the info. I'm setting up a Linux server with 2.5TB worth of space (6 500GB HDs RAID 5), so it's always good to have information.
Jonathan: When I was first installing Linux on my desktop system with a RAID 0 configuration, I had the same issue at first. Just a suggestion, if you are using DMRAID, make sure BEFORE you install GRUB but AFTER you get the OS installed that you first install libdevmapper and DMRAID in a chroot of your system. Also, once GRUB is installed, edit the menu.lst to remove ANY lines of "savedefault", as that will prevent it from loading as well.
If you really want or have any problems, email me (pyrodrake1134 {at} gmail {dot} com). I wrote out a tutorial for a friend of mine, and it works fairly well using Ubuntu 7.04.
January 23rd, 2008 at 4:53 am
I heard that you can already use more than 2 terabytes on a single hard disk if it was using NTFS since FAT32 is the one that has the 2 terabyte limit. NTFS has a limit of 256 terabytes. Correct me if I'm wrong.
January 23rd, 2008 at 10:13 am
UnknownMan: Read the article; you are wrong. NTFS has a huge theoretical limit, but the partition tables used by default for all version of Windows have a much lower limit (2TB). You have to have a version of Windows that supports GUID partition tables and you have to format the "big drive" with the NTFS file system and specify that it use GUID instead of MBR partition tables. Without GUID, the limit for NTFS is 2 TB.
January 29th, 2008 at 6:20 pm
This thread is EXTREMELY helpful for our situation where we're producing over 2.5TB of data per week and we're stuck with Windows on both the client and server side due to lab equipment vendor requirements. It's not practical for us to keep moving data around and/or have multiple 2TB mount points etc. We have 6 scanners (fancy confocal microscopes) producing data, potentially simultaneously, theoretically continuously, but as a practical matter fewer and not all the time. The Windows workstations controlling these scanners and processing these images expect to be able to write the data to a Windows share. So my question is, if we built a Windows fileserver with >2TB filesystem, could we expose it as a normal Windows share and mount it from the lab workstations as usual? or would there be some complications/barriers that aren't self-evident? THANKS!
January 30th, 2008 at 9:18 am
Lee: In general, there should be an issues sharing a >2TB volume with clients. The server takes care of the file system, which is transparent to the client.
February 25th, 2008 at 9:56 am
Yes, one single logical volume is limited to 2 TB. We can have multiple 2 TB logical volumes on one single storage array. What we can do is to use windows "disk management" and use "spanned" or "get together" as many 2TB logical volumes as we want and "extend" as many 2TB volumes as we want in order to have 1 single >2TB unit.
March 22nd, 2008 at 3:44 pm
GPT Mounter
April 3rd, 2008 at 6:57 pm
Note that to boot from a GPT volume you must have EFI firmware. Intel Macs have it, but even then you must make sure all OS installed on the HD uses EFI to boot, instead of booting through the CSM, which would require a MBR.
May 5th, 2008 at 12:29 pm
I have a HP server runing 2003 server sp2 on a 32 bit system.
I have a MSA attached with (8) 750gb sata hardives and want this to be all one drive for stoarage. I've went though and tried option 1 but can't get more than 2tb of hardspace to be recognized……….anyone have the same issues????
Thanks
May 23rd, 2008 at 3:59 pm
This is all great info… however if I have a IP-SAN with >2tb shares on them I realize that I can't see these in XP 32 bit. My next question is if I connect my IP-SAN to a Win2k3R2 server, can I share partitions >2tb out to my XP 32 bit stations?
I can't seem to locate this info…
Thanks for your responses!
June 26th, 2008 at 11:55 pm
How I broke out with a 2095.55GB NTFS partition (larger than 2TB) in Windows Vista 64 (3×750RAID0):
Originally, the drive was split in two with a 2048GB and a smaller almost 50GB partition. I should buy a RAID card. Best of luck
Go in to disk management and right-click on the new disk on the left hand side. ->>Convert to GPT<<- (~disk must be empty, unformated~) Please read what you're doing BEFORE you do this, I CAN'T OVER EMPHASIZE THIS, PERIOD. You can now create larger than two Tib partitions.
July 24th, 2008 at 8:43 am
I have created a server with Vista Ultimate and it has a 250GB boot drive and 15 1TB drives in a RAID5 configuration using a 3Ware RAID card and it works wonderfully. This article and thread was extremely usefull in helping me do this, therefore my thanks to all concerned. I can share the 13.5TB volume in Vista and XP Home clients have no problem in accessing the data. I just thought I'd share my experience here.