Expanding the capacity of my Media Server
Posted by Carlton Bale on April 9th, 2006
As I mentioned in a previous post, I've build a media server to store all of my music and movies for playback throughout my house. I decided to store everything the original, lossless format. This means that music is stored at full CD bit rate, and more significantly, movies are backed-up with all menus and extras in the original file format. Although I could save a bunch of space by compressing everything to a different format, I want to keep everything in the original format. The drawback is that this requires quite a bit of disk space, more than I originally anticipated.
I justed added another 500GB disk to my RAID array, for a total of 1.5TB of usable space. The great thing about the combination of an Areca 1230 RAID controller and Windows Server 2003 is that I can grow the array / partitions / drives as I add disks. Because this is not exactly a straight-forward process, and because I'm bound to need to do this again, I'm documenting the process here for my reference as well as for the benefit of others.
It is basically a three-step process. First, the RAID Set on the controller card must be expanded. For example, I had a 3-disk RAID 5 array with 1TB usable space (one 500GB disk is devoted to storing parity recovery data, so it is not usable space). After expanding the RAID Set, I have a 4-disk RAID 5 array with 1.5TB usable space. However, this additional space is not visible to the operating system; the operating system can only see Volume Sets and their sizes, not RAID sets (which are only visible to the controller card).
So at this point, the RAID Set is 1.5TB, but the Volume Set on that RAID set is still 1TB. So, the second step is to expand the Volume Set on the RAID controller card. A Volume Set on the RAID controller card can be thought of as a virtual disk drive that is presented to the operating system by the controller card. So I could either create a new Volume Set, which would appear as a new disk drive to the operating system, or expand the existing volume set. Since I don't want a bunch of useless drive letters, I expand the existing Volume Set using the RAID controller card utilities.
Now that the Raid Controller Volume Set is expanded, it appears as a new, larger 1.5TB drive to the operating system. However, this space is not allocated, so the third step is for the operating system to expand the size of an existing "partition" on this RAID Controller Volume Set. Now, here is the slightly confusing thing, the operating system refers to this 'partition" as a "volume set". This should not be confused with the RAID Controller Volume Set. The Operating System Volume Set is a "partition" within the RAID Controller Volume Set. At this point, the process is complete and an existing driver letter on the server will have more space available. Below are step-by-step instructions:
- Install the new drive in the server, connecting the drive to the RAID controller card
- Boot Windows Server 2003
- Go to the Areca RAID controller configuration webpage (must have the Areca ARC HTTP server software installed on the server, the default web address is http://127.0.0.1:81 if you are on the server itself, that machines physical IP address and port 81 otherwise. The default password is 0000)
- Expand the RAID Set using the "Expand RAID Set" feature.
- Continue to use the computer as usual. (It took 22 hours 50 minutes to add a 4th 500GB disk in background mode; it took 12 hours 20 minutes to add a 4th disk through the RAID bios a boot-up, but couldn't use the PC at that time.) You can even reboot if you need to, the expansion will continue in the background with no issues.
- When the RAID Set expansion is complete, perform a Volume Set expansion using the same Areca Web Interface. I simply expand an existing Volume Set to take up all of the available free space with the newly expanded RAID Set. It took about 1 hour 15 minutes to go from a 1TB Volume Set to a 1.5TB Volume Set and 2 hours 17 minutes to go from 1.5TB to 2.0TB.
- Once the expansion is complete, reboot Windows Server 2003, hit the F8 key the very instant Windows starts to boot, and select Safe Mode Command Prompt
- Logon to Windows, a command prompt will open, and run the DiskPart.exe utility. Here are some more details on the utility from the Microsoft website. (Windows XP does have this utility now. Norton Partition Magic may be an easier-to-use alternative if this seems too complicated. Instructions for Partition Magic are beyond the scope of this post.) This process takes only a few seconds. Here are the exact commands I typed:
- diskpart.exe (starts the partdisk utility)
- help (displays a list of commands, for reference only)
- list volume (to list the operating system volumes available and figure out the number of the volume you want to expand; think of these as disk partitions)
- select volume 2 (this selects my D drive, the drive I want to expand, may be different for you)
- extend (this extends the selected volume to take up all available free space after the current partition; if you don't want to use all available space, get more details from the Microsoft site listed above)
- list volume (to make sure the intended volume is in fact larger now)
- exit (to exit partdisk)
- CTRL+ALT+DEL (to reboot the computer back into "normal" mode)
Reboot Windows Server 2003; you're done!
May 13th, 2006 at 12:39 am
Great post and very well needed. Areca SUCKS as far as support goes, but the products is great. I've got a 1260 and using Linux. I'm pulling my hair out trying to get to 1.5tb so at least your posting gives me the general direction.
Again thx
January 28th, 2007 at 4:47 pm
To expand a Windows System volume, you need to boot to a Windows NTFS Boot Disk and run the commands above:
http://searchwincomputing.techtarget.com/originalContent/0,289142,sid68_gci1126671,00.html?topic=298540
Areca tech support is not that great, but if you contact their North American distributor, Tekram, you will receive great support.
February 25th, 2007 at 1:10 am
Thanks for documenting the above. I had figured out all the same except didn't know about the diskpart.exe tool. Thanks. I did hit another problem. I was going from 2TB to 2.5TB and diskpart said…. "Diskpart cannot extend partitions beyond the 2TB mark on large MBR disks." Bummer. Time to do some more research. Still, Thanks!
February 25th, 2007 at 10:51 am
Jim: For MBR volumes, the size limit is 2TB. You have to convert the volume to use a GUID Partition Table (GPT) to get past the limitation. GPT volumes/partitions can be as large as 18 exabytes. Here is a Microsoft TechNet article on the subject.
February 26th, 2007 at 3:00 pm
I work in a linux environment; regardless, this article is still helpful. F Mack is on point, Areca tech support is far far beyond awful; it is so bad, that it is unlikely that I will ever purchase from them again; It is so bad that I will work to convince all those I interact with on a professional basis, never to buy Areca.
Areca's US office is neglected by the Taiwan office [where all of the engineers are located]; if Areca wants to forgo the US market then they should continue to act the way they are currently acting. I love visiting a knowledge base on a website where the most recent post is three months old [sarcasm]. I love it when I email Areca support and the rep suggest what the solution to my problem is before I ever described the problem…
Does it make sense at this point to create a website that warns people never to Areca products in a datacenter?
February 26th, 2007 at 9:01 pm
Well, I guess I don't have quite the same complaints with Areca as you have. I think the manual is descent and their tech support (in Taiwan) did answer my e-mail. However, there is no substitute for knowledgable and immediate tech support over the phone. I would purchase another Areca card without hesitation, but I guess I've gotten past any potential learning curve.
March 14th, 2007 at 1:06 pm
This is GREAT Carlton, but here is the million $ question:
How do you expand a GPT disk that is already larger than 2TB?
If what I read is correct DiskPart.exe will not work beyond 2TB!!! Is that correct? Has anyone expanded a 2+TB GPT partition even further? My setup takes 16 drives and I have 8 500GB drives in now. If I add more drives I need to make sure I can expand without having to back everything up on tape 1st! While my Areca 1160ML will expand the RaidSet & VolumeSet just fine, it is Server 2003 SP1 I am worried about.
April 5th, 2007 at 12:06 pm
The answer to the million dollar questions is YES! You can use diskpart to expand a partition beyond 2TB with a GPT table. This (currently) is probably the only way to expand a GPT partition that I know of (even paragon server sees my gpt as unallocated space and only has the capability to expand/merge fat and ntfs).
I have a raid5 with a 3ware 9650se-16ml and had 4 750gbs in there (over 2 TBs to one drive so it's a GPT) and just added a fifth (and a 6th on the way). It took almost 72 hours to migrate the drive into the array (using the web utility), but once that was over using the diskpart command only took about 10 seconds to complete.
After the reboot the change was shown instantly in windows. This is however my 1st experience adding a hard drive to an exisiting array and I'm a bit curious because the hd I added was previously ntfs and I never changed the partition format, so I'm guessing that the migration wrote over all partition information (and any data on the hd).
I'm doing a verify unit now just to make sure everything is fine with my array, but so far it seems great.
Thanks and hopefully this info helps =)
Nate
April 18th, 2007 at 11:26 pm
Hi - I found this thread and have been dealing with the same thing. I'm Running windows 2003 Advanced Server / SP1. I have an Adaptec SATA RAID Controller that I started out with 5 - 500GB Drives. I added the remaining 3 drives and expanded the array. It is now showing at a 3.2TB Array / Volume. I went in to the O/S and ran the Diskpart.exe Utility and performed the steps to expand the partition and it worked but only expanded to 2TB. I have an additional 1.2TB of space showing on the Drive and can't seem to do anything with it whatsoever. Yes it is an MBR Partition so does that mean that I'm Screwed and have to remove the data and recreate the partition as a GPT (GUID Partition Table)?
April 19th, 2007 at 11:49 am
Pleblanc: You can only convert between MBR and GPT if the partition is empty. Here is an excerpt from the Microsoft Technet Article referenced in post #4:
April 19th, 2007 at 5:56 pm
Great now the question is how I offload / backup 2TB of Data so I can recreate the Array as a GPT that supports over 2TB. I wish I would have known that before I created it in the first place. Now I have over 1TB of space on the Array that I can't do anything with whatsoever. I can't even create a separate partition. So expanding it was a huge waste of time / money (took something like 24 days to expand (yes you read that right DAYS!!!).
April 19th, 2007 at 6:38 pm
24 days?!?! Using diskpart? Or migrating the additional units onto the unit?
Migration for me takes about 3 days with an additional 750gb, I guess if you do multiple drives it gets expotentially longer. As for disk part, that literally took 5-10 seconds to complete (I've added two drives at different times to this array and am going to add another shortly).
As for how to offload 2TB of data, I'd go to best buy and pick up 2 of those external terabyte drives and then return them once you're done =)
Nate
April 20th, 2007 at 12:09 am
Sorry if I wasn't clear the Expansion of the Array took something like 24 days. This was adding Two 500GB Drives. I previously added 1 500GB drive which took something like 12 days so I guess that makes sense but the array was online and fully accessible. I was supposed to have a "serious performance hit" but it was completely unnoticeable. When I was able to expand the original Expanded array to 2TB it took only seconds - WAY less than 10!
April 20th, 2007 at 12:25 am
I wonder why it takes you 12 days to migrate a 500gb drive and it only takes me 3 days to do a 750gb drive (my drives are 7200, but I'm sure yours are too). I also didn't notice a performance difference. I have my controller set to fastest rebuild/migrate, rather than i/o. I thought 3 days was an insane amount of time (and made me think raid 6 might be the better option).
April 20th, 2007 at 12:34 am
Keep in mind that if you have a drive fail a rebuild of the parity drive will be a whole lot faster than expanding. Expanding an RAID 5 array which is what I think we are all talking about involves a lot more than just a rebuild of a failed drive. I think that a rebuild will take a day or so which seems reasonable. Obviously if another drive fails in that time frame you are screwed which does make RAID 6 a little more attractive. It does have a performance trade off but offers a little more security. I'm not mission critical and even though 2 drives failing within a short amount of time is possible since they were bought in batches I think that RAID 5 is good for me for now. I might do the buying some Mass Storage devices to backup my data then return them. I really don't like doing that but in this case I don't really have a choice / need to keep them.
April 20th, 2007 at 6:41 am
Cool blog
I hope everybody read this article
thank you for infos
April 20th, 2007 at 9:43 am
The big difference between a rebuild and an expansion is the amount of data being relocated. With a rebuild, it is only 1 drives worth of data being reconstructed from parity (500GB for a 500GB drive) and it is not recreating parity info. For an expansion, the entire data set is migrated across all of the drives in the array (4->5 500GB drives results in 2.5TB of data being moved total and 500GB of parity being re-created.) So migrations take much longer. For Areca controllers, adding another disk and expanding does not not cause the loss of parity data, so if a drive fails during the migration, it will not result in data loss. Only if two drives fail will there be an issue, just as in standard RAID 5 operation.
April 20th, 2007 at 12:10 pm
Same with Adaptec Controllers - Once you apply the new configuration to the controller (Expansion no matter how many new drives) it is still a RAID 5 Array and can withstand a single drive failure.
April 20th, 2007 at 2:02 pm
Does anyone know if 3ware is like that (if a drive fails during migration if the parity bits are not lost). I'm assuming it's the same as the other brands.
Well 2 drive failures is rare in my opinion, however, what is more likely to happen is a bad parity bit. That's really why I like the looks of raid 6 (to me natural diaster will ruin my array before 2 of my seagates fail in the same day). I do maintenance regularly (once every 5 days), but it still worries me that a bit could be bad and when the drive fails that could be linked to something I need. Out of curoiusity, how often do you guys to maintenance?
In my case though, since it's for personal storage and not some enterprise solution, raid 5 is the right answer. I have never had to do a rebuild (i love seagates) so I'm not sure of how long it would take, but again I am very interested in why a migration for you with a smaller drive takes 4x longer than it does for me with a larger drive. I'm curious because whatever variable(s) that is would influence me on my next raid controller purchase.
Can you provide me the details of your raid setup?
Mine is this
3ware 9650SE-16ML PCI Express x8 SATA II Controller Card
Seagate Barracuda 7200.10 ST3750640AS 750GB 7200 RPM
TYAN S5160G2NR-RS Socket T (LGA 775) Intel E7230 ATX Server Motherboard
Intel Pentium D 925 Presler 3.0GHz
2 Sticks of Kingston 1GB 240-Pin DDR2 SDRAM DDR2 667 (PC2 5300) ECC Unbuffered Server Memory
And my installation of windows is on a seperate 160gb drive. I wonder if part of the reason it takes you so much longer is you have your os installed on your array.
Nate
April 20th, 2007 at 7:33 pm
3Ware is like this according to their website. They support Array Expansion as well as RAID Level Migration. Multiple Drive Failures can certainly occur especially if it is due to a manufacturing issues like the drive bearings from years gone by. This is why I mentioned I bought my drives in batches. If we are talking about a single drive that fails for whatever reason then yes a second failure in the time it takes to get a replacement drive if one isn't on hand and replace it in the array and rebuild is a risk that I'm willing to take at this point **knock on wood**
I don't do any maintenance on my array.
At the same time I don't know why it takes / took so long for my controller to expand the array. When I built the array it only took about 10 hours to create / initialize it. I thought it would take that long or maybe a little longer. I didn't expect days to weeks for this to happen.
I do understand the controller needs to recalculate the parity / data to utilize the new drives in the expansion process which I can see as being pretty time consuming. Obviously it would have taken a whole lot less time to backup the data and rebuild the array but since I don't have that kind of spare storage kicking around it wasn't an option but now it looks like I have no choice but to do something like that to get be able to use the additional 1TB of space I've added to the array.
My Setup is an Adaptec 2820SA SATA II Controller in an ASUS Pro Workstation Motherboard (PCI-X 133Mhz) with 1GB of Memory. AMD Semperon 3Ghz. All of this is installed in a Super Micro Server Chassis with 8 Hot Swap Drive bays and an SATA II Backplane. Oh yeah I also have a separate Hard Drive dedicated to the O/S.
Interestingly enough I have a second controller in another system that I had an External Array on but recently sold. Ultimately I think I will be going with a 16 Port 3Ware controller utilizing a 16 Port Hotswap / Backplane Chassis I picked up off ebay.
April 22nd, 2007 at 7:30 pm
Why don't you ever do any maintenance? I though this was an important task that should be done regularly on the array.
June 15th, 2007 at 9:22 am
super web site thank you very much
September 8th, 2007 at 1:09 am
Hey guys,
I thought id just share my experiences…
I have 2 X Highpoint 2320's. I very very highly recommend them, ill explain why shortly.
The first card has 8 X 320GB in raid5, or about 2.24TB (unformatted). Initially i was running XP SP2, so i could only see 1.99TB (2047GB issue). Anyway recently i purchased the second card and 5 X 500GB drives.
I created another RAID5 array showing up 2000GB (Unformated) space, which was about 1900GB in windows.
A few days ago i purchased 3 X 500GB drives and added them to array. The highpoint cards offer the OCE/ORLM option. So i added in the 3 drives into my server and expanded the array. Silly me moved the server into my bedroom while i was installing the new drives. It took about 26 hours to expand the array to include the new drives.
This is damn impressive as these cards are about half the price of the Acera ones.
Only have 1 sleepless night (server with 20HDD makes a lot of noise).
Anyway, when i went to expand the NTFS partition i run into massive issues. I couldnt get it to expand beyond the 2TB limit. After a lot of reading etc i decided to upgrade to Server 2003 SP1 (decided against 64bit xp). The 320GB Raid5 array was able to see the extra space, (although i couldnt expand it beyond 2TB).
However, i had converted the 500GB array to a dynamic drive to get past the limit.
Looks as though my only option now is to format the 500gb array, and restart it as a GPT drive….
Just need to find somewhere to backup 2TB of data :(.
Hopefully this helps someone else, and i recommend looking at the Highpoint cards… they are damn good.
Cheers
September 9th, 2007 at 12:24 am
Hey guys I thought I would check back in and give you an update on my set up. I've gotten away from the Adaptec Controllers and have now embraced 3Ware Controllers. I originally picked up a 3Ware 9500 12 port but since then I've settled on a 3Ware 9550-16ML Port Controller and originally started with 6 - 750GB Seagate Drives.
I'm running Windows 2003 Advanced Server and made this in to a GPT Disk so I can get past the 2TB Limitation.
Since building the Array I've added a 7th 750GB Drive. The expansion took a little under 4 days. I didn't notice any significant performance hit while expanding. Once the Expansion was complete I ran Diskpart.exe and expanded it to the full 4TB Capacity. This is obviously WAY better then the 28 days that it took to expand my old Adaptec set up.
I also built another server for a friend that is utilizing the 9500 12 Port that I sold him. He put in 4 - 320GB Drives to start then expanded that with a 5th. He said it took about 2 days for that expansion. He also put in 4 - 750's then added a 5th a while later and the same thing about 2 - 3 days to expand. He is running Windows XP Pro on this machine and has “Auto Carving” enabled and set to 2TB. Once he went past the 2TB the additional space showed up as a separate drive within the OS. Maybe not the ideal setup but hey a whole lot painful than paying for Server 2003!!!
On the 3Ware controllers I DO run the maintenance as it only takes about 4 hours for it to go through and check the parity which is sweet. The Adaptec was SO slow I couldn't deal with it. I also think that the 3Ware controllers offer better transfer speed than the original Adaptec 2820SA I was running.
These are by far the best servers I've ever built, I found an awesome case that I customized to give me 15 Hotswap Drive bays plus the DVD Drive and 2 internal 3.5" bays one for my boot / os drive and the other for my 16th drive if / when I get there.
If anyone is looking for a TON of storage I’d be more than happy to build more
September 10th, 2007 at 1:15 am
Since you guys are talking about GPT disk…
I have a server with HighPoint 2320 (PCI eX), 8 x 500GB Seagate, RAID5 in GPT mode (o/s is XP 64 bit, installed on a 80gb hd)
I'd like to "clone" the RAID5 array & expand to 15 x 750GB (keeping the existing MB & o/s)
The MB only has 1 PCI eX slot.
I need to be able to:
1) "clone" & "move" the data(approx 1.25TB worth) over the network from the RAID5 array and store it in my NAS
2) install a 16 channel 3Ware or Areca RAID6 card on the existing mb, after I remove the Highpoint
3) add 15 x 750 drives to the new controller
4) transfer the "cloned" data back into the new RAID6 array
Is this possible? How do I make this happen?
TIA
September 10th, 2007 at 2:10 am
I'm sorry to say that you will not be able to do a GPT Disk with Windows XP. GPT is only supposed in Windows 2003 and possibly in the new Home Server Edition however I can not confirm that since I haven't specifically played with that version yet.
This is the pain that I went though… However you Can still use all of the disk space if you enable Auto Carving on the 3Ware controller and set it to no higher than 2TB. This will report each 2TB increment of RAID Space to the os even thought they are still physically on the same Array.
Other then that the plan you have sounds perfectly doable unless I'm missing a question you are asking? If so let me know.
September 10th, 2007 at 2:44 am
Hey dexter,
My advice would be to purchase a board with 2 X PCIe ports. Install the second card, copy the data etc, then sell the board. You'll probably make a $20-50 loss on the board, but considering the cost of the raid cards and hdds its a small price to pay.
Cheers
September 10th, 2007 at 7:47 am
Errol: XP 64-bit does support GPT. If you have the free space available on your NAS, I see no problem with copying your data over to that server (copy / paste), installing the new card and drives, creating the new array, and copying the data back to the new array. Since it is only 1.25 TB, you could simply back it up to 2 of the 750 GB drives, create 13×750 GB array, copy the data from the 2 750 GB drives to the new array, then expand the array with the two 750 GB drives. You have a lot of options and I don't see a problem with any of them.
Funhouse69: Thanks for the info on the 3Ware controllers; auto carving is an interesting feature; it sounds like the automated equivalent of creating multiple 2 TB volume sets on single raid set.
September 10th, 2007 at 8:17 am
Errol: You are correct. A mb with 2 x PCIe would be a small price to pay.
If I take this route, how do I:
1) "copy" or "move" the o/s, C drive (installed on an 80gb hdd) to the new mb?
2) "copy" or "move" the RAID5 array over to the new RAID6 array?
September 10th, 2007 at 8:25 am
Carlton Bale: Since there is an app on the C drive, thats "talks" to the Highpoint controller, I assumed that the data from the existing RAID needed to be cloned instead of a copy/paste to the NAS?
If I were to copy/paste like you suggest, wouldn't the app on the C drive be "looking" for a specific RAID controller?
September 10th, 2007 at 10:13 am
Dexter: There is a driver for Windows that allows Windows to mount the volume set on the RAID array as a drive letter. After that, it is just a standard drive letter. The data on it can be treated the same way the data on any hard drive would be treated. Windows sees the data on the RAID array just as ti would the data on any hard drive and this is how you'll want to copy and backup the data. You seem to be talking about making an image of the RAID array or something; no need worry about this. Just backup your data to another drive, then restore it back to the new array.
September 10th, 2007 at 10:17 am
Carlton Bale: Thx for the advice. I was probably "overthinking" the RAID + imaging. Copy/pasting the data, certainly makes the operation a lot easier.
September 13th, 2007 at 1:38 pm
Plec you have an almost identically setup as me and glad to here it takes about the same amount of time. I'm actually migrating my 8th seagate 750 into the array now which should top the total space available to over 5 TBs. I have a few questions for you though.
Is the 9550 capable of OCE (Online Capacity Expansion I think it stands for)? I didn't think it was, meaning that if you added drives at a later time you couldn't add them to that specific array. You'd have to make a new raid5 or whatever on the card. This was a big downfall for me and is why I stuck with the 9650.
Also, can you let me know exactly what kind of case you have. When I purchased mine it said 17 bays and I never bothered to personally count. I realize now that it's actually only 14 bays (they counted 3 of them twice as both 3.5 and 5.25 or something stupid, I fell for their trick!) so I'm going to need to do a new case here eventually. I'm actually looking for a case that can hold 18 drives (16 in array + 1 cdrom + 1 hd for OS).
Thanks,
Nate
September 13th, 2007 at 1:40 pm
About the case, I'd prefer an 18 bay case, but the 17 you have will suffice since I can just pull out the cd rom and use an external one if I ever need it.
Thanks,
Nate
September 14th, 2007 at 3:23 am
Nate - I have a 9550-16ML that I got for a really great price. I was using Adaptec controllers and I have to say that 3Ware has impressed the heck out of me from the second I installed my first one. Because of that I ended up selling off my inventory of Adaptec controllers.
That said the 3Ware 9550 absolutely supports OCE. I'm not just saying that because the manual or spec sheet says so but because I have expanded my array successfully a few times now. I also know that the 9500 controllers support it as well as another system I built with a 9500-12 has been expanded a few times already with no issues.
When I say expanded I mean pop in another 750 Seagate, ad it to the existing array and when the expansion (3ware calls it migration) is complete the OS will see the additional space. From there run Diskpart.exe and extend the volume and you've got the full space.
The only thing that I can see that the 9650 offers that the 9550's does not is RAID 6. While it would be nice to have RAID 6 I don't think **knock on wood** that I will suffer multiple HD failures at the same time and would probably stick with RAID 5. At this point getting a replacement 750 from New Egg takes only a day so I guess I will take my chances and in my experience Seagate certainly stands by their product.
As for a case well I don't really know what your options are at this point. I haven't seen any cases to my knowledge with 17 external bays. However, if you are looking for a really nice case and don't mind spending the money then pick up a Super Micro SC836 which has 16 Hotswap SATA Drive Bays and a redundant power supply included. They also just released a 24 bay unit that looks really sweet as well. While I am a very big fan of SuperMicro I think that they are quite pricey for what I am looking to do.
With that in mind I went with a case that has 10 external 5-1/4" Bays and added Three 5 Bay hotswap enclosures which gives me a total of 15 Hotswap bays as well as the 10th bay for my DVD Drive. The case also has two internal 3-1/2" bays one of which I use for my Boot / OS drive and the other will be used for my 16th drive if / when I fill up the Hotswaps. Although this did require a little customization of the case I have to say that I am thrilled with the results and the case is one of the best made I've ever worked with including SuperMicro which in my experience is the best server case made.
Here is a picture if two of them that I have built.
http://www.pete-man.com/images/RaidServer.JPG
September 14th, 2007 at 11:17 pm
Pheblanc:
Could you please give me a part #/model # for the Black hard drive tray in your pic?
September 15th, 2007 at 10:55 am
Wow, that's really impressive (the case). And thanks for the company name on the 16 & 24 bay cases I'll have to check them out.
I have the 9650-16ml and yeah it supports raid 6, but I (like you) decided that the chances are two hd failures are pretty slim so I have mine setup for raid 5 (like I said in previous posts, I think the chance of an hd failure AND a bad parity bit are much higher).
As for the migration/disk part, yeah I've expanded my configuration about 4 times now (I started with 4 hds and now I'm up to I think 8 or 9.
When you're expanding your array do your hard drives make a lot of noise? Mine make one where it sounds likes the heads are crashing. The first time I heard it when expanding I was scared, but I've noticed it makes those awful noises (not just regular reading sounding noises) everytime I expand. Quite discomforting since my server sleeps in my bedroom closet so I hear what it's up to when I'm laying down.
I do have 1 more question for you. On your server do you have anything that manages temperature/heat and if it gets to high to shut down the server. As I said, my server is in my closet (so not the best ventilation) and I'm always paranoid about it overheating. I have a temperature monitor installed that came w/my motherboard (tyan system monitor or whatever), but if it starts to get too hot it doesn't have an option to shut it self down. I've seen some 3rd party software for this, but was wondering if you've had any personal experience with this.
And again great job on the case.
Thanks again,
Nate
September 15th, 2007 at 5:14 pm
My Seagate drives have a huge amount of head seek noise, that is very apparent during expansions, etc. My Samsung drives are pretty much inaudible regardless of what is going on. The Areca controller monitors drive temps and logs an error if they are too high (sounds alarm, sends e-mail etc.)
September 17th, 2007 at 12:51 am
Dexter - As you can see in the pictures I am using 3 Super Micro enclosures. These are SATA II Enclosures with built in Fans, Temp monitors with alarms. I've been using these enclosures for quite sometime (since back in the SCSI Days). I've tried others in the past and keep going back to Supermicro as I think that they just make a good high quality product compared to some of the cheap stuff that is out there.
The Part Numbers are:
CSE-M35T-1B (Black)
CSE-M35T-1 (White)
These are made to fit in to a specific Supermicro chassis but just so happen to be the same size as 3 5-1/4" Drive Bays which is what led me to this latest creation.
You can find these in several places but believe it or not one of the cheapest places I've found them at is Buy.com although they almost never seem to have the black ones.
September 17th, 2007 at 1:10 am
Nate – I’ve copied over several TB of data on to my array’s and I can’t say that the sound is any different from that as it is when I expand the array. That said I don’t feel that the drives I’m using (exclusively Seagate) are that loud. That said one of the other units I have built has been expanded using other drives (mix and match what ever is on sale) and I feel that has lowered the performance a little and does seem a little louder.
When it comes to cooling I’ve kind of had one of the “Live and Learn” experiences with my custom rig. On one of the hottest days of the year I was notified by one of my HDD Enclosures that I was reaching a Temp Threshold. Granted it was well over 100 degrees in the room it was in and very humid. Silly me was using the crappy stock fans that came with the case (3 – 90mm & 1 – 120mm).
Since then I replaced all of the fans with high volume / velocity fans and connected them to my ASUS motherboard which has the "Q-Fan" Option which allows the motherboard to control the speed of the fans depending on CPU / MB Temp and I have to say it works wonderful. Also the Supermicro enclosures also have fairly decent velocity 90mm fans in them as well so airflow hasn't been a problem since. It is great when you reboot the system all of the fans goes in to full velocity before the MB takes over the speed control (just like HP DL580’s do) and it sounds like the thing is going to lift off then it is so nice and quiet.
September 17th, 2007 at 5:43 pm
2 questions, slightly OT
What are my options:
1) if you have a pc with a "dead" motherboard & the mb is out of production. I need to keep the C drive.
2) Keeping the C drive intact, can you swap your mb from one brand to another without loss of data?
September 17th, 2007 at 5:54 pm
Dexter - I assume you mean you have to keep the content of the C drive. If you swap out the MB depending on the OS and how different the motherboard is it might boot then go nuts trying to identify all of the new devices but more than likely the system will blue screen.
The best bet is to put in a new drive to load the OS off of and put the old drive in as well as a secondary drive once you get the os loaded on the new drive copy all of your data off of the old drive.
If you are trying to save your applications that are loaded on the C drive you might be SOL without the original files / CD's.
September 18th, 2007 at 8:39 am
Dexter: You can try to boot to the old drive connected to a new motherboard and see what happens. It may auto-detect the correct drivers, but it will most likely have a problem with the IDE controller if it is different. What you can do though it re-install Windows on the old drive/new motherboard comb. Don't do a new installation, have it "update" the installation that is there. That will fix and driver issues and you will be able to keep your program installations. Generally, it's best to start from a fresh operating system install, but I've done the re-install several times with no issues.
Also, it's generally a better idea to partition your hard drive, with a C: partition for the operating system and program files and a D: partition for data and installation files. Partition Magic is an easy-to-use commercial product for re-partition existing drives into multiple partitions; otherwise, you can do this when you perform an operating system install on a new drive.
September 18th, 2007 at 10:05 am
Hello all,
Very good blog. It came really handy although I have a problem.
I have a Raid5 Volume of 3×750GB SATA II HDDs. The other day I decided to expand it so I purchased another 750gb same brand same fw HDD and plugged it on my controller.
Then I went from my Web configuration and expanded my raidset and now it gives me 3TB of raw capacity and 1TB of free raw capacity.
I then went to expand my volume set as you said, by clicking on the Modify Volume Set then selecting my volume and selecting the new Volume capacity to be 2250GB, but when I click on the Confirm Action and submit button I get a message "You Did Not Confirm The Operation. Action Aborted".
Here is my setup.
WinXP 64bit SP2
Controller: Areca ARC-1120
Firmware version: 1.42
Any help will be greatly appreciated.
September 18th, 2007 at 1:19 pm
Pannivas: Sounds like you are using the ARC HTTP interface. Try rebooting and going into the Areca Controller controller bios and expending it that way. By the way, are you using an file system format capable of addressing more than 2TB? If you are using Windows, you need Server 2003 or XP 64 or Vista, and you need a disk that is already formatted with GPT (not standard NTFS partitions.) Otherwise, you won't be able to address anything past 2TB. Perhaps you should create a new volume set if you don't have GPT file system already in use.
September 18th, 2007 at 1:38 pm
Thanks Carlton for the reply. I will try do that right now.
Yes i am using win xp 64bit with gtp. I had to do that the first time because the raw capacity of 3×750gb disks is more than 2gb so i was forced to anyway.
October 4th, 2007 at 6:59 pm
I am in the process of copying my old 500gb (some Adaptec raid5 IDE adapter, damn slow) into a new 1.5TB raid5 3ware 9650se array, and initialising at the same time… Pain in the but. Just my 3 cents. Copying should be finished in 20 hours.. no time known when initialisation will be finished ;D
October 17th, 2007 at 7:13 pm
I have used a lot of different RAID controllers over the years, with different capabilities and vastly different pricing. I've found my Areca 1220ML to be the best RAID controller I've ever used. I cannot tell if the support from Areca is good or bad, because I've needed it, and that's the point. It just works.
If you cannot afford an Areca controller, wait and save - or regret. Cheaper controllers tend to be just toys, and when you loose data you find yourself asking what you were thinking when saving a few bucks just to lose irreplaceable data.
October 17th, 2007 at 10:11 pm
I would say the same as Tad in that I've used to many RAID Controllers in the past but have never used any of the Areca Controllers as of yet. I'm certainly not opposed to giving them a try as I've heard / read good things about them.
With that said as I've mentioned in previous posts I've settled on 3Ware Controllers of late and I have to say that I'm absolutely thrilled with them. I haven't had any issues with them whatsoever and I've built several systems with them in recent months.
My personal server has just over 4TB of space on a 3Ware 9550-16 Controller. I've expanded the array a few times still with no issues whatsoever. I can't comment on 3Ware support either as I've never had to contract them.
I can however comment on 3Ware's sales. I ordered a set of Multi Lane to SATA breakout cables from them, they sent the wrong ones and when I called them up they said no problem and sent me the right cables over night on their dime or so I thought. I get my credit card statement and low and behold they charged me $95 for the overnight shipping (for something that weighs 2 pounds) for the mistake that they made and said they would cover. Well that was back in July and to date they still haven't credited me back that money. Each time I call, they see in my account what happened and say it will be credited the next business day. I'm at the point where I will have to fight this through My Credit Card Company. So keep that in mind if you ever deal with their sales department.
October 20th, 2007 at 7:24 pm
For 3ware I'd recommend ordering thru NewEgg.
December 12th, 2007 at 2:51 pm
This is a GREAT article and very helpful for anyone looking at single volume large capacity storage. However due to the ever increasing drive capacities and storage needs, it needs to be amended/enhanced with the following critical information:
When using an Areca controller (the most reliable I have ever used) you should setup your >2TB support as follows:
1) Try to create an initial RaidSet LARGER than 2TB to start off, so that the controller prompts you to choose how to handle that much data.
2) Use "LBA64" >2TB support option!
3) Once the RaidSet is created, create the VolumeSet.
4) In Server 2003 create an NTFS partition using GPT not MBR [you can convert from MBR to GPT (just right click-in Disk Manager) but only if there is no data on the partition as it will be lost!]
NOTE: If you already have data, and are not configured as described above, you will HAVE to backup your data and redo the config! No other workaround!
Now going forward, whenever you add more HD's to the controller and after you expand the Raidset & VolumeSet, you will need to boot into SAFE mode in windows and run DISKPART to expand the volume. If you try to do it while in normal startup mode it will NOT work!
With some of the Areca controller models handling 24 drives, following these guidelines you can have over 512TB in ONE NTFS volume as drives get larger, and never worry about space!
Enjoy!
Stefan.
Http://solutions.semotech.com
December 25th, 2007 at 2:30 am
Hello,
I will try and explain my case and maybe someone can help me with this problem that I have.
I had Win XP 64bit with 1 areca ARC-1120 SATA raid controller. On the raid controller I had 5×750GB Seagate 7200.10 HDDs on a single RAID5 volume of total 3TB.
When this volume was first configured I made it Dynamic Disk and used GTP so that Windows XP could read the greater than 2TB volume.
Everything was working fine but 2 days ago I decided to upgrade to Vista Ultimate 64bit.
I installed vista 64bit then chipset, SATA, Ethernet drivers. After that I installed the areca raid drivers (used latest) open Computer Management and import the 2.7TB foreign disk to Vista 64bit. It worked just fine, I even opened some files from windows explorer.
After that I installed some applications and video drivers (EVGA 8800GTS) and on the next reboot my volume was missing and when I opened Computer Management the volume was offline. From there on I couldn't re-import the disk.
After that I decided to install Win XP 64bit as a secondary OS to check if everything is ok from there. After raid driver installation (used latest) I couldn't even import the foreign disk from computer management under XP 64bit. I get this error in event viewer.
INTERNAL Error - The disk group contains no valid configuration copies (C10000B6).
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
The strange thing is that if I open the archttp software my volume is there and it’s shown at Normal state.
I also installed R-Studio software and that software can read my volume just fine.
I just cannot understand why I cannot import the volume on XP or Vista 64bit.
Any help will be greatly appreciated.
Hardware Config
Motherboard: Supermicro X7DAL-E+
CPU: Intel Xeon 5148
RAM: 4GB Kingston FB-DIMM
Video Card: EVGA 8800GTS 320MB
SATA RAID: Areca 1120 8port PCI-X
Hard Drives: 5×750GB Seagate 7200.10 series
December 27th, 2007 at 4:45 am
Still on topic of expanding the capacity of a storage server, I have ran into a new problem. My 650watt Thermaltake power supply cannot take any more hard drives.
I currently have 10 seagate 750gbs in the array + another hd for OS and a dvd rom. After installing the 11th hard drive the machine will power on and then power itself off after about 1-2 seconds. If I unplug the power of the new hd, the server works fine. This was not a problem I anticipated since my current calculations I should at MAX only be using 400watts (probably more like 350), unless the raid controller card takes an inordinate amount of power to work, but I don't that since it's just pci-express and doesn't even require the extra power running to it.
Anyways, so I'm wondering for those with large arrays, what type of power supply do you have?
Oh yeah and 1 more thing, I did attempt booting the machine with the 11th hard drive powered and un-powering the dvd rom, but received the same result. I was estimating that the hd's only used about 25 watts each, now I'm thinking it's much more than that.
Nate
December 27th, 2007 at 5:20 am
Nate - There are SO many variables when it comes to power consumption in any system. You have to consider not only the hard drives and the number of them, you also need to consider the CPU, Video card and lets not forget cooling as well.
What you are really running in to here is the initial surge. When a system powered up and the RAID Controller sends the start command to the drive each of them will start spinning at the same time which puts a huge load on the power supply. SCSI Drives dealt with this a lot better in allowing a delayed spin up command to be issued by the controller. While most SATA RAID Controllers support this most current drives do not. The only ones that I have found that do are Western Digital Drives and even so that was something I have to dig the information up for (you need to install a jumper on it then set the command in the RAID Controller).
If you would put your system on a power strip that has an AMP Meter or use some other method to measure the amount of current your system draws you will see that when you power up the system it might take as much as 10Amps when the drives all spin up but once they settled down it will drop down to something much lower probably in the 2 - 3Amp range which is roughly 240 to 360 Watts.
I have a system with 7 drives in it right now and it is taking only 1.2 to 1.3 amps of power once running. That is roughly 150 Watts of power.
A friend of mine built pretty much the same exact system (Same case, Motherboard, Hot Swap Cages and Drives) but went with an insane CPU and an even more insane video card. When I did some work on it I was shocked to see that it was drawing a constant 4.5 amps of power which is 540 watts! He's got a total of 15 drives in his so I thought that was the difference. When I saw the dramatic difference I did some checking and removed half the drives and it was still drawing just about the same amount of power. So that shows that the CPU / Video Card (I'm assuming the video card is the issue seeing that it has a direct power connection in to it).
With that said I am using Mushkin Power Supplies and haven't had any issues whatsoever. Both of the systems I mentioned are running the 550 Watt Dual-Rail versions which take the initial surge very nicely.
Hope this helps!
December 27th, 2007 at 5:47 am
Hard drive consume only about 8-10 watts once they are running. However, the initial start-up can require 2-3 times more power and I'm guessing that is your issue.
I have 10 drives connected to my Areca RAID controller + 1 system drive connected to the motherboard. I have an Antec Neo Power 500 ATX12V 500W Power Supply. The difference is that the Areca RAID controller supportes staggered spin-up, so the drives come online one after the other. The controller allows me to specify the interval between drive spin-ups; I use 2 seconds as the interval.
In my experience, it has nothing to do with the SATA hard drives and everything to do with the hard drive controller. All SATA drives will allow staggered spin-up; the key it controlling and coordinating that. My Areca RAID controller has a BIOS that loads at startup. It pauses the boot process while the drives are being powered-up. Once all drives are online, the controller bios allows the PC to continue booting.
December 27th, 2007 at 5:57 am
I have 3Ware Controllers and they support Staggered Spin up but according to them Seagate Drives SATA Drives do not support it??? I searched high and low on their website and couldn't come up with anything. I can say that without any doubt the drive all spin up at the same time with 3ware as well as Adaptec controllers with Delay Spin Up enabled and the settings set properly.
I have not used any other controllers except for SCSI which almost all support Delayed Spin up that actually works.
If someone knows how to get my 3Ware Controllers to work with Staggered Spin up please let me know.
December 27th, 2007 at 6:03 am
I think spin-up, spin-down are standard SATA commands. The Areca controller supports spin-down of the drive during normal operation for power savings. But maybe some drives spin-up as soon as they receive power and don't give the controller a chance to stagger spin-ups. I'm very pleased with the Samsung drives I have, much quieter than the Seagate drives.
December 27th, 2007 at 6:46 pm
Yes, thank you for the replies, that makes a lot of sense. I should have been more specific in my system though, it's pretty barebones. It's headless (no keyboard, mouse, monitor) so I use onboard video (I plug a monitor + keyboard in when I need to expand the array) and ethernet. The one card plugged in on the motherboard is the 3ware raid controller. I can't remember the processor, but it's nothing power consuming since I built this as a storage only machine. That's why I was so shocked about having a power issue, but after you explained the initial power surge it makes complete sense. I will look on my 3ware controller for a staggered startup option - I hope it has one because if it doesn't then my only other option is to upgrade the power supply (which will be a LOT).
However, since the controller supports hot-swapping, what if I have the computer on and then plug in the power to the drive? I've never done that before, but it's an option. Obviously if I ever power it down (which I only do when I'm installing a new hd), I'll have to unplug the power again before powering it on, but it seems viable.
As for measuring the power, I have my storage server plugged into an apc unit with an lcd screen so I will have to watch it and see if it updates quickly enough to show the draw.
Nate
December 27th, 2007 at 7:01 pm
Nate -
Yes the Controller Supports Hot-swapping but if you would power the system on without the drives plugged in the controller will see them as missed and fail them. This could lead to issues and possibly data loss. I don't think I would go this route.
I don't know what model 3Ware controller you have but it should support staggered spin up but as I mentioned my controller (9550-16 and a 9550-8) it doesn't work with my Seagate 750's for some reason. I actually opened up a web case this morning with 3Ware to see if I can get this resolved.
December 27th, 2007 at 7:26 pm
To add to the initial power draw of a hard drive. According to the Specs of a typical 750GB Seagate Hard Drive the start up power draw can be just under 35 Watts of power but once it is up and running it settles down to right around 13 watts of power. So you can see how your power supply can have an issue with this initial surge.
Also what I meant to say in my last post is that if you remove the power from the drives then connect them the controller will see them as missing during the initial post and fail them. This would be bad!
December 27th, 2007 at 7:29 pm
Pleblanc - I have a similar setup to you. Seagate's and a 3ware controller. I have the 9650-16ML and I can see in the web interface where it says staggered start-up, but I can't change those numbers (or enable/disable it) and the help file is useless as it says it can be changed, but doesn't say how.
I'm guessing this can't be changed in the web interface and needs to be changed in the other interface during boot-up. In the help file though it does say that you can enable it, but that if the hard drives don't support it, the setting will just be ignored (so even though it will appear to be working in the controller setup, the staggered setup doesn't actually happen). Now I'm thinking I should have opted for the enterprise editions of the hard drives rather than the regular seagate 750s…. Hopefully the regular ones will support it, but I'm doubtful.
Nate
December 27th, 2007 at 7:37 pm
Pleblanc - Yeah I would have to boot into the motherboard's bios each time, give the hd power, then exit bios, and then the motherboard bios runs again followed by the 3ware controller which would then see the HD, therefore the controller would have never known that the HD wasn't initially there. It's a poor solution don't get me wrong, but I might have to do it temporarily as I have an extra 750 in the machine right now not being used and I need the space.
The real solution is to get staggered startup working, but I don't see any mention of that feature on newegg's site. I have the 750's that end in "AS" - when I get home I will have to play around with it more - currently I'm just remotely connecting to the server and looking for the option in the web interface which I'm not seeing.
Nate
December 28th, 2007 at 6:19 pm
Hey everyone I thought I would let you know about something I discovered over the last few days.
As mentioned above I have a 3Ware 9550SX Controller. Recently I expanded my Array from 7 to 8 drives this process took about 2 days to complete. This is where it gets interesting, I have my controller set to auto-verify the array once a week. This past Wednesday my usual verify kicked in and I was surprised to see that I got a few hundred sector relocate errors.
This made me wonder why this wasn't caught when I expanded the array. The reason behind that is because the controller does NOT verify the data when it expands. To confirm this what I did was remove the failing drive then reinsert it and rebuild the array. This process completed successfully in only about 5 hours or so (which I thought was impressive) I was kind of surprised to see that I didn't have any errors. Once the rebuild completed I kicked off a verify and sure enough an hour or so in to it the same drive started to show errors / issues.
So a nice little safety tip is to Make sure that once you expand your array you immediately kick off a verify. This holds true especially if you don't have auto-verify enabled (which I do not believe is the default) or if you have verify scheduled for sometime in the future. Also if you do have "Follow Schedule" enabled in the Controller then you might have to Set Ignore Schedule to get it to kick off immediately.
I have an open case with 3Ware support and I mentioned this to them and they confirmed that when expanding the system does NOT verify the data. This to me seems WAY more risky then I would have ever thought. I guess that is why they say to make sure you have your data backed up huh?
BTW - I opened a case with Seagate to see if their ST3750 AS Drives support Staggered Spin up cause 3Ware won't answer the question. They keep referring me back to the drive manufacturer.
December 31st, 2007 at 3:36 pm
Nate - I thought I would drop you a note and let you know what I have discovered with my setup. As I mentioned I have a 3Ware 9550SX-16ML Controller and 8 Seagate ST3750640AS Drives. I thought that I tried the various Staggered / Delayed Spin up settings in the controller but wanted to take another look at them as they were not working.
I looked through my 3Ware settings through the 3DM software but couldn't really find anything that would let me change this setting. I was only able to when I rebooted the system and accessed the controller directly.
I am very happy to announce that I got my delayed spin up working. I am not sure what the default settings are for the controller but the config that worked for me was to choose a single drive at a time (just to make it easier for me to follow the start up), 4 second delay and OOBSATA for the method (this was not the default).
I power Cycled the system and sure enough the drives spun up one at a time. I can't believe that it took me messing with this myself to figure out. I have cases open with both 3Ware and Seagate and get nothing but each company telling me to check with the other company. Well this means that these drives ARE Power On Standby Capable.
It is worth mentioning that in my set up I have my drives in Supermicro Hotswap Enclosures but I do not believe that has anything to do with this working.
January 1st, 2008 at 5:12 am
Well I'm glad you got it working, because I sure can't seem to get it! I saw in one of 3ware's manuals about that staggered option only being able to be set in it's bios on bootup so I got that step, but the staggered startup was already enabled for me w/1 drive and 1 second delay and on sata. I played around with the numbers a bit (1 drive at a time, 6 seconds inbetween, 2 drives at a time, 4 seconds - all that stuff), but still no luck.
I watched the power meter on my ups unit and sure enough when I plug in that extra drive it hits 480watts-ish and the machine shuts off a second later. As soon as I unplug it the machine takes up about 470watts and boots fine (then shortly after goes to 200-220, and once booted stayed around 180).
I thought the problem might be in the motherboard bios (not letting a pci-e card have controller over power or something), but I didn't see any option that would make a difference. I would say that the drives are not compatible with staggered startup, but since you've got them to work I'm pretty stumped.
I'll play around some more, but if I get it working I'm guessing it will be a bios option somewhere and with this tyan motherboard they've got a million sub-menus and settings. I of course checked in windows that the 3ware controller accepted the changes correctly and everytime I would also turn off and unplug the computer wait a minute and attempt to boot it back up (with the additional drive plugged in). Everytime the power would surge to 480watts and the machine would shut off. Not the way I wanted to start the new year, but oh well, I'll get it working somehow! Any ideas?
Nate
January 1st, 2008 at 5:41 am
Do you have the "Staggered Method" set to SATA OOB or ATA-6? This is what made the difference for me, I believe mine was set to ATA-6 SATA6 and when I changed it I got it working.
I don't believe that anything in the Motherboard BIOS should prevent this from working. I believe the way that this works is the Controller is waiting to send out the Start Signal to the drive based on the staggered start up parameters.
From what it sounds like this should be supported by any SATA II Drive as it is part of the standard. I do however know that the Western Digital Drives need a jumper installed on it to work.
January 1st, 2008 at 5:43 am
Other background info my machine (don't think it matters), the OS does not boot from the storage array, but a single and separate sata drive, I have pulled out all of the 1.5gb jumpers on my seagate 750s to make them run at 3.0gbs, I do not have BBU unit installed on the raid controller.
If you changed any other settings anywhere, let me know. Thanks.
Nate
January 1st, 2008 at 6:09 am
It sounds like we have very similar set ups but just to confirm my set up…
I also boot of another HDD that is connected directly to the motherboard and not the 3Ware controller. I do not have a BBU installed on the controller and have had my drives set to 3Gbs since day one (and confirmed through 3Ware Software). Finally I do not have the BIOS enabled on the controller as I do not boot off it.
So I believe the only differences we have are motherboard, I am using an ASUS Workstation / Server Board and I also am using Supermicro CSE-M35T Hatswap Enclosures for my drives.
January 1st, 2008 at 11:06 am
thanks for the great tips guys.
I run a areca 1220 raid card with 4*500GB disks, I just added a 5th to expand the capacity. However, when i expanded to server2003 "dynamic" partition is has create two 'blocks' both called Data (D:)
is this correct or should there be only one 'block' called 'D' - the array is reporting the right size (2tb) but doesnt look optimal to me.
thanks again
Ian
January 1st, 2008 at 12:22 pm
Sounds correct to me if I understand your description. You expand the RAID set, then you have to expand the NTFS data partition to fill the newly available space.
January 1st, 2008 at 5:09 pm
Yeah the drives power just is not being controlled by the raid card, they all just instantly spin as soon as it's turned on, hmmm. In 3ware bios it's been turned on to OOB-SATA, but I even tried ata just as a fluke to see if I could that to work. I've played around with several motherboard settings, but no luck there either (although I did find one about enabling sata raid - which I did, but I'm guessing that one is more for software raid - but now that I type that I don't think my mb has a software raid). I guess I could try contacting 3ware's support, but from the sounds of it, they'll be of no assistance.
It's also lame that the power supply I have is for 650watts and craps out as soon as it peaks above 480 watts for a second.
I turned 3ware bios "off," but all that did was make the controller say "bios not installed" during it's initialization. I'll have to start digging through the motherboard manual, since I can't possible see how the settings for the 3ware controller are off.
Nate
January 1st, 2008 at 5:16 pm
Ian - I don't have the RAID Controller that you have so I can't speak exactly to your situation but I have expended my arrays several times without an issue.
Once you finished expanding the array through the controller are you saying that the additional space showed up within Disk Manager at the end of the same bar but at the end? If this is the case then all you have to do is run Diskpart.exe and expand the NTFS Partition.
Couple of things to keep in mind here though, With the set up you mentioned you would have just went over the 2TB limit of a "Regular" Partition. Did you make this a GPT Disk within Windows? Also some RAID Controllers have something called Auto-Carving that will report anything over 2GB Partitions to the operating system as a separate disk.
So the bottom line is that if you expanded the array properly and the partition within windows is GPT you can keep expanding the same drive without any issues.
January 1st, 2008 at 7:47 pm
Pete, what case was it that you modified to accept the SuperMicro drive cages?
What do you guys use to backup your 4+ TB arrays, or are you using them strictly as backup servers?
January 1st, 2008 at 8:13 pm
Jim - I have built several servers with 3 of the SuperMicro Hotswaps in each of them and haven't had any issues whatsoever. I am using AeroCool Masstige Cases. Each of them have a total of 10 5-1/4" Bays which allows for 3 of the hotswaps and one CD or DVD-ROM Drives.
Here is a picture of a few that I've built recently. http://www.pete-man.com/images/RaidServer.JPG
As for backup that is really the 64 thousand dollar question isn't it? My large servers are Video Servers so pretty much everything I have on there I have the original for. Obviously there are many, many backup solutions that are capable of this kind of backup but they are NOT cheap. Hopefully someone somewhere will come up with a low cost backup solution for these types of applications. I certainly don't plan on backup this amount of data on upwards of 1000 single layer DVD's
January 1st, 2008 at 9:44 pm
Yeah this is my storage server and the backup is that it's raid 5 with a ups unit. If my apt burns down, well then I have insurance and my storage server will be a small problem compared to the real stuff I lose.
Anyways, I am just about ready to give up w/my problem, I am absolutely stumped. I updated the driver and firmware for the controller (which btw is amazing, the controller was in use the whole time the firmware updated, no down time or annoying command prompts necessary), I played around with numerous motherboard settings - I thought I had it with enabling the memory cache to write through during bootup (the cache for irq port 16, which is the port that the raid controller is on, I saw the memory range in the driver info in windows), but that didn't work. Out of frustration I enabled everything to write in cache and then I wasn't able to get into bios (or bootup) so I had to flash it.
I think I am going to buy the motherboard that Pleblanc is using and try that - and I think tomorrow I will try to talk to tyan support, but that's pretty much a waste of my time. I don't really know what else could be different or going on, I have checked and re-checked everything, the power just always surges up every time the button is pressed. I really wish there was some software app out there to test the hard drive to see if staggered startup would work (perhaps the firmware on my 750's isn't compatible or something).
Nate
January 1st, 2008 at 10:12 pm
Assuming it's not my motherboard, then the only thing it could be IMO would be the firmware on one or all of these seagates, Out of the 10 drives I have, I have 3 different firmwares, they are
3.AAJ
3.AAC
3.AAK
I'm thinking of turning off auto-rebuild and unplugging all the older firmware versions (I'm assuming the lower the alpha letter the older it is) and just trying to see if I can get the newer ones to stagger-start. Pleblanc if you don't mind can you let me know what firmware versions you have so if you have one of the ones I have I can just use that initially for my test. Thanks,
Nate
January 1st, 2008 at 10:18 pm
Nate - I don't know if the firmware of the drives would have anything to do with it or not but just to compare I have a total of 8 drives in my array and only 2 firmwares at this point they are as follows. 3 Of them are 3.AAK and the remaining 5 are 3.AAE according to the 3Ware software. Lets say for the sake of argument that this would have something to do with it then some of the drives would spin up and others wouldn't???
Again I personally do not think that this has anything to do with the motherboard. Maybe it has something to do with the SuperMicro Enclosures that I am using but I don't think that is the case either. From what I understand the way this is supposed to work is the controller has to tell the drives not to spin up until it tells them to. If the controller doesn't tell them to "Wait" then they will spin up upon power up which is what they are doing. So as long as these drives connected to the controller which they are then it should work. I don't know what else could be the issue.
Also I forgot to mention that I am also using my RAID 5 as my backup so I would have to suffer data corruption or loss of 2 drives at the same time for data loss to occur. Hopefully **Knock on Wood** that doesn't happen anytime soon
January 1st, 2008 at 10:53 pm
I agree Pleblanc, I don't think the motherboard or the firmware versions is the issue, but I'm reaching here since I can't think of anything else and am looking for any differences between a working setup (yours) and one that's not (mine). I am going to try just the AAK's initially since we have that in common. If you have any other ideas, please let me know! I'm desperate!
Nate
January 2nd, 2008 at 2:24 am
Well on port 2 of the controller I had all AAK's, so I attempted to just boot up with just that plugged into the controller (but everything has power, so I was hoping that it would stagger the 4 AAk's that were on the 2nd port), but it didn't work (same result, 480 watts).
I even tried putting it on another PCI-E port in the motherboard, but didn't work. I will say that when you boot up into BIOS upon failure (since it's not seeing all the drives) it tells you that if you've unplugged them just to plug them right back in and do a re-scan. This gave me confidence that my workaround that I don't want to do will be okay. That being to manually plug in the power to the additional drives AFTER bootup has started. This is of course not what I want to do, but it's not really that big of a deal since I never have to turn off machine and a restart shouldn't produce that same huge jolt of power.
To me the problem has to either be 1) faulty 3ware card 2) motherboard is not allowing the card to have control over the devices 3) I just have drives that aren't compatible 4) is your setup really working?
I say #4, because I noticed that it sounds like my drives stagger start sometimes, but the power draw shows otherwise. Are you 100% sure that your drives truly are stagger-starting?
To anyone thinking about using these seagates I'd say they are fantastic drives, but be aware that stagger-start may not work. If I could do it all over again I would have spent a little bit more on the enterprise drives since I presume they HAVE to be stagger-start compatible.
Nate
January 2nd, 2008 at 4:29 am
Nate - I don't know what to say… I will make 100% sure that it is working on my end when get home in the morning. I can say that once I set the "Method" to OOB on the controller and set the Staggered Settings to 1 Drive every 4 Seconds without any doubt the controller was kicking on one drive every 4 seconds. I would hear them individually spin up then click once they were up to speed then it would go to the next drive. I didn't see the huge spike in power either.
Keep in mind that I have these drives in the Supermicro Enclosures which have Activity LED's on them. Each time one of the drives was powering on I would see that LED come on and stay on until the drive was up to speed and the heads engaged with the noticeable click then it would move on to the next drive.
I have another system in my network that has Western Digital Drives in it. I haven't tried that one yet but that would be comparing apples to oranges seeing that the drives aren't Seagate and I've been told that the WD's need to have a jumper installed on them to enable Staggered Spinup / Power On Standby.
One of the other systems I've built recently is with a friend of mine. He's got some WD drives as well as the same Seagate 750's. He is going to give it a try as well to see what happens on his. At the same time one of the systems I built recently has a total of 15 drives in it (all Seagate 750's) connected to two controllers. I should have access to that system sometime this week at which time I will check to see if it works on his.
January 2nd, 2008 at 5:45 am
Well definitely sounds like staggered startup is working for you - I'm curious to see how your friends works out. I do not have the LEDs plugged in from the card, but perhaps I should get them up to see exactly how everything is responding. It would be pretty funny if 3ware accidentally made their cards require that jumper to be connected on the card for this to work. Thanks for all your help so far, I wish other people with this setup would find this blog - I couldn't find anyone else on google who had this going on…
Nate
January 2nd, 2008 at 5:53 am
Nate - I do NOT have the LED's connected to the controlled. The LED Activity comes through the Supermicro Enclosures. The RAID Controller LED Connections have not worked for anyone that I know so far so I didn't even bother connecting them. So this would have nothing to do with this working at this point.
It seems that the only differences we have here are the Model of the cards, Motherboard and Enclosures. I suppose it is possible that this could have something to do with the enclosures but I don't believe that the backplane has any intelligence associated with it.
Just to make 100% sure as I said I will power cycle my main server (with the 750's) in a few hours when I get home. I will keep you posted as I get more information on the other systems that others have.
January 2nd, 2008 at 8:01 am
Nate - The Staggered Spin up functionality on my main server has been verified. This morning I rebooted the system, changed the "Method" to ATA-6 then power cycled. When I did this ALL of the drives spun up at the same time. This resulted in a power surge of just under 5 Amps of power which is almost 600 Watts of power. From there I went in to the controller and reset it back to OOB and power cycled again to compare results. This time the drives were spinning up one at a time and the power "Surge" never went past 2 Amps which is about 240 Watts. Just for the heck of it I went back in and changed the settings to spin up 2 drives at a time still with 4 seconds in between. This resulted in about the same power usage and visually the drives were showing me 2 at a time and I could hear them spinning up and the distinctive "Click" when the drive was ready.
I have e-mailed some of the others that I have built systems for to see if they could try this one of them said he will try today. This person has a somewhat similar set up as I do but the following differences. His controller is an old 9500 12 Port and he is using a Standard ASUS Motherboard (does not have any PCI-X Ports, he's just using it in a regular PCI Slot). Again comparing Apples to Oranges in a way but I still want to hear the results.
January 3rd, 2008 at 5:15 pm
In my testing of Staggered Spin-up or as some other companies might refer to as "Power On Standby" I opened a case with Seagate on their ST3750640AS 750 SATA Drives. After going back and forth with them trying to get something other than a reference to a Knowledge base article I finally got this answer.
**This drive fully supports staggered spin up but there is nothing that you need to do to "enable" the function on the drive itself. Usually these functions are fully controlled by the controller card at least on our drives. If you have any additional questions, let me know.**
At least this will answer this question but for others it will raise others. I also asked the same question to 3Ware who did the same thing by quoting nothing but KB articles which is why I went to Seagate as well. I did however find the proper "Method Setting" in one of their articles so I guess I did get some answers.
~ Pete
January 10th, 2008 at 1:50 pm
When I open the D drive under my computer on server it freezes for about 15 seconds before it will let me browse the directories (there's only like 10 directories on the D: not like it's loading thousands of entries here). When I went to do an analyze for defrag I saw that data (according to windows) was extremely fragmented. My question is, if I defrag through windows will it even do it appropriately (I don't know if it knows how to handle parity bits), and if not, is there any such of way cleaning up the data on a drive so that it runs more efficiently.
Nate
January 10th, 2008 at 3:04 pm
I'm positive that Windows has zero visibility to the parity bits. I also doubt it will make much difference, since every file is broken into blocks and written across multiple drives. But it definitely won't hurt anything. It might optimize the partition table or something, but I doubt you'll see much difference speed-wise.
March 7th, 2008 at 10:35 am
Lots of replies and didnt read through them all. But to point something out, there is no need to restart the server into safe mode to extend the partition using diskpart, likewise no reason to restart after this has been completed either. Simply rescan and refresh within disk manager. We do this all the time . Infact I have never restarted a server since nt4 to extend a partition or restart for the new disk space to show up either.
March 7th, 2008 at 5:16 pm
Thanks for the feedback. I think you have to go into safemode if you are expanding a system partition, which was the case for me.
March 10th, 2008 at 7:02 pm
I felt the need to add to this post - as I found this thread to be the most informative on expansion.
I was pretty sure what I was doing, going by the manual and having done it once in the past, but this thread re-assured me as to the procedure.
The once caveat of my adventure, was that I was taking a 6TB array to 9TB (RAID-6)without (and shoot me now) a full backup! So was was terribly afraid (rightfully so).
This is where I have something to add (besides really needing a backup):
My procedure was going fine - as I expanded the RaidSet on the controller - that took a day or so - left it running over the weekend. When I went to expand the VolumeSet, I selected Foreground Priority in the build - thinking that it should get it done faster. I hesitantly pressed verify and go. The progress of the controller showed 67% and slowly counting up - so far, so good.
Then, the defication hit the rotary oscillator!!!
I went to access the drive that the 6-9TB was supposed to house and the data was not there - the DRIVE was not THERE!!!
OH MY FREAKING GOSH !!! WHAT HAVE I DONE !!!
Do I pull the plug, and start recovery now? Do I wail bricks against my head for not having a backup? Do I start telling customers that files from the past 3 years are GONE!
I scoured the web - faintly remembering that I selected foreground tasking. NOTHING!! Nobody seems to have ever selected this.
Back to the manual - for the 7th time. Finally searching on the right terms. There it was:
"• Foreground Availability/Background Initialization
RAID 0 and RAID 1 volume sets can be used immediately after
creation because they do not create parity data. However,
RAID 3, 5 and 6 volume sets must be initialized to generate
parity information. In Backgorund Initialization, the initialization
proceeds as a background task, and the volume set is fully
accessible for system reads and writes. The operating system
can instantly access the newly created arrays without requiring
a reboot and without waiting for initialization to complete.
Furthermore, the volume set is protected against disk failures
while initialing. If using Foreground Initialization, the initialization
process must be completed before the volume set is ready
for system accesses."
To highlight:
"If using Foreground Initialization, the initialization
process must be completed before the volume set is ready
for system accesses."
So I said several prayers in hopes that it means that the volume is COMPLETELY offline until it is done. And then I waited….
…..
……..
…………
about 12 hours later, it came online. (then had to rebuild to a hot spare for a new drive that failed mid initialization - more pucker factor!)
I then ran diskpart.exe - thinking I couldn't screw it up any more!
To my relief - the drive is now 9TB and all the data is there!! AMEN!! Time to clean out my shorts!!
I guess the good news is that it doesn't take days when you choose foreground - but you do take it offline for that whole time.
BTW - my hardware consists of:
SuperMicro Xeon mobo with only one 1.8Ghz HT XEON cpu and 256Mb memory (hey, it's only a file server)
Areca 24port SATA controller - PCI-X
Lian-Lu? V2100 case (I think - 21 drive bays) housing 16 Seagate 750Gb drives - 14 drives in RAID-6 (paranoid RAID-5 with two parity drives instead of one) two dedicated hot spares
The array is not effectively de-fragged and alot faster with it not being so full.
Anyway, Thanks to Carlton on this very nice bit of information on the web to help us get through it all!
Hammy.