More Fibre Channel at Home 3/2/2002

I've added and changed my home setup a bit, and it seemed time to document some of the additions and changes.

First, I used a 4-bay chassis from Sandin Fuess (ebay username chirixi, he's also on the resources page.) This, combined with a single T-card from CSI (also on the resources page) and a used mini-tower, gave me a very nice 5-bay solution. I found 4 9G 10k RPM disks on ebay for $10 each to fill the four bays, and I had an extremely fast array.

Pictures of case and chassis

anticipation-times-four.jpg 40357 bytes 529 x 428	case-1.jpg 46861 bytes 576 x 768	case-2.jpg 76228 bytes 571 x 646	case-back-panel-2.jpg 46358 bytes 592 x 499	case-back-panel.jpg 28838 bytes 533 x 369
case-detail-1.jpg 114324 bytes 932 x 699	chassis-detail-2.jpg 103831 bytes 845 x 675	chassis-detail-3.jpg 139249 bytes 934 x 775	chassis-detail.jpg 104699 bytes 708 x 770

Unsafe at Any Speed

So fast, in fact, that it crashes the Linux QLogic driver if used in any configuration with parallel writes. RAID 0 is an especially fast way to kill the box. However, reads work fine and are godlike. Here are some rough numbers from my P3/504, 512MB, Debian, 2.4.17 box:

RAID level     Read rate (MB/second), sustained transfers
5              45
0              75-86
linear         20-23
1+0            30-37

Ohmygod its fast

Yes, that's not a typo - eighty five megabytes per second, sustained! It'll read a 1G file into /dev/null in under twelve seconds!

Too bad about the crashes, though. I tested the beejesus out of the hardware with the aptly named "molest" disk test script, so I am 100% confident that the drives and chassis are not the problem. More on that in a bit.

Heeding the Clariion Call

Shortly after I got all of this figured out, I was able to check out from work a monster ten-bay Clariion chassis, as OEM'd by SGI. (They called it a "Deskside Fibre Vault"). This has ten bays with custom rails, power supply sufficient to the task, and is way way oversized. Not to mention noisy, but what can you expect when you put ten spindles in one enclosure?

Clariion pictures

clar-front-1.jpg 55377 bytes 524 x 699	clar-full.jpg 79907 bytes 648 x 864	clar-mounting-detail.jpg 99033 bytes 940 x 441	clar-start-2.jpg 69766 bytes 576 x 432	clar-start.jpg 63774 bytes 471 x 629
success.jpg 39664 bytes 576 x 432

I, of course, immediately needed to fill the case. If nothing else, the blinking lights are magnificent. I ended up buying four more of the 9G 10k RPM disks from Ebay, giving me a spiffy matched set of seven disks. I figured that this could easily saturate the 100+MB/sec loop. The case uses odd standoffs to mount drives into the rails; I was able to use a combination of washers and polymer discs.

So, I have 1 36G IBM system disk (7200RPM), two Seagate 18G 7200s for misc data (in a RAID0 stripe), and the seven 9G disks, currently in in JBOD ("linear") configuration. Due to the problems noted above, I cannot currently run the planned RAID0 7-drive setup. I need to email whoever is maintaining the QLogic driver, the person in code comments denied any responsibility. Ahh, the cons of Open Source. "If it's broke, you get to keep all of the pieces."

Good safety tip, Egon

When one gets this much storage, a nerds' thoughts turn naturally to backup. It's almost spring, after all.

This array is about 105G, depending on whether the arrays are parity'd or not, so you need a Big Backup Device. From my times at Fermilab, I went shopping for a used tape robot. I found a nice Exabyte 10h on Ebay for $250, this is a 7G drive (uncompressed) that holds ten tapes. I got it, it worked even after I took it apart and reassembled it. (Note to my Dad, if he ever reads this: See, I have grown up!)

exb10h-1.jpg
66068 bytes
524 x 699

exb10h-2
867705 bytes
465 x 622

exb10h-3.jpg
76832 bytes
674 x 448

exb10h-4.jpg
63188 bytes
547 x 401

exb10h-5.jpg
47802 bytes
520 x 694

The Ferrari Testarossa of Serial Media

Of course, having put in the work, I managed to get my hands on a used Sony AIT-2 -- this is a nice toy. 50G uncompressed, 3.5 half-height, and 6MB/sec sustained transfers. Needless to say, I sold the tape robot and am happy with the Sony. To paraphrase Ferris Bueller, if you have the cash I highly recommend picking one up. It is so sweet.

Actually, I like the Exabyte Mammoth-2 quite a bit better, but none presented themselves. The LTO is even nicer in some ways if you're in that market. Feel free to mail me an eval unit!

Conclusions and further work

I've been experimenting with other Fibre channel cards - Interphase 5526 and Emulex LP-8000. Neither is as good as the Qlogic (non bootable, kernel driver is external). I can't get either into enough of a working state to see if the parallel-write crash is in the SCSI layer or in the qlogic driver itself. I've worked around it for now by using JBOD mode, which is slower but reliable. Of course, "slower" is relative since these are 10k RPM disks; it's certainly no slouch!

I'm thinking of moving the system disk to a 9G disk for speed and space, and doing some partitioning to speed the system up a bit. With both the Clariion and Feuss chassis, the system is reliable and fast. Both are recommended; I send my Feuss chassis to a friend since even I can live with only ten bays.

Credits

Thumbnails generated by HTMLThumbnail 1.1.4

Navigation Links

Introduction / HOWTO: Fibre Channel

FC at CDF page

FC at home page

FC at home, part two - chassis and tapes

Resources page

Back to home page