Never did quite get the two new boot disks set up with identical partition sizes, but it doesn’t matter since I attached them both to the mirror (which was limited by the size of the old 80GB disks) first, and then detached the old disks.  At that point it expanded up to the available size, which was the smallest partition on the two new drives.  They differ by a MB or two out of 160GB, not important.

ZFS root pools have some requirements and best-practices at variance to other ZFS pools. One of the most annoying is that you can’t use a whole disk, and you can’t use an EFI-labeled disk. This is annoying because for most ZFS uses using a whole disk is the best practice, and when you do that ZFS puts an EFI label on that disk.

So, when you try to use in a root pool a disk you’d previously used somewhere else in ZFS, you often see this:

bash-3.2$ pfexec zpool attach rpool c4t0d0s0 c9t0d0s0
cannot attach c9t0d0s0 to c4t0d0s0: EFI labeled devices are not supported
on root pools.

What do you do then? Well, you google, of course. And you find many sites explaining how to overwrite an EFI label on a disk. And every single one of them omits several things that seem to me to be key points (and which I had to play around with a lot to get any understanding of). The fact that ZFS is what drew me back into Solaris, and that I wasn’t ever really comfortable with their disk labeling scheme to begin with, is no doubt a contributing factor.

This is going to get long, so I’m putting in a cut here.

And I’ve now found, in the relatively obvious place (but it doesn’t show up in the google searches I tried) the good and fairly official article on how to do this. So maybe you should skip me and go right there. They’ve got a couple of neater ways of doing things than I did.

The first point, which comes fairly easily from the google hits, is that I need to use “format -e”, expert mode.  That much was useful.

bash-3.2$ pfexec format -e
Searching for disks...done

AVAILABLE DISK SELECTIONS:
 0. c4t0d0 <DEFAULT cyl 9726 alt 2 hd 255 sec 63>
 /pci@0,0/pci10de,cb84@5/disk@0,0
 1. c4t1d0 <ATA-ST380811AS-E cyl 9726 alt 2 hd 255 sec 63>
 /pci@0,0/pci10de,cb84@5/disk@1,0
 2. c5t0d0 <ATA-HITACHI HDS7240S-A47A-372.61GB>
 /pci@0,0/pci10de,cb84@5,1/disk@0,0
 3. c5t1d0 <ATA-ST3400620AS-C-372.61GB>
 /pci@0,0/pci10de,cb84@5,1/disk@1,0
 4. c6t0d0 <ATA-HITACHI HDS7240S-A47A-372.61GB>
 /pci@0,0/pci10de,cb84@5,2/disk@0,0
 5. c6t1d0 <ATA-HITACHI HDS7240S-A47A-372.61GB>
 /pci@0,0/pci10de,cb84@5,2/disk@1,0
 6. c9t0d0 <ATA-WDC WD1600BEKT-6-1A13-149.05GB>
 /pci@0,0/pci10de,376@a/pci15d9,a580@0/sd@0,0
 7. c9t2d0 <ATA-WDCWD1600BEKT-6-1A13 cyl 21543 alt 2 hd 16 sec 128>
 /pci@0,0/pci10de,376@a/pci15d9,a580@0/sd@2,0
Specify disk (enter its number): 6
selecting c9t0d0
[disk formatted]

Note that c9t0d0 (#6) has a variant label string compared to the other identical drive (#7). We’ll fix that while we’re at it.

Next we have to delete the existing EFI label using fdisk.

format> fdisk

 Total disk size is 19457 cylinders
 Cylinder size is 16065 (512 byte) blocks

 Cylinders
 Partition   Status    Type          Start   End   Length    %
 =========   ======    ============  =====   ===   ======   ===
 1                 EFI               0  19457    19458    100

SELECT ONE OF THE FOLLOWING:
 1. Create a partition
 2. Specify the active partition
 3. Delete a partition
 4. Change between Solaris and Solaris2 Partition IDs
 5. Exit (update disk configuration and exit)
 6. Cancel (exit without updating disk configuration)

Enter Selection: 3

Specify the partition number to delete (or enter 0 to exit): 1

Are you sure you want to delete partition 1? This will make all files and
programs in this partition inaccessible (type "y" or "n"). y

Partition 1 has been deleted.

             Total disk size is 19457 cylinders
 Cylinder size is 16065 (512 byte) blocks

Cylinders
 Partition   Status    Type          Start   End   Length    %
 =========   ======    ============  =====   ===   ======   ===

WARNING: no partitions are defined!

SELECT ONE OF THE FOLLOWING:
 1. Create a partition
 2. Specify the active partition
 3. Delete a partition
 4. Change between Solaris and Solaris2 Partition IDs
 5. Exit (update disk configuration and exit)
 6. Cancel (exit without updating disk configuration)
Enter Selection: 1
Select the partition type to create:
 1=SOLARIS2  2=UNIX        3=PCIXOS     4=Other
 5=DOS12     6=DOS16       7=DOSEXT     8=DOSBIG
 9=DOS16LBA  A=x86 Boot    B=Diagnostic C=FAT32
 D=FAT32LBA  E=DOSEXTLBA   F=EFI        0=Exit? 1

Specify the percentage of disk to use for this partition
(or type "c" to specify the size in cylinders). 100

Should this become the active partition? If yes, it  will be activated
each time the computer is reset or turned on.
Please type "y" or "n". y

Partition 1 is now the active partition.

 Total disk size is 19457 cylinders
 Cylinder size is 16065 (512 byte) blocks

 Cylinders
 Partition   Status    Type          Start   End   Length    %
 =========   ======    ============  =====   ===   ======   ===
 1       Active    Solaris2          1  19456    19456    100

SELECT ONE OF THE FOLLOWING:
 1. Create a partition
 2. Specify the active partition
 3. Delete a partition
 4. Change between Solaris and Solaris2 Partition IDs
 5. Exit (update disk configuration and exit)
 6. Cancel (exit without updating disk configuration)

Enter Selection: 5
format>

Sorry that’s a bit long. It’s also edited; fdisk uses cursor positioning to update the screen in place, and I’ve hacked through that in an attempt to present it as a readable transcript. Linespacing has been extensively altered. But it shows us having successfully removed the EFI label and put on a new label.

However, we still don’t have quite the right partitions for a ZFS root pool.

format> p

PARTITION MENU:
 0      - change `0' partition
 1      - change `1' partition
 2      - change `2' partition
 3      - change `3' partition
 4      - change `4' partition
 5      - change `5' partition
 6      - change `6' partition
 7      - change `7' partition
 8      - change '8' partition
 9      - change `9' partition
 expand - expand label to use whole disk
 select - select a predefined table
 modify - modify a predefined partition table
 name   - name the current table
 print  - display the current table
 label  - write partition map and label to the disk
 !<cmd> - execute <cmd>, then return
 quit
partition> p
Current partition table (default):
Total disk sectors available: 312565390 + 16384 (reserved sectors)

Part      Tag    Flag     First Sector         Size         Last Sector
 0        usr    wm                34      149.04GB          312565390
 1 unassigned    wm                 0           0               0
 2 unassigned    wm                 0           0               0
 3 unassigned    wm                 0           0               0
 4 unassigned    wm                 0           0               0
 5 unassigned    wm                 0           0               0
 6 unassigned    wm                 0           0               0
 7 unassigned    wm                 0           0               0
 8   reserved    wm         312565391        8.00MB          312581774

partition> l
[0] SMI Label
[1] EFI Label
Specify Label type[1]: 0
Auto configuration via format.dat[no]?
Auto configuration via generic SCSI-2[no]? yes
Warning: no backup labels
partition> p
Current partition table (default):
Total disk cylinders available: 152615 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders          Size            Blocks
 0       root    wm       0                 0         (0/0/0)              0
 1       swap    wu       0                 0         (0/0/0)              0
 2     backup    wu       0 - 152616      149.04GB    (152617/0/0) 312559616
 3 unassigned    wm       0                 0         (0/0/0)              0
 4 unassigned    wm       0                 0         (0/0/0)              0
 5 unassigned    wm       0                 0         (0/0/0)              0
 6        usr    wm       1 - 152614      149.04GB    (152614/0/0) 312553472
 7 unassigned    wm       0                 0         (0/0/0)              0
 8       boot    wu       0 -      0        1.00MB    (1/0/0)           2048
 9 alternates    wm       0                 0         (0/0/0)              0

format> type

AVAILABLE DRIVE TYPES:
 0. Auto configure
 1. DEFAULT
 2. ATA-ST380811AS-E
 3. ATA-WDCWD1600BEKT-6-1A13
 4. other
Specify disk type (enter its number)[3]:
selecting c9t0d0
[disk formatted]

format> p

PARTITION MENU:
 0      - change `0' partition
 1      - change `1' partition
 2      - change `2' partition
 3      - change `3' partition
 4      - change `4' partition
 5      - change `5' partition
 6      - change `6' partition
 7      - change `7' partition
 9      - change `9' partition
 select - select a predefined table
 modify - modify a predefined partition table
 name   - name the current table
 print  - display the current table
 label  - write partition map and label to the disk
 !<cmd> - execute <cmd>, then return
 quit
partition> p
Current partition table (original):
Total disk cylinders available: 152615 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders          Size            Blocks
 0       root    wm       1 - 152614      149.04GB    (152614/0/0) 312553472
 1       swap    wu       0                 0         (0/0/0)              0
 2 unassigned    wu       0                 0         (0/0/0)              0
 3 unassigned    wm       0                 0         (0/0/0)              0
 4 unassigned    wm       0                 0         (0/0/0)              0
 5 unassigned    wm       0                 0         (0/0/0)              0
 6 unassigned    wm       0                 0         (0/0/0)              0
 7 unassigned    wm       0                 0         (0/0/0)              0
 8       boot    wu       0 -      0        1.00MB    (1/0/0)           2048
 9 alternates    wm       0                 0         (0/0/0)              0

Turns out there’s a reason I can’t just delete the overlapping s2 slice:

bash-3.2$ pfexec /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2
/dev/rdsk/c9t0d0s0
cannot open/stat device /dev/rdsk/c9t0d0s2

I have no idea what it’s doing; maybe assuming that s2 is the whole disk?

So I have to go back and recreate s2.

partition> 2
Part      Tag    Flag     Cylinders          Size            Blocks
 2 unassigned    wu       0                 0         (0/0/0)              0

Enter partition id tag[unassigned]: back
Enter partition permission flags[wu]:
Enter new starting cyl[0]:
Enter partition size[0b, 0c, 0e, 0.00mb, 0.00gb]: ?
Expecting up to 312555520 blocks, 152615 cylinders,  152615 end cylinder,  15261\
5.00 megabytes, or 149.04 gigabytes
Enter partition size[0b, 0c, 0e, 0.00mb, 0.00gb]: 152615c
partition> p
Current partition table (unnamed):
Total disk cylinders available: 152615 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders          Size            Blocks
 0       root    wm       1 - 152614      149.04GB    (152614/0/0) 312553472
 1       swap    wu       0                 0         (0/0/0)              0
 2     backup    wu       0 - 152614      149.04GB    (152615/0/0) 312555520
 3 unassigned    wm       0                 0         (0/0/0)              0
 4 unassigned    wm       0                 0         (0/0/0)              0
 5 unassigned    wm       0                 0         (0/0/0)              0
 6 unassigned    wm       0                 0         (0/0/0)              0
 7 unassigned    wm       0                 0         (0/0/0)              0
 8       boot    wu       0 -      0        1.00MB    (1/0/0)           2048
 9 alternates    wm       0                 0         (0/0/0)              0

partition> l
[0] SMI Label
[1] EFI Label
Specify Label type[0]: 0
Ready to label disk, continue? y

Warning: no backup labels

partition> q

FORMAT MENU:
[long menu snipped]
format> q

bash-3.2$

And that was in fact that.

bash-3.2$ pfexec zpool attach rpool c4t0d0s0 c9t0d0s0
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c9t0d0s0 overlaps with /dev/dsk/c9t0d0s2

bash-3.2$ pfexec zpool attach rpool c4t0d0s0 c9t0d0s0
Please be sure to invoke installgrub(1M) to make 'c9t0d0s0' bootable.

bash-3.2$ pfexec /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c9t0d0s0
stage1 written to partition 0 sector 0 (abs 16065)
stage2 written to partition 0, 271 sectors starting at 50 (abs 16115)

bash-3.2$ zpool status rpool
 pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
 continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 23.08% done, 0h1m to go
config:

 NAME          STATE     READ WRITE CKSUM
 rpool         ONLINE       0     0     0
 mirror      ONLINE       0     0     0
 c4t0d0s0  ONLINE       0     0     0
 c4t1d0s0  ONLINE       0     0     0
 c9t2d0s0  ONLINE       0     0     0
 c9t0d0s0  ONLINE       0     0     0  1.47G resilvered

errors: No known data errors

bash-3.2$

Oh, we should check the partitioning directly:

bash-3.2$ pfexec prtvtoc /dev/dsk/c9t0d0s0
* /dev/dsk/c9t0d0s0 partition map
*
* Dimensions:
*     512 bytes/sector
*     128 sectors/track
*      16 tracks/cylinder
*    2048 sectors/cylinder
*   21545 cylinders
*   21543 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
* Unallocated space:
*       First     Sector    Last
*       Sector     Count    Sector
*           0      2048      2047
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
 0      2    00       2048 312553472 312555519
 2      5    01          0 312555520 312555519
 8      1    01          0      2048      2047

bash-3.2$ pfexec prtvtoc /dev/dsk/c9t2d0s0
* /dev/dsk/c9t2d0s0 partition map
*
* Dimensions:
*     512 bytes/sector
*     128 sectors/track
*      16 tracks/cylinder
*    2048 sectors/cylinder
*   21545 cylinders
*   21543 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
* Unallocated space:
*       First     Sector    Last
*       Sector     Count    Sector
*           0      2048      2047
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
 0      2    00       2048 312553472 312555519
 2      5    01          0 312559616 312559615
 6      4    00       2048 312553472 312555519
 8      1    01          0      2048      2047
bash-3.2$

I’m going to call this a success, for now.

Actually, most of the excitement was earlier. In playing around with the new disks, I’d made them into a ZFS pool using the whole disk.  This put EFI labels on the disks, which Solaris / ZFS don’t support in a root pool.  So then I had to somehow get the disks relabeled and the partitions redrawn.  This turns out to be a horrible process which is not documented anywhere. The blogosphere is full of pages saying how to do it, and none of them actually tell you much.  Okay, use format -e, that’s helpful.  But they never say what device file to use, and none of the obvious ones exist.  I think you can maybe use any device pointed at the right disk for part of it. Also, I had to create an S0 manually, and I”m not sure I did it ideally (doesn’t matter much, since these disks are 4 times as big as they need to be).

I’m deeply confused by Solaris disk labeling, going back to SunOS days; even then, I thought it was absurd,  fact suicidally idiotic, to describe regions of the disk used for different filesystems which overlap. Okay, you’re not supposed to use any two that overlap for filesystems, but nothing stops you. The whole setup is just baroque, weird, stupid. And then, on x86 hardware, this Solaris idiocy takes place within one real partition (although Solaris documentation tends to call their things partitions).

So, I had to find a way to overwrite EFI labels with SMI labels. Apparently the secret is to use “format -e”. None of the pages said anything about manually creating partitions (or gave any clues for what space you could use; I believe you have to leave space at the start for the boot stuff). Anyway, totally infuriating partial documentation, and then a large group of aficionados giving slightly variant documentation with slight differences, all of it missing the key points.

Did I mention that I’m annoyed?

So I’m going to chase this for a while, until I get it actually figured out, or until I go postal; whichever comes first.

I’ve temporarily given up on getting incremental send/receive working. I’m liking my results with full backups, and have some reason to believe that the next OpenSolaris release will fix the bug I’m seeing, and can’t see how to proceed on that in the short term. With a third backup drive, having just full backups isn’t so limiting, either.

I’ve got the 4x 2.5″ adapter installed in the 5.25″ bay, and the 8 port SATA controller hooked up to it, and two drives in it, and I’ve copied the system rpool to a new pool there called rp2. And am now looking at details of how to finish making that pool bootable, probably involving booting from a livecd to rename it among other things.

Weird problem with the SATA controller—the bracket on the end of the card doesn’t match the cutout in the case. I had to remove the bracket, leaving the card unsupported, which is clearly not viable in the long term with two sets of stiff cables attached to it (right now it’s supported with a bit of gaffer’s tape).

Haven’t looked into booting from the new controller, either; it’s possible I can’t, I suppose, but if so putting both boot disks on the old controller isn’t terribly painful, though it ruins my perfect plan to split every mirror pair across controllers.

There’s also a problem with the bottom left 2.5″ tray, but I’m ignoring that for now since I only need two drives in the 2.5″ bay to finish my upgrade.

Don’t know that it might not be better to install to the new drives from scratch, but there are issues duplicating the configuration down to UIDs and GIDs, which is necessary for the data pool to be accessible to the users when I import it.

Still, all the new hardware seems to be working, which is good.

Then it occurred to me to compute the ratio.

2,916,667

The price per byte of that first disk drive was very nearly three million times higher.

Three.  Million.

That’s a lot.

I like living in the future.

Seems to be hung when I got up this morning, dammit.  System and pools are responsive, but there’s been no progress and no disk IO since I first checked when I got up. Haven’t tried to kill any processes yet; waiting to see if the zfs-discuss list has any data-gathering suggestions.

This older software version doesn’t support the -p option in zfs send, but that won’t be the cause of the hang; that will simply require me to recreate some key properties manually if I have to restore from backup.

ETA: My detailed report on the zfs-discuss mailing list.

It’s sitting at the same spot after work, after sitting all day.  Offiicially hung.

I wonder what it will take to stop the backup and export the pool?  Well, that’s nice; a straight “kill” terminated the processes, at least.

zpool status shows no errors. zfs list shows backup filesystems mounted.

zpool export -f is running…no disk I/O now…starting to look hung.

Ah, the zfs receive process is still in the process table.  kill -9 doesn’t help.

Kill and kill -9 won’t touch the zpool export process, either.

Pulling the USB cable on the drive doesn’t seem to be helping any either.

zfs list now hangs, but giving it a little longer just in case.

Kill -9 doesn’t touch any of the hung jobs.

Closing the ssh sessions doesn’t touch any of them either.

zfs list on pools other than bup-wrack works. zpool list works, and shows bup-wrack.

Attempting to set failmode=continue gives an I/O error.

Plugging the USB back in and then setting failmode gives the same I/O error.

cfgadm -al lists known disk drives and usb3/9 as “usb-storage connected”. I think that’s the USB disk that’s stuck.

cfgadm -cremove usb3/9 failed “configuration operation not supported”.

cfgadm -cdisconnect usb3/9 queried if I wanted to suspend activity, then failed with “cannot issue devctl to ap_id: /devices/pci@0,0/pci10de,cb84@2,1:9″

Still -al the same.

cfgadm -cunconfigure same error as disconnect.

I was able to list properties on bup-wrack:

bash-3.2$ zpool get all bup-wrack
NAME       PROPERTY       VALUE               SOURCE
bup-wrack  size           928G                -
bup-wrack  used           438G                -
bup-wrack  available      490G                -
bup-wrack  capacity       47%                 -
bup-wrack  altroot        /backups/bup-wrack  local
bup-wrack  health         UNAVAIL             -
bup-wrack  guid           2209605264342513453  default
bup-wrack  version        14                  default
bup-wrack  bootfs         -                   default
bup-wrack  delegation     on                  default
bup-wrack  autoreplace    off                 default
bup-wrack  cachefile      none                local
bup-wrack  failmode       wait                default
bup-wrack  listsnapshots  off                 default

It’s not healthy, alright. And the attempt to set failmode really did fail.

ETA: So I had to reboot.  However, that worked fine, and I recreated the pool, and I ran the same full backup script overnight, and it completed successfully.  Took 392:23, a bit over 6 hours, but it completed. (Something broke the ssh connection, but luckily I had run the backup under screen, so it just got detached and I could reconnect and see what happened. And it was making a log file, anyway.)

There’s a ‘cut’ version error in some of my after-backup processing that I’ll need to fix.

Meanwhile, yesterday the new SAS disk controller arrived. I’ve now got all the hardware to install into the box, and I can do that without changing anything associated with the current boot or data disks, so I probably will. Try things out on the new hardware and new disks, before cutting over.

Meanwhile, not sure why Wordpress is displaying the time in UTC.

ETA: Okay, install log is fairly clean.  There’s “device pciclass,030000@3(display#0) keeps up device sd@0,0(sd#0), but the latter is not power managed” and “/usr/lib/powerd: [ID387247 daemon.error] Able to open /dev/srn” and “SUNW_piclmemcfg init mc failed!”

Added Emacs.  Going to shutdown, snapshot, and start update from there.

ETA: On boot, several popup errors, and this in the logs:

Feb  5 07:47:08 osol-play-002 nwamd[37]: [ID 116842 daemon.error] sysevent_bind_handle: Permission denied
Feb  5 07:47:11 osol-play-002 genunix: [ID 127566 kern.info] device pciclass,030000@2(display#0) keeps up device sd@0,0(sd#0), but the former is not power managed
Feb  5 07:47:11 osol-play-002 /usr/lib/power/powerd: [ID 387247 daemon.error] Able to open /dev/srn
Feb  5 07:48:53 osol-play-002 inetd[6089]: [ID 702911 daemon.error] Failed to update state of instance svc:/application/x11/xfs:default in repository: entity not found
Feb  5 07:48:53 osol-play-002 inetd[6089]: [ID 702911 daemon.error] Failed to get instance for svc:/application/x11/xfs:default
Feb  5 07:48:53 osol-play-002 inetd[6089]: [ID 702911 daemon.error] Failed to update state of instance svc:/application/x11/xfs:default in repository: No such file or directory
Feb  5 07:48:53 osol-play-002 inetd[6089]: [ID 702911 daemon.error] Failed to update state of instance svc:/application/x11/xfs:default in repository: entity not found
Feb  5 07:48:53 osol-play-002 inetd[6089]: [ID 702911 daemon.error] Failed to get instance for svc:/application/x11/xfs:default
Feb  5 07:48:53 osol-play-002 inetd[6089]: [ID 702911 daemon.error] Failed to update state of instance svc:/application/x11/xfs:default in repository: No such file or directory
Feb  5 07:49:01 osol-play-002 ip: [ID 224711 kern.warning] WARNING: Memory pressure: TCP defensive mode on

But, unlike the previous attempt, the system did boot. I halted and snapshotted it immediately after the boot.

This makes software update key, not that it wasn’t already.

So I have had to re-install Virtualbox (because VMWare player won’t work with virtual machines on my network drive, whereas VirtualBox will), and reinstall Solaris. Then I will learn how to upgrade to various builds, because I’m ashamed to say I don’t know how except to “current”, which may not be the place to be.

Huh; almost looks like there isn’t a way.In future, I can update more often, and keep the old snapshots around. Though that doesn’t give any way to reinstall if what I really need is an old version.

I’m updating a virtual system, to test techniques and such. It’s downloading very slowly, equally slowly in bridged or NAT mode. So it’s not going to be done tonight, which means the testing will be delayed and the actual upgrade thus even more delayed. Well, things take time.

So far, knock on wood, nothing has gone terribly wrong.

ETA: The update (switching to the dev branch) completed overnight, with a number of errors. The new Boot Environment doesn’t come all the way up. No time to check more this morning.

Last night was step one (well, maybe step 30; counting all the consideration and research and ordering steps). I installed the memory upgrade, taking it from 1GB to 3GB (and filling all available memory banks), and then upgraded the swap space.

For what I’m afraid will be the last time in this series, I can report that it went completely smoothly.  I remembered how to open the case (on this one, you release the tabs holding the front on at the top, tilt the faceplate down, and that gives you access to the screws holding the side panels), I didn’t break anything while vacuuming out the dust, I figured out which way the memory sticks went in, and they went right in).

Background on This Server

I think this hardware dates to August 2006. It’s running Solaris using the ZFS filesystem (which is why it’s running Solaris).  It’s named “fsfs”, because it’s my redundant fileserver (all the data disks are mirrored).

Chenbro case

The hardware is a Chenbro case, actually a 4u rackmount with 8 hot-swap bays, but I’m using it as a tower. It’s very deep, since the bays in front don’t overlap the motherboard at all. ASUS M2N-SLI Deluxe motherboard with AMD dual-core processor. That motherboard has 6 SATA ports, so I’m not currently using  two of the hot-swap bays.

Two of the bays are for the system disks (mirrored).  The other 4 constitute the data pool.  Currently this is two mirrored pairs of 400GB disks (two of them free from Sun, when they gave up on using anything that small in the Streaming lab).

While the processor is old, it’s marvelously adequate for what it’s called on to do; and the case was the expensive part. I don’t want to mess with it, and certainly don’t want to buy another hotswap unit at this level. So I’m upgrading this one.

Upgrade Plan

It’s getting new boot drives (2.5″ disks in a new hot-swap bay; you can get a 4-drive 2.5″ hot-swap bay that mounts in a standard 5.25″ bay), a new disk controller to give me 8 more ports, and new boot disks (in the 2.5″ bays, freeing two of the main 3.5″ bays for data disks).

The current plan of operations goes like this:

1. Upgrade memory (completed)
2. Update backups
3. Update software
4. Update backups
5. Install 2.5″ hot-swap bay
6. Install new 2.5″ boot drives
7. Install new controller
8. Install software (or transfer existing installation) to new drives
9. Remove old boot drives
10. Install a new pair of drives in the data pool
11. Update backups

The new controller is to be the Supermicro UIO MegaRAID AOC-USAS-L8i. This needs weird expensive cable sets, but with the right ones, it’ll drive 8 SATA drives from something that looks like a SCSI controller to the computer (it’ll also drive SAS drives, handle port expanders, and generally do a lot of stuff, and Solaris supposedly has good drivers for it). This will be my first experience with SAS; we’ll see how exciting that is.

You’ll notice I’ll end up with 12 hot-swap bays (4 2.5″ and 8 3.5″). With 6 SATA ports on the motherboard, this means that I can split each mirrored pair across controllers, meaning that hardware failures and driver bugs can’t take out both sides of the mirror at once.  There are still plenty of other single points of failure, including power surges, higher level software bugs, user error, and so forth; that’s why we still need backups.

ZFS

ZFS is the “zettabyte filesystem”. It’s a combined volume manager and filesystem, engineered from scratch. And in fact it eliminates all sorts of annoyances, and adds some big features. It was created by a team at Sun Microsystems (now a division of Oracle)

For me, the three big features are:

• Pool expansion

• Data checksum and scrub

• Snapshots

This server  holds my photo archive, including scans of very old stuff. While I may not look at it very often, I do care about it.  I have backups, and can restore if necessary, but a precondition to that is realizing it’s damaged. I could continue making backups and rotating them through storage and not notice a picture got corrupted for years, by which time the last backup that included it would be gone.

ZFS stores a checksum for every block. An operation called “scrubbing” goes through all the blocks and verifies them against their stored checksums. By doing this, I can discover early that something is damaged. If the damage is a disk error, the redundant copy on the mirror drive will be automatically used to fix the problem (if its checksum is correct). If there isn’t a valid copy of the data online it can at least tell me which file is bad, early enough that I should still have a valid backup.  This makes me feel much safer. I think it even makes my data really somewhat safer.

Snapshots are great for keeping the last month or so of changes easily accessible online. Furthermore, I use snapshots on the backup disks too, so on only two backup drives I actually keep several years worth of change history.

Pool expansion lets me add more space without recopying everything. I’ve chosen to use mirrored pairs instead of RAID-5-style parity schemes (RAIDZ, it would be in ZFS). It lets me upgrade disks in smaller chunks (because in addition to adding a vdev to a pool, I can also replace the drives in an existing vdev with larger drives, and have that space become available). I don’t expect to ever have more than 8 hot-swap slots (well, 8 for the big drives I’m using for data), so working in small units, pairs, gives me more flexibility.  I rather expect to stop at 6 data disks (3 pair) and put in a seventh “hot spare” that ZFS can immediately copy data to if something happens to one of the live disks. Using 4-disk RAIDZ vdevs, I would get a higher proportion of the drive space available for use, but I’d have no space for hot spares, and I’d have to replace disks in groups of 4, and I’d have less redundancy and hence more exposure to data loss (luckily I do keep backups).

If Dr Lammers and Dr Galinsky are right, the sense which some powerful people seem to have that different rules apply to them is not just a convenient smoke screen. They genuinely believe it.