And I think I’m going to call it a win. The new disks are in and working. I’ve even got the regular snapshot script working pretty well.

Never did quite get the two new boot disks set up with identical partition sizes, but it doesn’t matter since I attached them both to the mirror (which was limited by the size of the old 80GB disks) first, and then detached the old disks.  At that point it expanded up to the available size, which was the smallest partition on the two new drives.  They differ by a MB or two out of 160GB, not important.

This is kind of an adjunct to part 4 of my “Server Upgrade Chronicles”.

ZFS root pools have some requirements and best-practices at variance to other ZFS pools. One of the most annoying is that you can’t use a whole disk, and you can’t use an EFI-labeled disk. This is annoying because for most ZFS uses using a whole disk is the best practice, and when you do that ZFS puts an EFI label on that disk.

So, when you try to use in a root pool a disk you’d previously used somewhere else in ZFS, you often see this:

bash-3.2$ pfexec zpool attach rpool c4t0d0s0 c9t0d0s0
cannot attach c9t0d0s0 to c4t0d0s0: EFI labeled devices are not supported
on root pools.

What do you do then? Well, you google, of course. And you find many sites explaining how to overwrite an EFI label on a disk. And every single one of them omits several things that seem to me to be key points (and which I had to play around with a lot to get any understanding of). The fact that ZFS is what drew me back into Solaris, and that I wasn’t ever really comfortable with their disk labeling scheme to begin with, is no doubt a contributing factor.

This is going to get long, so I’m putting in a cut here. (more…)

I got the two new system disks attached to the root ZFS pool and resilvered, so right now I’m running a 4-disk mirror for my root!  And I just booted off the #1 new disk, meaning that the Grub installation as well as the mirroring worked, and that the new controller really does support booting.

Actually, most of the excitement was earlier. In playing around with the new disks, I’d made them into a ZFS pool using the whole disk.  This put EFI labels on the disks, which Solaris / ZFS don’t support in a root pool.  So then I had to somehow get the disks relabeled and the partitions redrawn.  This turns out to be a horrible process which is not documented anywhere. The blogosphere is full of pages saying how to do it, and none of them actually tell you much.  Okay, use format -e, that’s helpful.  But they never say what device file to use, and none of the obvious ones exist.  I think you can maybe use any device pointed at the right disk for part of it. Also, I had to create an S0 manually, and I”m not sure I did it ideally (doesn’t matter much, since these disks are 4 times as big as they need to be).

I’m deeply confused by Solaris disk labeling, going back to SunOS days; even then, I thought it was absurd,  fact suicidally idiotic, to describe regions of the disk used for different filesystems which overlap. Okay, you’re not supposed to use any two that overlap for filesystems, but nothing stops you. The whole setup is just baroque, weird, stupid. And then, on x86 hardware, this Solaris idiocy takes place within one real partition (although Solaris documentation tends to call their things partitions).

So, I had to find a way to overwrite EFI labels with SMI labels. Apparently the secret is to use “format -e”. None of the pages said anything about manually creating partitions (or gave any clues for what space you could use; I believe you have to leave space at the start for the boot stuff). Anyway, totally infuriating partial documentation, and then a large group of aficionados giving slightly variant documentation with slight differences, all of it missing the key points.

Did I mention that I’m annoyed?

So I’m going to chase this for a while, until I get it actually figured out, or until I go postal; whichever comes first.

Very briefly, since it’s late!

I’ve temporarily given up on getting incremental send/receive working. I’m liking my results with full backups, and have some reason to believe that the next OpenSolaris release will fix the bug I’m seeing, and can’t see how to proceed on that in the short term. With a third backup drive, having just full backups isn’t so limiting, either.

I’ve got the 4x 2.5″ adapter installed in the 5.25″ bay, and the 8 port SATA controller hooked up to it, and two drives in it, and I’ve copied the system rpool to a new pool there called rp2. And am now looking at details of how to finish making that pool bootable, probably involving booting from a livecd to rename it among other things.

Weird problem with the SATA controller—the bracket on the end of the card doesn’t match the cutout in the case. I had to remove the bracket, leaving the card unsupported, which is clearly not viable in the long term with two sets of stiff cables attached to it (right now it’s supported with a bit of gaffer’s tape).

Haven’t looked into booting from the new controller, either; it’s possible I can’t, I suppose, but if so putting both boot disks on the old controller isn’t terribly painful, though it ruins my perfect plan to split every mirror pair across controllers.

There’s also a problem with the bottom left 2.5″ tray, but I’m ignoring that for now since I only need two drives in the 2.5″ bay to finish my upgrade.

Don’t know that it might not be better to install to the new drives from scratch, but there are issues duplicating the configuration down to UIDs and GIDs, which is necessary for the data pool to be accessible to the users when I import it.

Still, all the new hardware seems to be working, which is good.

Recreated wrack, the USB backup drive with the oldest data on it, using my updated scripts, and started a full backup, also with the updated scripts.  Was running fine when I went to bed.

Seems to be hung when I got up this morning, dammit.  System and pools are responsive, but there’s been no progress and no disk IO since I first checked when I got up. Haven’t tried to kill any processes yet; waiting to see if the zfs-discuss list has any data-gathering suggestions.

This older software version doesn’t support the -p option in zfs send, but that won’t be the cause of the hang; that will simply require me to recreate some key properties manually if I have to restore from backup.

ETA: My detailed report on the zfs-discuss mailing list.

It’s sitting at the same spot after work, after sitting all day.  Offiicially hung.

I wonder what it will take to stop the backup and export the pool?  Well, that’s nice; a straight “kill” terminated the processes, at least.

zpool status shows no errors. zfs list shows backup filesystems mounted.

zpool export -f is running…no disk I/O now…starting to look hung.

Ah, the zfs receive process is still in the process table.  kill -9 doesn’t help.

Kill and kill -9 won’t touch the zpool export process, either.

Pulling the USB cable on the drive doesn’t seem to be helping any either.

zfs list now hangs, but giving it a little longer just in case.

Kill -9 doesn’t touch any of the hung jobs.

Closing the ssh sessions doesn’t touch any of them either.

zfs list on pools other than bup-wrack works. zpool list works, and shows bup-wrack.

Attempting to set failmode=continue gives an I/O error.

Plugging the USB back in and then setting failmode gives the same I/O error.

cfgadm -al lists known disk drives and usb3/9 as “usb-storage connected”. I think that’s the USB disk that’s stuck.

cfgadm -cremove usb3/9 failed “configuration operation not supported”.

cfgadm -cdisconnect usb3/9 queried if I wanted to suspend activity, then failed with “cannot issue devctl to ap_id: /devices/pci@0,0/pci10de,cb84@2,1:9″

Still -al the same.

cfgadm -cunconfigure same error as disconnect.

I was able to list properties on bup-wrack:

bash-3.2$ zpool get all bup-wrack
NAME       PROPERTY       VALUE               SOURCE
bup-wrack  size           928G                -
bup-wrack  used           438G                -
bup-wrack  available      490G                -
bup-wrack  capacity       47%                 -
bup-wrack  altroot        /backups/bup-wrack  local
bup-wrack  health         UNAVAIL             -
bup-wrack  guid           2209605264342513453  default
bup-wrack  version        14                  default
bup-wrack  bootfs         -                   default
bup-wrack  delegation     on                  default
bup-wrack  autoreplace    off                 default
bup-wrack  cachefile      none                local
bup-wrack  failmode       wait                default
bup-wrack  listsnapshots  off                 default

It’s not healthy, alright. And the attempt to set failmode really did fail.

ETA: So I had to reboot.  However, that worked fine, and I recreated the pool, and I ran the same full backup script overnight, and it completed successfully.  Took 392:23, a bit over 6 hours, but it completed. (Something broke the ssh connection, but luckily I had run the backup under screen, so it just got detached and I could reconnect and see what happened. And it was making a log file, anyway.)

There’s a ‘cut’ version error in some of my after-backup processing that I’ll need to fix.