This is kind of an adjunct to part 4 of my “Server Upgrade Chronicles”.
ZFS root pools have some requirements and best-practices at variance to other ZFS pools. One of the most annoying is that you can’t use a whole disk, and you can’t use an EFI-labeled disk. This is annoying because for most ZFS uses using a whole disk is the best practice, and when you do that ZFS puts an EFI label on that disk.
So, when you try to use in a root pool a disk you’d previously used somewhere else in ZFS, you often see this:
bash-3.2$ pfexec zpool attach rpool c4t0d0s0 c9t0d0s0 cannot attach c9t0d0s0 to c4t0d0s0: EFI labeled devices are not supported on root pools.
What do you do then? Well, you google, of course. And you find many sites explaining how to overwrite an EFI label on a disk. And every single one of them omits several things that seem to me to be key points (and which I had to play around with a lot to get any understanding of). The fact that ZFS is what drew me back into Solaris, and that I wasn’t ever really comfortable with their disk labeling scheme to begin with, is no doubt a contributing factor.
This is going to get long, so I’m putting in a cut here.
And I’ve now found, in the relatively obvious place (but it doesn’t show up in the google searches I tried) the good and fairly official article on how to do this. So maybe you should skip me and go right there. They’ve got a couple of neater ways of doing things than I did.
The first point, which comes fairly easily from the google hits, is that I need to use “format -e”, expert mode. That much was useful.
bash-3.2$ pfexec format -e Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c4t0d0 <DEFAULT cyl 9726 alt 2 hd 255 sec 63> /pci@0,0/pci10de,cb84@5/disk@0,0 1. c4t1d0 <ATA-ST380811AS-E cyl 9726 alt 2 hd 255 sec 63> /pci@0,0/pci10de,cb84@5/disk@1,0 2. c5t0d0 <ATA-HITACHI HDS7240S-A47A-372.61GB> /pci@0,0/pci10de,cb84@5,1/disk@0,0 3. c5t1d0 <ATA-ST3400620AS-C-372.61GB> /pci@0,0/pci10de,cb84@5,1/disk@1,0 4. c6t0d0 <ATA-HITACHI HDS7240S-A47A-372.61GB> /pci@0,0/pci10de,cb84@5,2/disk@0,0 5. c6t1d0 <ATA-HITACHI HDS7240S-A47A-372.61GB> /pci@0,0/pci10de,cb84@5,2/disk@1,0 6. c9t0d0 <ATA-WDC WD1600BEKT-6-1A13-149.05GB> /pci@0,0/pci10de,376@a/pci15d9,a580@0/sd@0,0 7. c9t2d0 <ATA-WDCWD1600BEKT-6-1A13 cyl 21543 alt 2 hd 16 sec 128> /pci@0,0/pci10de,376@a/pci15d9,a580@0/sd@2,0 Specify disk (enter its number): 6 selecting c9t0d0 [disk formatted]
Note that c9t0d0 (#6) has a variant label string compared to the other identical drive (#7). We’ll fix that while we’re at it.
Next we have to delete the existing EFI label using fdisk.
format> fdisk Total disk size is 19457 cylinders Cylinder size is 16065 (512 byte) blocks Cylinders Partition  Status   Type         Start  End  Length   % =========  ======   ============ =====  ===  ======  === 1                EFI              0 19457   19458   100 SELECT ONE OF THE FOLLOWING: 1. Create a partition 2. Specify the active partition 3. Delete a partition 4. Change between Solaris and Solaris2 Partition IDs 5. Exit (update disk configuration and exit) 6. Cancel (exit without updating disk configuration) Enter Selection: 3 Specify the partition number to delete (or enter 0 to exit): 1 Are you sure you want to delete partition 1? This will make all files and programs in this partition inaccessible (type "y" or "n"). y Partition 1 has been deleted.             Total disk size is 19457 cylinders Cylinder size is 16065 (512 byte) blocks Cylinders Partition  Status   Type         Start  End  Length   % =========  ======   ============ =====  ===  ======  === WARNING: no partitions are defined! SELECT ONE OF THE FOLLOWING: 1. Create a partition 2. Specify the active partition 3. Delete a partition 4. Change between Solaris and Solaris2 Partition IDs 5. Exit (update disk configuration and exit) 6. Cancel (exit without updating disk configuration) Enter Selection: 1 Select the partition type to create: 1=SOLARIS2 2=UNIX       3=PCIXOS    4=Other 5=DOS12    6=DOS16      7=DOSEXT    8=DOSBIG 9=DOS16LBA A=x86 Boot   B=Diagnostic C=FAT32 D=FAT32LBA E=DOSEXTLBA  F=EFI       0=Exit? 1 Specify the percentage of disk to use for this partition (or type "c" to specify the size in cylinders). 100 Should this become the active partition? If yes, it will be activated each time the computer is reset or turned on. Please type "y" or "n". y Partition 1 is now the active partition. Total disk size is 19457 cylinders Cylinder size is 16065 (512 byte) blocks Cylinders Partition  Status   Type         Start  End  Length   % =========  ======   ============ =====  ===  ======  === 1      Active   Solaris2         1 19456   19456   100 SELECT ONE OF THE FOLLOWING: 1. Create a partition 2. Specify the active partition 3. Delete a partition 4. Change between Solaris and Solaris2 Partition IDs 5. Exit (update disk configuration and exit) 6. Cancel (exit without updating disk configuration) Enter Selection: 5 format>
Sorry that’s a bit long. It’s also edited; fdisk uses cursor positioning to update the screen in place, and I’ve hacked through that in an attempt to present it as a readable transcript. Linespacing has been extensively altered. But it shows us having successfully removed the EFI label and put on a new label.
However, we still don’t have quite the right partitions for a ZFS root pool.
format> p PARTITION MENU: 0     - change `0' partition 1     - change `1' partition 2     - change `2' partition 3     - change `3' partition 4     - change `4' partition 5     - change `5' partition 6     - change `6' partition 7     - change `7' partition 8     - change '8' partition 9     - change `9' partition expand - expand label to use whole disk select - select a predefined table modify - modify a predefined partition table name  - name the current table print - display the current table label - write partition map and label to the disk !<cmd> - execute <cmd>, then return quit partition> p Current partition table (default): Total disk sectors available: 312565390 + 16384 (reserved sectors) Part     Tag   Flag    First Sector        Size        Last Sector 0       usr   wm               34     149.04GB         312565390 1 unassigned   wm                0          0              0 2 unassigned   wm                0          0              0 3 unassigned   wm                0          0              0 4 unassigned   wm                0          0              0 5 unassigned   wm                0          0              0 6 unassigned   wm                0          0              0 7 unassigned   wm                0          0              0 8  reserved   wm        312565391       8.00MB         312581774 partition> l [0] SMI Label [1] EFI Label Specify Label type[1]: 0 Auto configuration via format.dat[no]? Auto configuration via generic SCSI-2[no]? yes Warning: no backup labels partition> p Current partition table (default): Total disk cylinders available: 152615 + 2 (reserved cylinders) Part     Tag   Flag    Cylinders         Size           Blocks 0      root   wm      0                0        (0/0/0)             0 1      swap   wu      0                0        (0/0/0)             0 2    backup   wu      0 - 152616     149.04GB   (152617/0/0) 312559616 3 unassigned   wm      0                0        (0/0/0)             0 4 unassigned   wm      0                0        (0/0/0)             0 5 unassigned   wm      0                0        (0/0/0)             0 6       usr   wm      1 - 152614     149.04GB   (152614/0/0) 312553472 7 unassigned   wm      0                0        (0/0/0)             0 8      boot   wu      0 -     0       1.00MB   (1/0/0)          2048 9 alternates   wm      0                0        (0/0/0)             0 format> type AVAILABLE DRIVE TYPES: 0. Auto configure 1. DEFAULT 2. ATA-ST380811AS-E 3. ATA-WDCWD1600BEKT-6-1A13 4. other Specify disk type (enter its number)[3]: selecting c9t0d0 [disk formatted] format> p PARTITION MENU: 0     - change `0' partition 1     - change `1' partition 2     - change `2' partition 3     - change `3' partition 4     - change `4' partition 5     - change `5' partition 6     - change `6' partition 7     - change `7' partition 9     - change `9' partition select - select a predefined table modify - modify a predefined partition table name  - name the current table print - display the current table label - write partition map and label to the disk !<cmd> - execute <cmd>, then return quit partition> p Current partition table (original): Total disk cylinders available: 152615 + 2 (reserved cylinders) Part     Tag   Flag    Cylinders         Size           Blocks 0      root   wm      1 - 152614     149.04GB   (152614/0/0) 312553472 1      swap   wu      0                0        (0/0/0)             0 2 unassigned   wu      0                0        (0/0/0)             0 3 unassigned   wm      0                0        (0/0/0)             0 4 unassigned   wm      0                0        (0/0/0)             0 5 unassigned   wm      0                0        (0/0/0)             0 6 unassigned   wm      0                0        (0/0/0)             0 7 unassigned   wm      0                0        (0/0/0)             0 8      boot   wu      0 -     0       1.00MB   (1/0/0)          2048 9 alternates   wm      0                0        (0/0/0)             0
Turns out there’s a reason I can’t just delete the overlapping s2 slice:
bash-3.2$ pfexec /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c9t0d0s0 cannot open/stat device /dev/rdsk/c9t0d0s2
I have no idea what it’s doing; maybe assuming that s2 is the whole disk?
So I have to go back and recreate s2.
partition> 2 Part     Tag   Flag    Cylinders         Size           Blocks 2 unassigned   wu      0                0        (0/0/0)             0 Enter partition id tag[unassigned]: back Enter partition permission flags[wu]: Enter new starting cyl[0]: Enter partition size[0b, 0c, 0e, 0.00mb, 0.00gb]: ? Expecting up to 312555520 blocks, 152615 cylinders, 152615 end cylinder, 15261\ 5.00 megabytes, or 149.04 gigabytes Enter partition size[0b, 0c, 0e, 0.00mb, 0.00gb]: 152615c partition> p Current partition table (unnamed): Total disk cylinders available: 152615 + 2 (reserved cylinders) Part     Tag   Flag    Cylinders         Size           Blocks 0      root   wm      1 - 152614     149.04GB   (152614/0/0) 312553472 1      swap   wu      0                0        (0/0/0)             0 2    backup   wu      0 - 152614     149.04GB   (152615/0/0) 312555520 3 unassigned   wm      0                0        (0/0/0)             0 4 unassigned   wm      0                0        (0/0/0)             0 5 unassigned   wm      0                0        (0/0/0)             0 6 unassigned   wm      0                0        (0/0/0)             0 7 unassigned   wm      0                0        (0/0/0)             0 8      boot   wu      0 -     0       1.00MB   (1/0/0)          2048 9 alternates   wm      0                0        (0/0/0)             0 partition> l [0] SMI Label [1] EFI Label Specify Label type[0]: 0 Ready to label disk, continue? y Warning: no backup labels partition> q FORMAT MENU: [long menu snipped] format> q bash-3.2$
And that was in fact that.
bash-3.2$ pfexec zpool attach rpool c4t0d0s0 c9t0d0s0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c9t0d0s0 overlaps with /dev/dsk/c9t0d0s2 bash-3.2$ pfexec zpool attach rpool c4t0d0s0 c9t0d0s0 Please be sure to invoke installgrub(1M) to make 'c9t0d0s0' bootable. bash-3.2$ pfexec /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c9t0d0s0 stage1 written to partition 0 sector 0 (abs 16065) stage2 written to partition 0, 271 sectors starting at 50 (abs 16115) bash-3.2$ zpool status rpool pool: rpool state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress for 0h0m, 23.08% done, 0h1m to go config: NAME         STATE    READ WRITE CKSUM rpool        ONLINE      0    0    0 mirror     ONLINE      0    0    0 c4t0d0s0 ONLINE      0    0    0 c4t1d0s0 ONLINE      0    0    0 c9t2d0s0 ONLINE      0    0    0 c9t0d0s0 ONLINE      0    0    0 1.47G resilvered errors: No known data errors bash-3.2$
Oh, we should check the partitioning directly:
bash-3.2$ pfexec prtvtoc /dev/dsk/c9t0d0s0 * /dev/dsk/c9t0d0s0 partition map * * Dimensions: *    512 bytes/sector *    128 sectors/track *     16 tracks/cylinder *   2048 sectors/cylinder *  21545 cylinders *  21543 accessible cylinders * * Flags: *  1: unmountable * 10: read-only * * Unallocated space: *      First    Sector   Last *      Sector    Count   Sector *          0     2048     2047 * *                         First    Sector   Last * Partition Tag Flags   Sector    Count   Sector Mount Directory 0     2   00      2048 312553472 312555519 2     5   01         0 312555520 312555519 8     1   01         0     2048     2047 bash-3.2$ pfexec prtvtoc /dev/dsk/c9t2d0s0 * /dev/dsk/c9t2d0s0 partition map * * Dimensions: *    512 bytes/sector *    128 sectors/track *     16 tracks/cylinder *   2048 sectors/cylinder *  21545 cylinders *  21543 accessible cylinders * * Flags: *  1: unmountable * 10: read-only * * Unallocated space: *      First    Sector   Last *      Sector    Count   Sector *          0     2048     2047 * *                         First    Sector   Last * Partition Tag Flags   Sector    Count   Sector Mount Directory 0     2   00      2048 312553472 312555519 2     5   01         0 312559616 312559615 6     4   00      2048 312553472 312555519 8     1   01         0     2048     2047 bash-3.2$
I’m going to call this a success, for now.