ZFS: i/o error – all block copies unavailable -zpool unavailable

A server restart produced this output on the console:

ZFS: i/o error - all block copies unavailable
ZFS: failed to read pool zroot directory object
qptzfsboot: failed to mount default pool zroot

FreeBSD/x86 boot
ZFS: i/o error - all block copies unavailable
ZFS: can't fild dataset 0
Default: zroot/<0x0>
boot:

I booted the host from the usb livecd and mounted the /etc directory under /tmp to enable ssh access:

ifconf -a # get available i/f names
ifconf em0 inet 192.168.216.46
route add default 192.168.216.1
hostname vhost06.internal
mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
echo 'PermitRootLogin yes' >> /etc/sshd_config
passwd
Changing local password for root
New Password:
Retype New Password:
service sshd onestart

There are no zfs pools available to import:

root@vhost06:~ # zpool status
no pools available
root@vhost06:~ # zpool list
no pools available
root@vhost06:~ # zfs list
no datasets available

gpart shows this geometry:

gpart show

=>         40  15628053088  ada0  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
         1064          984        - free -  (492K)
         2048     16777216     2  freebsd-swap  (8.0G)
     16779264  15611273216     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

=>         40  15628053088  ada1  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
         1064          984        - free -  (492K)
         2048     16777216     2  freebsd-swap  (8.0G)
     16779264  15611273216     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

=>         40  15628053088  ada2  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
         1064          984        - free -  (492K)
         2048     16777216     2  freebsd-swap  (8.0G)
     16779264  15611273216     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

=>         40  15628053088  ada3  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
         1064          984        - free -  (492K)
         2048     16777216     2  freebsd-swap  (8.0G)
     16779264  15611273216     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

=>         40  15628053088  diskid/DISK-VAGWJ6VL  GPT  (7.3T)
           40         1024                     1  freebsd-boot  (512K)
         1064          984                        - free -  (492K)
         2048     16777216                     2  freebsd-swap  (8.0G)
     16779264  15611273216                     3  freebsd-zfs  (7.3T)
  15628052480          648                        - free -  (324K)

=>         40  15628053088  diskid/DISK-VAGWV89L  GPT  (7.3T)
           40         1024                     1  freebsd-boot  (512K)
         1064          984                        - free -  (492K)
         2048     16777216                     2  freebsd-swap  (8.0G)
     16779264  15611273216                     3  freebsd-zfs  (7.3T)
  15628052480          648                        - free -  (324K)

=>         40  15628053088  diskid/DISK-VAHZAD2L  GPT  (7.3T)
           40         1024                     1  freebsd-boot  (512K)
         1064          984                        - free -  (492K)
         2048     16777216                     2  freebsd-swap  (8.0G)
     16779264  15611273216                     3  freebsd-zfs  (7.3T)
  15628052480          648                        - free -  (324K)

=>         40  15628053088  diskid/DISK-VAH3PXYL  GPT  (7.3T)
           40         1024                     1  freebsd-boot  (512K)
         1064          984                        - free -  (492K)
         2048     16777216                     2  freebsd-swap  (8.0G)
     16779264  15611273216                     3  freebsd-zfs  (7.3T)
  15628052480          648                        - free -  (324K)

=>       1  30240767  da0  MBR  (14G)
         1      1600    1  efi  (800K)
      1601   2012560    2  freebsd  [active]  (983M)
   2014161  28226607       - free -  (13G)

=>      0  2012560  da0s2  BSD  (983M)
        0       16         - free -  (8.0K)
       16  2012544      1  freebsd-ufs  (983M)

=>       1  30240767  diskid/DISK-00241D8CE51BB011B9A694C1  MBR  (14G)
         1      1600                                     1  efi  (800K)
      1601   2012560                                     2  freebsd  [active]  (983M)
   2014161  28226607                                        - free -  (13G)

=>      0  2012560  diskid/DISK-00241D8CE51BB011B9A694C1s2  BSD  (983M)
        0       16                                          - free -  (8.0K)
       16  2012544                                       1  freebsd-ufs  (983M)

How do I recover from here?

<—— end of original question

I have made some progress and managed to import and mount one root dataset – iocage. This is a boot on zfs system but I cannot find the dataset that contains the root filesystem so I cannot get at /var/log to see if anything is there:

mkdir /tmp/zroot                       # /tmp is a writable file system
zpool -f zroot                         # force the zpool import
zfs set mountpoint=/tmp/zroot  zroot   # mount the imported pool in a writable fs
zfs mount -a                           # find and mount all the datasets
ll /tmp/zroot
total 12
drwxr-xr-x  9 root  wheel  11 Feb 27 13:09 iocage/

Fortunately, all of the absolutely critical stuff is in /zroot/iocage as the host simply acts as a platform for the jail. However the absence of the root dataset is bothersome to me.

A zpool status showed zroot with no errors.

I next transferred the iocage dataset to another system using zfs send.

zfs snapshot -r zroot/iocage@vh6iocsend1
zfs send -R zroot/iocage@vh6iocsend1 | ssh 192.168.216.45 zfs receive zroot/iocagev6

This took a while but it has completed successfully.

Now I need to get the problem host started. This host was restarted yesterday at noon without a problem. I do not recall running freebsd-update fetch, but even if I had there was nothing for fetch to deliver as the system was already at 12.1p2

I still need help getting the host to boot.

<———-

Additional notes:

I was able to mount the entire zpool using the altroot option of zpool import:

  1. Boot into the live cd shell.
  2. Import the zfs pool(s) but do not allow import to auto mount any datasets: zpool import -o altroot=/tmp/altroot -N -a.
  3. Mount the root / dataset first: zfs mount zroot/ROOT/default.
  4. Now mount the remaining datasets: zfs mount -a.

The entire zroot pool’s file system is now accessible at /tmp/altroot.

I have used this with zfs send to move the contents of /var to another host. Actually, I sent the entire pool.

However, the original system still will not boot.

Answer

As a last resort I pulled the four hdds from the host that failed to boot and placed them in an identically configured server. That server booted from those hdds. Evidently a hardware problem.

There is still an anomaly on the new host: if bay 1 is occupied then the system will only boot if at least one of bay 0 or bay 2 is likewise occupied. The configuration 0-,1A,2-,3B will not boot. I do not have a clue as to why this is so.

Attribution
Source : Link , Question Author : James B. Byrne , Answer Author : James B. Byrne

Leave a Comment