root@stefan-zfs-test1:~# zpool status pool: blacksite state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM blacksite ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 vda ONLINE 0 0 0 sdb ONLINE 0 0 0 errors: No known data errors Scenario 1 hot-remove of sdb device from stefan-zfs-test1 stefan-zfs-test1 writes data to A disk (vda) sdb (DISK B) is attached to stefan-zfs-test2 and force imported stefan-zfs-test2 writes data to B disk (sdb) hot-remove of sdb device from stefan-zfs-test2 hot-add sdb device on stefan-zfs-test1 hot-remove of sdb device from stefan-zfs-test1 sdb (DISK B) is attached to stefan-zfs-test2 and force imported hot-remove of sdb device from stefan-zfs-test2 Conclusion: When devices are hot-removed, imported elsewhere, hot-added again, then data from stefan-zfs-test1 is synced to device (dataloss off stefan-zfs-test2) [ 717.918421] sd 2:0:1:0: Power-on or device reset occurred [ 717.918866] sd 2:0:1:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB) [ 717.918911] sd 2:0:1:0: [sdb] Write Protect is off [ 717.918913] sd 2:0:1:0: [sdb] Mode Sense: 63 00 00 08 [ 717.918997] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 717.928619] sd 2:0:1:0: [sdb] Attached SCSI disk [ 746.996123] vda: vda1 vda9 [ 747.217957] sdb: sdb1 sdb9 [10802.876115] sd 2:0:1:0: [sdb] Synchronizing SCSI cache [10802.886783] sd 2:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK root@stefan-zfs-test1:~# zpool status pool: blacksite state: DEGRADED status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: none requested config: NAME STATE READ WRITE CKSUM blacksite DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 vda ONLINE 0 0 0 sdb OFFLINE 0 0 0 errors: No known data errors root@stefan-zfs-test1:~# echo DISKA > /blacksite/DISKA root@stefan-zfs-test1:~# sync root@stefan-zfs-test1:~# cat /blacksite/DISKA DISKA [10947.385903] scsi 2:0:1:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5 [10947.390933] scsi 2:0:1:0: Attached scsi generic sg2 type 0 [10947.391077] sd 2:0:1:0: Power-on or device reset occurred [10947.392936] sd 2:0:1:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB) [10947.392977] sd 2:0:1:0: [sdb] Write Protect is off [10947.392979] sd 2:0:1:0: [sdb] Mode Sense: 63 00 00 08 [10947.393068] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [10947.401687] sdb: sdb1 sdb9 [10947.402481] sd 2:0:1:0: [sdb] Attached SCSI disk root@stefan-zfs-test2:~# zpool import blacksite cannot import 'blacksite': pool was previously in use from another system. Last accessed by stefan-zfs-test1 (hostid=43695048) at Wed Nov 24 13:50:25 2021 The pool can be imported, use 'zpool import -f' to import the pool. root@stefan-zfs-test2:~# zpool import blacksite -f root@stefan-zfs-test2:~# zpool status pool: blacksite state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-4J scan: none requested config: NAME STATE READ WRITE CKSUM blacksite DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 6589980440734479215 UNAVAIL 0 0 0 was /dev/vda1 sdb ONLINE 0 0 0 errors: No known data error root@stefan-zfs-test2:~# echo DISKB > /blacksite/DISKB root@stefan-zfs-test2:~# sync root@stefan-zfs-test2:~# cat /blacksite/DISKB DISKB root@stefan-zfs-test1:~# zpool status pool: blacksite state: ONLINE scan: resilvered 367K in 0 days 00:00:00 with 0 errors on Wed Nov 24 14:29:46 2021 config: NAME STATE READ WRITE CKSUM blacksite ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 vda ONLINE 0 0 0 sdb ONLINE 0 0 0 errors: No known data errors [12909.125601] sd 2:0:1:0: Power-on or device reset occurred [12909.126929] sd 2:0:1:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB) [12909.126970] sd 2:0:1:0: [sdb] Write Protect is off [12909.126972] sd 2:0:1:0: [sdb] Mode Sense: 63 00 00 08 [12909.127037] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [12909.135246] sdb: sdb1 sdb9 [12909.144578] sd 2:0:1:0: [sdb] Attached SCSI disk [13375.928746] sd 2:0:1:0: [sdb] Synchronizing SCSI cache [13375.934974] sd 2:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [ 336.963691] sd 2:0:1:0: Power-on or device reset occurred [ 336.964118] sd 2:0:1:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB) [ 336.965197] sd 2:0:1:0: [sdb] Write Protect is off [ 336.965199] sd 2:0:1:0: [sdb] Mode Sense: 63 00 00 08 [ 336.965326] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 336.973477] sdb: sdb1 sdb9 [ 336.974256] sd 2:0:1:0: [sdb] Attached SCSI disk root@stefan-zfs-test2:~# zpool status pool: blacksite state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-4J scan: resilvered 367K in 0 days 00:00:00 with 0 errors on Wed Nov 24 14:29:46 2021 config: NAME STATE READ WRITE CKSUM blacksite DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 6589980440734479215 UNAVAIL 0 0 0 was /dev/vda1 sdb ONLINE 0 0 0 errors: No known data errors root@stefan-zfs-test2:~# cat /blacksite/DISKA DISKA Scenario 2 stefan-zfs-test2 writes data (DISKB) to B disk (sdb) stefan-zfs-test2 removes DISKA data from sdb hot-remove of vda device from stefan-zfs-test1 hot-add of vda device on stefan-zfs-test2 Conclusion: When devices are hot-removed, imported elsewhere, hot-added again, then data from stefan-zfs-test2 is synced to device (dataloss off stefan-zfs-test1) [ 992.649083] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003) [ 992.695035] virtio_blk virtio4: [vda] 104857600 512-byte logical blocks (53.7 GB/50.0 GiB) [ 992.706256] vda: vda1 vda root@stefan-zfs-test2:~# zpool status pool: blacksite state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-4J scan: resilvered 367K in 0 days 00:00:00 with 0 errors on Wed Nov 24 14:29:46 2021 config: NAME STATE READ WRITE CKSUM blacksite DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 6589980440734479215 UNAVAIL 0 0 0 was /dev/vda1 sdb ONLINE 0 0 0 errors: No known data errors root@stefan-zfs-test2:~# zpool online blacksite 6589980440734479215 root@stefan-zfs-test2:~# zpool status pool: blacksite state: ONLINE scan: resilvered 416K in 0 days 00:00:01 with 0 errors on Wed Nov 24 14:50:03 2021 config: NAME STATE READ WRITE CKSUM blacksite ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 vda ONLINE 0 0 0 sdb ONLINE 0 0 0 errors: No known data errors root@stefan-zfs-test2:~# cat /blacksite/DISKB DISKB Scenario 2 stefan-zfs-test1 writes data to /blacksite called tagA cold-remove of sdb device from stefan-zfs-test1 cold-attach of sdb device on stefan-zfs-test2 sdb (DISK B) is attached to stefan-zfs-test2 and force imported stefan-zfs-test2 writes data to /blacksite called tagB cold-remove of sdb device from stefan-zfs-test2 cold-attach of sdb device on stefan-zfs-test1 Conclusion: data loss on stefan-zfs-test1, stefan-zfs-test2 data prevails root@stefan-zfs-test2:~# zpool import blacksite cannot import 'blacksite': pool was previously in use from another system. Last accessed by stefan-zfs-test1 (hostid=43695048) at Thu Nov 25 07:16:26 2021 The pool can be imported, use 'zpool import -f' to import the pool. root@stefan-zfs-test2:~# zpool import blacksite -f cat /blacksite/tagA stefan-zfs-test1 Thu 25 Nov 2021 07:16:22 AM CET root@stefan-zfs-test2:~# rm /blacksite/tagA root@stefan-zfs-test2:~# echo stefan-zfs-test2 >> /blacksite/tagB root@stefan-zfs-test2:~# echo `date` >> /blacksite/tagB root@stefan-zfs-test2:~# cat /blacksite/tagB stefan-zfs-test2 Thu 25 Nov 2021 07:24:47 AM CET root@stefan-zfs-test1:~# zpool status pool: blacksite state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-9P scan: resilvered 454K in 0 days 00:00:01 with 0 errors on Thu Nov 25 07:28:26 2021 config: NAME STATE READ WRITE CKSUM blacksite ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 vda ONLINE 0 0 4 sdb ONLINE 0 0 0 errors: No known data errors root@stefan-zfs-test1:~# cat /blacksite/tagB stefan-zfs-test2 Thu 25 Nov 2021 07:24:47 AM CET Scenario 3 stefan-zfs-test1 writes data to /blacksite called tagA cold-remove of sdb device from stefan-zfs-test1 cold-attach of sdb device on stefan-zfs-test2 sdb (DISK B) is attached to stefan-zfs-test2 and force imported stefan-zfs-test2 writes data to /blacksite called tagB cold-remove of sdb device from stefan-zfs-test2 stefan-zfs-test1 boots and imports pool with a later timestamp than stefan-zfs-test2 stefan-zfs-test1 powered off cold-attach of sdb device on stefan-zfs-test1 Conclusion: data loss on stefan-zfs-test1, not sure why. Is an import of a pool not enough to mark this vdev as most recently used? Should some data be written? root@stefan-zfs-test2:~# zpool import blacksite -f root@stefan-zfs-test2:~# zpool status pool: blacksite state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-4J scan: none requested config: NAME STATE READ WRITE CKSUM blacksite DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 17844320233460051301 UNAVAIL 0 0 0 was /dev/vda1 sdb ONLINE 0 0 0 errors: No known data errors root@stefan-zfs-test2:~# cat /blacksite/tagA stefan-zfs-test1 root@stefan-zfs-test2:~# rm /blacksite/tagA root@stefan-zfs-test2:~# echo stefan-zfs-test2 >> /blacksite/tagB root@stefan-zfs-test2:~# echo `date` >> /blacksite/tagB root@stefan-zfs-test2:~# sync root@stefan-zfs-test2:~# cat /blacksite/tagB stefan-zfs-test2 Thu 25 Nov 2021 10:29:14 AM CET root@stefan-zfs-test1:~# zpool history blacksite History for 'blacksite': 2021-11-25.09:02:39 zpool create blacksite mirror /dev/vda /dev/sdb 2021-11-25.10:32:12 zpool import -c /etc/zfs/zpool.cache -aN root@stefan-zfs-test1:~# date Thu 25 Nov 2021 10:36:21 AM CET root@stefan-zfs-test1:~# zpool status pool: blacksite state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-9P scan: resilvered 498K in 0 days 00:00:00 with 0 errors on Thu Nov 25 10:40:04 2021 config: NAME STATE READ WRITE CKSUM blacksite ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 vda ONLINE 0 0 4 sdb ONLINE 0 0 0 errors: No known data errors root@stefan-zfs-test1:~# ls -lrt /blacksite/ total 4 drwxr-xr-x 16 root root 28 Nov 25 09:03 lib -rw-r--r-- 1 root root 49 Nov 25 10:29 tagB root@stefan-zfs-test1:~# cat /blacksite/tagB stefan-zfs-test2 Thu 25 Nov 2021 10:29:14 AM CET Scenario 4 stefan-zfs-test1 writes data to /blacksite called tagA cold-remove of sdb device from stefan-zfs-test1 cold-attach of sdb device on stefan-zfs-test2 sdb (DISK B) is attached to stefan-zfs-test2 and force imported stefan-zfs-test2 writes data to /blacksite called tagB cold-remove of sdb device from stefan-zfs-test2 stefan-zfs-test1 boots and imports pool with a later timestamp than stefan-zfs-test2 stefan-zfs-test1 writes data to /blacksite called tagB stefan-zfs-test1 powered off cold-attach of sdb device on stefan-zfs-test1 Conclusion: stefan-zfs-test2 data prevails. Why? root@stefan-zfs-test1:~# zpool create blacksite mirror /dev/vda /dev/sdb root@stefan-zfs-test1:~# cp -R /lib /blacksite/ root@stefan-zfs-test1:~# echo stefan-zfs-test1 >> /blacksite/tagA root@stefan-zfs-test1:~# echo `date` >> /blacksite/tagA root@stefan-zfs-test1:~# sync root@stefan-zfs-test2:~# zpool import blacksite -f root@stefan-zfs-test2:~# zpool status pool: blacksite state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-4J scan: none requested config: NAME STATE READ WRITE CKSUM blacksite DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 9489098380462644466 UNAVAIL 0 0 0 was /dev/vda1 sdb ONLINE 0 0 0 errors: No known data errors root@stefan-zfs-test2:~# cat /blacksite/tagA stefan-zfs-test1 Thu 25 Nov 2021 12:25:23 PM CET root@stefan-zfs-test2:~# rm /blacksite/tagA root@stefan-zfs-test2:~# echo stefan-zfs-test2 >> /blacksite/tagB root@stefan-zfs-test2:~# echo `date` >> /blacksite/tagB root@stefan-zfs-test2:~# sync root@stefan-zfs-test2:~# cat /blacksite/tagB stefan-zfs-test2 Thu 25 Nov 2021 12:30:00 PM CET root@stefan-zfs-test1:~# zpool status pool: blacksite state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-4J scan: none requested config: NAME STATE READ WRITE CKSUM blacksite DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 vda ONLINE 0 0 0 11555729381587750457 UNAVAIL 0 0 0 was /dev/sdb1 errors: No known data errors root@stefan-zfs-test1:~# cat /blacksite/tagA stefan-zfs-test1 Thu 25 Nov 2021 12:25:23 PM CET root@stefan-zfs-test1:~# echo `date` >> /blacksite/tagA root@stefan-zfs-test1:~# cat /blacksite/tagA stefan-zfs-test1 Thu 25 Nov 2021 12:25:23 PM CET Thu 25 Nov 2021 12:32:14 PM CET root@stefan-zfs-test1:~# zpool status pool: blacksite state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-9P scan: resilvered 470K in 0 days 00:00:01 with 0 errors on Thu Nov 25 12:34:05 2021 config: NAME STATE READ WRITE CKSUM blacksite ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 vda ONLINE 0 0 4 sdb ONLINE 0 0 0 errors: No known data errors root@stefan-zfs-test1:~# cat /blacksite/tagB stefan-zfs-test2 Thu 25 Nov 2021 12:30:00 PM CET