root@stefan-zfs-test1:~# zpool status
pool: blacksite
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
blacksite ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
vda ONLINE 0 0 0
sdb ONLINE 0 0 0
errors: No known data errors
Scenario 1
hot-remove of sdb device from stefan-zfs-test1
stefan-zfs-test1 writes data to A disk (vda)
sdb (DISK B) is attached to stefan-zfs-test2 and force imported
stefan-zfs-test2 writes data to B disk (sdb)
hot-remove of sdb device from stefan-zfs-test2
hot-add sdb device on stefan-zfs-test1
hot-remove of sdb device from stefan-zfs-test1
sdb (DISK B) is attached to stefan-zfs-test2 and force imported
hot-remove of sdb device from stefan-zfs-test2
Conclusion: When devices are hot-removed, imported elsewhere, hot-added again, then data from stefan-zfs-test1 is synced to device (dataloss off stefan-zfs-test2)
[ 717.918421] sd 2:0:1:0: Power-on or device reset occurred
[ 717.918866] sd 2:0:1:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB)
[ 717.918911] sd 2:0:1:0: [sdb] Write Protect is off
[ 717.918913] sd 2:0:1:0: [sdb] Mode Sense: 63 00 00 08
[ 717.918997] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 717.928619] sd 2:0:1:0: [sdb] Attached SCSI disk
[ 746.996123] vda: vda1 vda9
[ 747.217957] sdb: sdb1 sdb9
[10802.876115] sd 2:0:1:0: [sdb] Synchronizing SCSI cache
[10802.886783] sd 2:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
root@stefan-zfs-test1:~# zpool status
pool: blacksite
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: none requested
config:
NAME STATE READ WRITE CKSUM
blacksite DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
vda ONLINE 0 0 0
sdb OFFLINE 0 0 0
errors: No known data errors
root@stefan-zfs-test1:~# echo DISKA > /blacksite/DISKA
root@stefan-zfs-test1:~# sync
root@stefan-zfs-test1:~# cat /blacksite/DISKA
DISKA
[10947.385903] scsi 2:0:1:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5
[10947.390933] scsi 2:0:1:0: Attached scsi generic sg2 type 0
[10947.391077] sd 2:0:1:0: Power-on or device reset occurred
[10947.392936] sd 2:0:1:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB)
[10947.392977] sd 2:0:1:0: [sdb] Write Protect is off
[10947.392979] sd 2:0:1:0: [sdb] Mode Sense: 63 00 00 08
[10947.393068] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[10947.401687] sdb: sdb1 sdb9
[10947.402481] sd 2:0:1:0: [sdb] Attached SCSI disk
root@stefan-zfs-test2:~# zpool import blacksite
cannot import 'blacksite': pool was previously in use from another system.
Last accessed by stefan-zfs-test1 (hostid=43695048) at Wed Nov 24 13:50:25 2021
The pool can be imported, use 'zpool import -f' to import the pool.
root@stefan-zfs-test2:~# zpool import blacksite -f
root@stefan-zfs-test2:~# zpool status
pool: blacksite
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: none requested
config:
NAME STATE READ WRITE CKSUM
blacksite DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
6589980440734479215 UNAVAIL 0 0 0 was /dev/vda1
sdb ONLINE 0 0 0
errors: No known data error
root@stefan-zfs-test2:~# echo DISKB > /blacksite/DISKB
root@stefan-zfs-test2:~# sync
root@stefan-zfs-test2:~# cat /blacksite/DISKB
DISKB
root@stefan-zfs-test1:~# zpool status
pool: blacksite
state: ONLINE
scan: resilvered 367K in 0 days 00:00:00 with 0 errors on Wed Nov 24 14:29:46 2021
config:
NAME STATE READ WRITE CKSUM
blacksite ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
vda ONLINE 0 0 0
sdb ONLINE 0 0 0
errors: No known data errors
[12909.125601] sd 2:0:1:0: Power-on or device reset occurred
[12909.126929] sd 2:0:1:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB)
[12909.126970] sd 2:0:1:0: [sdb] Write Protect is off
[12909.126972] sd 2:0:1:0: [sdb] Mode Sense: 63 00 00 08
[12909.127037] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[12909.135246] sdb: sdb1 sdb9
[12909.144578] sd 2:0:1:0: [sdb] Attached SCSI disk
[13375.928746] sd 2:0:1:0: [sdb] Synchronizing SCSI cache
[13375.934974] sd 2:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[ 336.963691] sd 2:0:1:0: Power-on or device reset occurred
[ 336.964118] sd 2:0:1:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB)
[ 336.965197] sd 2:0:1:0: [sdb] Write Protect is off
[ 336.965199] sd 2:0:1:0: [sdb] Mode Sense: 63 00 00 08
[ 336.965326] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 336.973477] sdb: sdb1 sdb9
[ 336.974256] sd 2:0:1:0: [sdb] Attached SCSI disk
root@stefan-zfs-test2:~# zpool status
pool: blacksite
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: resilvered 367K in 0 days 00:00:00 with 0 errors on Wed Nov 24 14:29:46 2021
config:
NAME STATE READ WRITE CKSUM
blacksite DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
6589980440734479215 UNAVAIL 0 0 0 was /dev/vda1
sdb ONLINE 0 0 0
errors: No known data errors
root@stefan-zfs-test2:~# cat /blacksite/DISKA
DISKA
Scenario 2
stefan-zfs-test2 writes data (DISKB) to B disk (sdb)
stefan-zfs-test2 removes DISKA data from sdb
hot-remove of vda device from stefan-zfs-test1
hot-add of vda device on stefan-zfs-test2
Conclusion: When devices are hot-removed, imported elsewhere, hot-added again, then data from stefan-zfs-test2 is synced to device (dataloss off stefan-zfs-test1)
[ 992.649083] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)
[ 992.695035] virtio_blk virtio4: [vda] 104857600 512-byte logical blocks (53.7 GB/50.0 GiB)
[ 992.706256] vda: vda1 vda
root@stefan-zfs-test2:~# zpool status
pool: blacksite
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: resilvered 367K in 0 days 00:00:00 with 0 errors on Wed Nov 24 14:29:46 2021
config:
NAME STATE READ WRITE CKSUM
blacksite DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
6589980440734479215 UNAVAIL 0 0 0 was /dev/vda1
sdb ONLINE 0 0 0
errors: No known data errors
root@stefan-zfs-test2:~# zpool online blacksite 6589980440734479215
root@stefan-zfs-test2:~# zpool status
pool: blacksite
state: ONLINE
scan: resilvered 416K in 0 days 00:00:01 with 0 errors on Wed Nov 24 14:50:03 2021
config:
NAME STATE READ WRITE CKSUM
blacksite ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
vda ONLINE 0 0 0
sdb ONLINE 0 0 0
errors: No known data errors
root@stefan-zfs-test2:~# cat /blacksite/DISKB
DISKB
Scenario 2
stefan-zfs-test1 writes data to /blacksite called tagA
cold-remove of sdb device from stefan-zfs-test1
cold-attach of sdb device on stefan-zfs-test2
sdb (DISK B) is attached to stefan-zfs-test2 and force imported
stefan-zfs-test2 writes data to /blacksite called tagB
cold-remove of sdb device from stefan-zfs-test2
cold-attach of sdb device on stefan-zfs-test1
Conclusion: data loss on stefan-zfs-test1, stefan-zfs-test2 data prevails
root@stefan-zfs-test2:~# zpool import blacksite
cannot import 'blacksite': pool was previously in use from another system.
Last accessed by stefan-zfs-test1 (hostid=43695048) at Thu Nov 25 07:16:26 2021
The pool can be imported, use 'zpool import -f' to import the pool.
root@stefan-zfs-test2:~# zpool import blacksite -f
cat /blacksite/tagA
stefan-zfs-test1
Thu 25 Nov 2021 07:16:22 AM CET
root@stefan-zfs-test2:~# rm /blacksite/tagA
root@stefan-zfs-test2:~# echo stefan-zfs-test2 >> /blacksite/tagB
root@stefan-zfs-test2:~# echo `date` >> /blacksite/tagB
root@stefan-zfs-test2:~# cat /blacksite/tagB
stefan-zfs-test2
Thu 25 Nov 2021 07:24:47 AM CET
root@stefan-zfs-test1:~# zpool status
pool: blacksite
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-9P
scan: resilvered 454K in 0 days 00:00:01 with 0 errors on Thu Nov 25 07:28:26 2021
config:
NAME STATE READ WRITE CKSUM
blacksite ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
vda ONLINE 0 0 4
sdb ONLINE 0 0 0
errors: No known data errors
root@stefan-zfs-test1:~# cat /blacksite/tagB
stefan-zfs-test2
Thu 25 Nov 2021 07:24:47 AM CET
Scenario 3
stefan-zfs-test1 writes data to /blacksite called tagA
cold-remove of sdb device from stefan-zfs-test1
cold-attach of sdb device on stefan-zfs-test2
sdb (DISK B) is attached to stefan-zfs-test2 and force imported
stefan-zfs-test2 writes data to /blacksite called tagB
cold-remove of sdb device from stefan-zfs-test2
stefan-zfs-test1 boots and imports pool with a later timestamp than stefan-zfs-test2
stefan-zfs-test1 powered off
cold-attach of sdb device on stefan-zfs-test1
Conclusion: data loss on stefan-zfs-test1, not sure why. Is an import of a pool not enough to mark this vdev as most recently used? Should some data be written?
root@stefan-zfs-test2:~# zpool import blacksite -f
root@stefan-zfs-test2:~# zpool status
pool: blacksite
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: none requested
config:
NAME STATE READ WRITE CKSUM
blacksite DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
17844320233460051301 UNAVAIL 0 0 0 was /dev/vda1
sdb ONLINE 0 0 0
errors: No known data errors
root@stefan-zfs-test2:~# cat /blacksite/tagA
stefan-zfs-test1
root@stefan-zfs-test2:~# rm /blacksite/tagA
root@stefan-zfs-test2:~# echo stefan-zfs-test2 >> /blacksite/tagB
root@stefan-zfs-test2:~# echo `date` >> /blacksite/tagB
root@stefan-zfs-test2:~# sync
root@stefan-zfs-test2:~# cat /blacksite/tagB
stefan-zfs-test2
Thu 25 Nov 2021 10:29:14 AM CET
root@stefan-zfs-test1:~# zpool history blacksite
History for 'blacksite':
2021-11-25.09:02:39 zpool create blacksite mirror /dev/vda /dev/sdb
2021-11-25.10:32:12 zpool import -c /etc/zfs/zpool.cache -aN
root@stefan-zfs-test1:~# date
Thu 25 Nov 2021 10:36:21 AM CET
root@stefan-zfs-test1:~# zpool status
pool: blacksite
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-9P
scan: resilvered 498K in 0 days 00:00:00 with 0 errors on Thu Nov 25 10:40:04 2021
config:
NAME STATE READ WRITE CKSUM
blacksite ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
vda ONLINE 0 0 4
sdb ONLINE 0 0 0
errors: No known data errors
root@stefan-zfs-test1:~# ls -lrt /blacksite/
total 4
drwxr-xr-x 16 root root 28 Nov 25 09:03 lib
-rw-r--r-- 1 root root 49 Nov 25 10:29 tagB
root@stefan-zfs-test1:~# cat /blacksite/tagB
stefan-zfs-test2
Thu 25 Nov 2021 10:29:14 AM CET
Scenario 4
stefan-zfs-test1 writes data to /blacksite called tagA
cold-remove of sdb device from stefan-zfs-test1
cold-attach of sdb device on stefan-zfs-test2
sdb (DISK B) is attached to stefan-zfs-test2 and force imported
stefan-zfs-test2 writes data to /blacksite called tagB
cold-remove of sdb device from stefan-zfs-test2
stefan-zfs-test1 boots and imports pool with a later timestamp than stefan-zfs-test2
stefan-zfs-test1 writes data to /blacksite called tagB
stefan-zfs-test1 powered off
cold-attach of sdb device on stefan-zfs-test1
Conclusion: stefan-zfs-test2 data prevails. Why?
root@stefan-zfs-test1:~# zpool create blacksite mirror /dev/vda /dev/sdb
root@stefan-zfs-test1:~# cp -R /lib /blacksite/
root@stefan-zfs-test1:~# echo stefan-zfs-test1 >> /blacksite/tagA
root@stefan-zfs-test1:~# echo `date` >> /blacksite/tagA
root@stefan-zfs-test1:~# sync
root@stefan-zfs-test2:~# zpool import blacksite -f
root@stefan-zfs-test2:~# zpool status
pool: blacksite
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: none requested
config:
NAME STATE READ WRITE CKSUM
blacksite DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
9489098380462644466 UNAVAIL 0 0 0 was /dev/vda1
sdb ONLINE 0 0 0
errors: No known data errors
root@stefan-zfs-test2:~# cat /blacksite/tagA
stefan-zfs-test1
Thu 25 Nov 2021 12:25:23 PM CET
root@stefan-zfs-test2:~# rm /blacksite/tagA
root@stefan-zfs-test2:~# echo stefan-zfs-test2 >> /blacksite/tagB
root@stefan-zfs-test2:~# echo `date` >> /blacksite/tagB
root@stefan-zfs-test2:~# sync
root@stefan-zfs-test2:~# cat /blacksite/tagB
stefan-zfs-test2
Thu 25 Nov 2021 12:30:00 PM CET
root@stefan-zfs-test1:~# zpool status
pool: blacksite
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: none requested
config:
NAME STATE READ WRITE CKSUM
blacksite DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
vda ONLINE 0 0 0
11555729381587750457 UNAVAIL 0 0 0 was /dev/sdb1
errors: No known data errors
root@stefan-zfs-test1:~# cat /blacksite/tagA
stefan-zfs-test1
Thu 25 Nov 2021 12:25:23 PM CET
root@stefan-zfs-test1:~# echo `date` >> /blacksite/tagA
root@stefan-zfs-test1:~# cat /blacksite/tagA
stefan-zfs-test1
Thu 25 Nov 2021 12:25:23 PM CET
Thu 25 Nov 2021 12:32:14 PM CET
root@stefan-zfs-test1:~# zpool status
pool: blacksite
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-9P
scan: resilvered 470K in 0 days 00:00:01 with 0 errors on Thu Nov 25 12:34:05 2021
config:
NAME STATE READ WRITE CKSUM
blacksite ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
vda ONLINE 0 0 4
sdb ONLINE 0 0 0
errors: No known data errors
root@stefan-zfs-test1:~# cat /blacksite/tagB
stefan-zfs-test2
Thu 25 Nov 2021 12:30:00 PM CET