Linux ZFS Notes

1. Reference Docs

Ubuntu

Ubuntu Source Doc

Fedora

Fedora Source Doc.

Linux

Cheat Sheet

2. Installation

2.1. Ubuntu

Install

sudo apt update && sudo apt install -y zfsutils-linux

Confirm

zfs --version

Example

zfs-2.4.1-1ubuntu5
zfs-kmod-2.4.1-1ubuntu5

Upgrade
```
zpool upgrade -v
```

2.2. Rocky

Import the ZFS repository for Rocky Linux

sudo dnf -y install https://zfsonlinux.org/epel/zfs-release-3-0$(rpm --eval "%{dist}").noarch.rpm

Confirm the ZFS package exists in the repo

Run

dnf repolist | grep zfs

Sample output

zfs                  OpenZFS for EL10 - dkms

Install the ZFS package
```
sudo dnf install -y zfs
```
This will take some time…

2.3. Fedora

Remove zfs-fuse
```
sudo rpm -e --nodeps zfs-fuse
```

Add ZFS repo

sudo dnf install -y https://zfsonlinux.org/epel/zfs-release-3-0$(rpm --eval "%{dist}").noarch.rpm

Install kernal headers

sudo dnf install -y kernel-devel-$(uname -r | awk -F'-' '{print $1}')

Install ZFS packages
```
sudo dnf install -y zfs
```
This will take some time…

2.4. Common Config

Load kernal module
```
sudo modprobe zfs
```
By default, ZFS kernel modules are loaded upon detecting a pool. To always load the modules at boot

This step is required for ZFS to startup automatically on reboots.
Become root
```
su
```
```
echo zfs > /etc/modules-load.d/zfs.conf
```
By default, ZFS may be removed by kernel package updates. To lock the kernel version to only ones supported by ZFS to prevent this:
```
echo 'zfs' > /etc/dnf/protected.d/zfs.conf
```
Exit su mode via Ctrl+d .

3. Manage with GUI

3.1. Via `Cockpit`

Cockpit and the ZFS plugin will be required.
Use Cockpit or the following cmdline instructions:

4. Manage via cmd line

4.1. Create a `Storage Pool`

4.1.1. Get list of Disks

Method 1

sudo fdisk -l | grep Disk | grep sectors | grep -v loop

Example

Disk /dev/sda: 153.39 GiB, 164696555520 bytes, 321672960 sectors
Disk /dev/sdb: 698.64 GiB, 750156374016 bytes, 1465149168 sectors
Disk /dev/sdc: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/nvme0n1: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk /dev/sdd: 58.44 GiB, 62746787840 bytes, 122552320 sectors
Disk /dev/zram0: 8 GiB, 8589934592 bytes, 2097152 sectors

Method 2

lsblk -o name,size,fstype,type,mountpoint | grep -v loop

Example

NAME          SIZE FSTYPE TYPE MOUNTPOINT
sda         153.4G        disk
├─sda1      153.4G        part
└─sda9          8M        part
sdb         698.6G        disk
├─sdb1      698.6G        part
└─sdb9          8M        part
sdc         931.5G        disk
├─sdc1      931.5G        part
└─sdc9          8M        part
sdd          58.4G        disk
└─sdd1       58.4G vfat   part
zram0           8G        disk [SWAP]
nvme0n1     232.9G        disk
├─nvme0n1p1   600M vfat   part /boot/efi
├─nvme0n1p2     1G ext4   part /boot
└─nvme0n1p3 231.3G btrfs  part /var/lib/containers/storage/overlay

4.1.2. Create a `Storage Pool`

Create one of the following Storage Pool types:
Single: Use the single 1TB disk to create a 1TB Storage Pool
```
sudo zpool create SingleDisk /dev/sdc
```
Stripped: Combine the 165GiB sda & 750GiB sdb disks into a 915GiB RAID 0 Storage Pool
```
sudo zpool create StrippedPool /dev/sda /dev/sdb
```
Mirrored: Combine the sda & sdb disks into a RAID 1 Storage Pool
```
sudo zpool create MirroredPool mirror /dev/sda /dev/sdb
```
RAID5: Combine the sda, sdb & sdc disks into a RAID-Z Storage Pool
```
sudo zpool create RAIDZPool raidz /dev/sda /dev/sdb /dev/sdc
```

4.2. Pool Status

Get the Pool statuses

zpool status | grep ONLINE

Example output

 state: ONLINE
        SingleDisk  ONLINE       0     0     0      (1)
          sdc       ONLINE       0     0     0

 state: ONLINE
        StrippedPool  ONLINE       0     0     0    (2)
          sda         ONLINE       0     0     0
          sdb         ONLINE       0     0     0

1	Single Disk `Storage Pool`
2	Stripped `Storage Pool`

List the Storage Pools

zpool list

Example output

NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
SingleDisk     928G   372K   928G        -         -     0%     0%  1.00x    ONLINE  -
StrippedPool   849G   211K   849G        -         -     0%     0%  1.00x    ONLINE  -

4.3. Grow a `Storage Pool`

Generally, you’ll need to replace all the disks with larger ones.

4.3.1. Grow The Easy Way

Run this first
```
zpool set autoexpand=on <name of pool>
```

4.3.2. Grow The Hard way

If you didn’t run the first step of the easy way, then you’ll need to run this:

Get the list of devices

sudo zpool status

Sample output

  pool: MirroredDisk
 state: ONLINE
  scan: resilvered 254G in 00:28:50 with 0 errors on Fri Aug 28 17:38:46 2026
config:

        NAME                                  STATE     READ WRITE CKSUM
        MirroredDisk                          ONLINE       0     0     0
          mirror-0                            ONLINE       0     0     0
            ata-TOSHIBA_DT01ACA200_23ON8A2GS  ONLINE       0     0     0
            wwn-0x5000039ff3d74acd            ONLINE       0     0     0  (1)

1	This is the newly replaced device

Run this:

zpool online -e MirroredDisk wwn-0x5000039ff3d74acd

4.4. Delete a `Storage Pool`

Destroy
```
sudo zpool destroy SingleDisk
```

4.5. Import Storage Pools

Find Pools that can be imported:

sudo zpool import

Sample output

  pool: MirroredDisk
    id: 5839134755486227288
 state: ONLINE
action: The pool can be imported using its name or numeric identifier. (1)
config:

        MirroredDisk  ONLINE
          mirror-0    ONLINE
            sdb       ONLINE
            sdc       ONLINE

1	This pool can be imported

Import the Pool
```
sudo zpool import -f MirroredDisk
```

List the imported Pool

sudo zpool list

Sample output

NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
MirroredDisk  29.5G  11.3G  18.2G        -         -     3%    38%  1.00x    ONLINE  -

4.5.1. Troubled Pools

For an issue like the following:

sudo zpool import -f USB-16TB
cannot import 'USB-16TB': I/O error
        Recovery is possible, but will result in some data loss.
        Returning the pool to its state as of Sun 11 Aug 2024 10:05:01 AM EDT
        should correct the problem.  Approximately 10 seconds of data
        must be discarded, irreversibly.  Recovery can be attempted
        by executing 'zpool import -F USB-16TB'.  A scrub of the pool
        is strongly recommended after recovery.

Force the import
```
sudo zpool import -fF USB-16TB
```

Check scrub status

status

sudo zpool status USB-16TB

scrub progress

  pool: USB-16TB
 state: ONLINE
  scan: scrub in progress since Sun Aug 11 00:24:25 2024
        5.79T scanned at 0B/s, 4.09T issued at 220M/s, 5.79T total
        0B repaired, 70.59% done, 02:15:08 to go    (1)
config:

        NAME                                            STATE     READ WRITE CKSUM
        USB-16TB                                        ONLINE       0     0     0
          usb-ATA_ST16000NT001-3LV_0123456789ABCDE-0:0  ONLINE       0     0     0

errors: No known data errors                        (2)

1	Scrub progress with 2 hours 15 minutes to go
2	No discovered errors yet

4.6. Datasets

An existing ZFS volume can be divided into datasets. Permissions, compression, deduplication, and quotas can be set on a per-dataset basis, allowing more granular control over access to storage data….

— http://doc.freenas.org/9.10/storage.html#create-dataset

4.7. Create a Dataset

Create the DataSet within an existing Pool
```
sudo zfs create RAIDZPool/Folder1
```

Show the created DataSet and its default mountpoint

zfs get all | grep mountpoint

Output

RAIDZPool          mountpoint            /RAIDZPool             default
RAIDZPool/Folder1  mountpoint            /RAIDZPool/Folder1     default

Change the DataSet’s mountpoint

sudo zfs set mountpoint=/mnt/RAIDZPool_Folder1 RAIDZPool/Folder1

Show the DataSet’s new mountpoint

zfs get all | grep mountpoint

Output

RAIDZPool          mountpoint            /RAIDZPool              default
RAIDZPool/Folder1  mountpoint            /mnt/RAIDZPool_Folder1  local  (1)

1	New mountpoint

Permission the MountPoint

sudo chown -R mattosd:mattosd /mnt/RAIDZPool_Folder1

4.7.1. Remove a DataSet

Destroy the DataSet
```
sudo zfs destroy RAIDZPool/Folder1
```
Remove the mountpoint
```
sudo rm -R /mnt/RAIDZPool_Folder1
```

4.8. Zvols

A zvol is a feature of ZFS that creates a raw block device over ZFS. This allows you to use a zvol as an iSCSI device extent.

— http://doc.freenas.org/9.10/storage.html#create-zvol

ZVols can be used as virtual disk for VMs

4.9. Replacing Bad Disks

Physically plug in the new disk

Do not use alow quality USB encloser or cable!
A ZFS resilver will hammer the drive with maximum read/write operations for hours or days at a time.
The USB connection will almost certainly drop or reset during this process, which can cause the data rebuild to fail and can potentially stall or crash your entire RAIDZPool

If using USB, you need to see Driver=uas, 5000M for maximum throughput:

Run

lsusb -t

Example

lsusb -t
/: Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/10p, 5000M
|__ Port 002: Dev 005, If 0, Class=Mass Storage, Driver=uas, 5000M  (1)

1	Must see Driver=uas, 5000M

Determine the bad disk

zpool status

example

pool: RAIDZPool
 state: ONLINE
config:

	NAME                        STATE     READ WRITE CKSUM
	RAIDZPool                   ONLINE       0     0     0
	  raidz1-0                  ONLINE       0     0     0
	    wwn-0x50024e9002719836  ONLINE       1     0     0 (1)
	    wwn-0x5000c5000d5a30b9  ONLINE       0     0     0
	    wwn-0x50014ee2b53376c6  ONLINE       0     0     0

1	This is the bad disk

Get a list of disks and identify the original and replacement disks:

ls -l /dev/disk/by-id/

Example

lrwxrwxrwx 1 root root  9 Nov  1 08:14 wwn-0x50024e9002719836 -> ../../sda          (1)
lrwxrwxrwx 1 root root 10 Nov  1 08:14 wwn-0x50024e9002719836-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov  1 08:14 wwn-0x50024e9002719836-part9 -> ../../sda9
lrwxrwxrwx 1 root root  9 Nov  1 14:54 wwn-0x50024e900271983a -> ../../sdd          (2)
lrwxrwxrwx 1 root root 10 Nov  1 14:55 wwn-0x50024e900271983a-part1 -> ../../sdd1

1	Original
2	Replacement

Wipe the replacement so ZFS can safely claim the entire disk without a "device is in use" error.:
```
sudo wipefs -a /dev/disk/by-id/wwn-0x50024e900271983a
```

Replace with:

Command syntax

sudo zpool replace <pool-name> <old-disk-path> <new-disk-path>

Example

sudo zpool replace RAIDZPool wwn-0x50024e9002719836 wwn-0x50024e900271983a

Monitor status

zpool status

Example Output

  pool: RAIDZPool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Nov  1 16:40:03 2026
        662G / 721G scanned at 12.7G/s, 0B / 721G issued
        0B resilvered, 0.00% done, no estimated completion time
config:

        NAME                          STATE     READ WRITE CKSUM
        RAIDZPool                     ONLINE       0     0     0
          raidz1-0                    ONLINE       0     0     0
            replacing-0               ONLINE       0     0     0
              wwn-0x50024e9002719836  ONLINE       2     0     0
              wwn-0x50024e900271983a  ONLINE       0     0     0
            wwn-0x5000c5000d5a30b9    ONLINE       0     0     0
            wwn-0x50014ee2b53376c6    ONLINE       0     0     0

Checked again:

zpool status
  pool: RAIDZPool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Nov  1 16:40:03 2026
        721G / 721G scanned, 119G / 721G issued at 224M/s
        40.1G resilvered, 16.52% done, 00:45:48 to go
config:

        NAME                          STATE     READ WRITE CKSUM
        RAIDZPool                     DEGRADED     0     0     0
          raidz1-0                    DEGRADED     0     0     0
            replacing-0               DEGRADED     0     0   280
              wwn-0x50024e9002719836  FAULTED    289     0     0  too many errors
              wwn-0x50024e900271983a  ONLINE       0     0     0  (resilvering)
            wwn-0x5000c5000d5a30b9    ONLINE       0     0     0
            wwn-0x50014ee2b53376c6    ONLINE       0     0     0

Wait until the pool state has changed from DEGRADED to ONLINE:

zspool status

Finished

  pool: RAIDZPool
 state: ONLINE
  scan: resilvered 241G in 00:54:48 with 0 errors on Sun Nov  1 17:34:51 2026
config:

        NAME                        STATE     READ WRITE CKSUM
        RAIDZPool                   ONLINE       0     0     0
          raidz1-0                  ONLINE       0     0     0
            wwn-0x50024e900271983a  ONLINE       0     0     0  (1)
            wwn-0x5000c5000d5a30b9  ONLINE       0     0     0
            wwn-0x50014ee2b53376c6  ONLINE       0     0     0

1	Replacement drive

Power down the server
Swap the bad drive for the replacement
Power up
Run a data validation pass to clean up any remaining transfer errors:
```
sudo zpool scrub RAIDZPool
```

Linux ZFS Notes

1. Reference Docs

2. Installation

2.1. Ubuntu

2.2. Rocky

2.3. Fedora

2.4. Common Config

3. Manage with GUI

3.1. Via Cockpit

4. Manage via cmd line

4.1. Create a Storage Pool

4.1.1. Get list of Disks

4.1.2. Create a Storage Pool

4.2. Pool Status

4.3. Grow a Storage Pool

4.3.1. Grow The Easy Way

4.3.2. Grow The Hard way

4.4. Delete a Storage Pool

4.5. Import Storage Pools

4.5.1. Troubled Pools

4.6. Datasets

4.7. Create a Dataset

4.7.1. Remove a DataSet

4.8. Zvols

4.9. Replacing Bad Disks

3.1. Via `Cockpit`

4.1. Create a `Storage Pool`

4.1.2. Create a `Storage Pool`

4.3. Grow a `Storage Pool`

4.4. Delete a `Storage Pool`