TrueNAS iSCSI Thin Provisioning Error

Posted on June 18, 2026

Introduction and Problem Description

Problem Description
When using the default configuration of TrueNAS to setup an iSCSI shared LUN, the iSCSI extent will generate Thin provisioning soft threshold reached errors. The LUN can and will report dozens, or hundreds, possibly thousands of these errors, flooding the logs of the systems accessing the LUN. In this case it was VMware vCenter logging thousands of messages a day (Space utilization on thin-provisioned device naa.xxx exceeded configured threshold). This problem exists on TrueNAS version 12.0-U8.1 and 13.0-U6.4.

It took us a couple of months of troubleshooting before we were able to locate what was causing the problem. This blog post details what we learned, the commands that we tried, and finally the solution.
TLDR
Fear not, our goal isn't to keep you trapped on this site forcing you to read the whole thing to find the answer. In the next section we will post the simple config change to prevent the error.
Why This is a Problem
This error message means that either VMware or the SAN believe that the LUN is consuming more storage than the thin provisioned threshold is set to. Thin provisioning is when you tell a server that it has more storage than it actually has.
For example, we could build a new VM with 100 GB of storage assigned to it, but only have a 25 GB drive attached. So we thin provisioned another 75 GB of space above and beyond what the physical drive contains. We invented the extra space and are lying to the VM, telling it that it has more storage than it actually does. The problem with thin provisioning is, if we tried to write 26 GB of data to that server, the server would probably crash, because there is no place to actually write that much data to. To prevent servers from crashing, we would setup thin provisioning thresholds to alert us when the storage was getting too full. For example, at 80% of total storage usage, we would start to generate errors and send out notifications, so that we have time to add more actual storage to the thin provisioned storage, hopefully preventing a crash because we ran out of storage. The error messages that we are receiving, are the warnings that we have reached or gone past that threshold.

TL;DR (Too Long; Didn't Read)

To prevent the Space utilization on thin-provisioned device naa.xxx exceeded configured threshold errors from being generated when using iSCSI LUN's on TrueNAS with VMware:

In the TrueNAS GUI, when you go to Sharing -> Block Shares (iSCSI) -> Extents -> and edit s1t01e01, just uncheck the Enable TPC.

A screenshot of the TrueNAS GUI with Sharing, Block Shares (iSCSI), Extents and the edit menu for s1t01e01 open. There is a green circle around the unchecked box for Enable TPC with a blue save button at the bottom, all other settings are at their defaults.

This setting can be changed after the fact without causing any disruptions or downtime to production.

Commands

ctladm lunlist
Display a list of all of the iSCSI LUNs by ctl-lun id.
cat {file_name}
Display the contents of a text file.
ctladm readcap {lun_id}
Display the last message from the iSCSI LUN.
ctladm reqsense {lun_id}
Display the last error message from the iSCSI LUN.
ctladm devlist -v
Display each LUN along with verbose detail about their configurations.
zpool status
Display the status of each zpool, along with some disk information.
zpool list {zpool_id}
Display information such as Size, Allocation, and Free space of a zpool based on the zpool ID.
zpool events -v
Display the event logs from the zpools.
zfs get all {zpool_id}
Display all of the configurations of the zpool (reservations, guid, quota, etc...).
zfs list -t volume
Display all zvol information such as Used, Available, and Referenced space.
glabel status
Display disk label information, in our case the gptid to dev id mappings.
geom disk list
Display disk I/O information, in our case pulling raw disk sizes.

Finding the Errors

Finding vCenter Errors
These instructions are for vCenter and ESXi v7.x, but may work for other versions as well.
Once we are signed into vCenter, navigate to the datastores then select a LUN. In this case, we will select s1lun11. From there we will go to the Monitor tab, and then to Tasks and Events and finally to Events.

If we look through the event logs, we may start to see a bunch of Warnings that say something like Space utilization on thin-provisioned device naa.6589cfc000000146a4302d61f1ef4116 exceeded configured threshold. Affected datastores (if any): s1lun11. In this case, I see the error message being generated every 3 to 4 minutes on that LUN.

Finding ZFS Errors
We should examine the ZFS events to see if the errors are coming from ZFS, or possibly somewhere else. This is a pretty simple command, but it has a lot of output. So for this example, we will just look at a single days output to keep things simple.

(bash) zpool events -v
Jun 14 2026 00:00:01.096587179 sysevent.fs.zfs.scrub_start
        version = 0x0
        class = "sysevent.fs.zfs.scrub_start"
        pool = "s1z6p01"
        pool_guid = 0x5090c805d7da8c67
        pool_state = 0x0
        pool_context = 0x0
        time = 0x6a2e5171 0x5c1cdab
        eid = 0xba

Jun 14 2026 00:00:01.096587179 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "s1z6p01"
        pool_guid = 0x5090c805d7da8c67
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "san01.corp.combobulate.test"
        history_internal_str = "func=1 mintxg=0 maxtxg=2462043"
        history_txg = 0x2591b
        history_time = 0x6a2e5171
        time = 0x6a2efd61 0x2e67c749
        eid = 0xbc

Jun 14 2026 00:00:01.096587179 sysevent.fs.zfs.scrub_finish
        version = 0x0
        class = "sysevent.fs.zfs.scrub_finish"
        pool = "s1z6p01"
        pool_guid = 0x5090c805d7da8c67
        pool_state = 0x0
        pool_context = 0x0
        time = 0x6a2efd61 0x2e67c749
        eid = 0xbd

These were the last events that were logged, and there are no errors that have been generated. This log is just showing the start and finish of a pool scrub, which is perfectly normal. So it looks like we need to search elsewhere for our errors.

Finding SCSI Errors
We will continue to use s1lun11 as the LUN. In this case, the LUN number is part of the name (LUN 11) that vCenter is using, so that makes it a little easier.

The next piece that we need to figure out, is the LUN ID that is used. LUN11 is the target name, not the actual ctl-lun identifier. The information on the LUN's, extents, targets, auth-groups, and portal groups are stored in /etc/ctl.conf. So let's examine that file to see what is going on.

(bash) cat /etc/ctl.conf
portal-group "default" {
}

portal-group "pg1" {
    tag "0x0001"
    discovery-filter "portal-name"
    discovery-auth-group "no-authentication"
    listen "10.10.250.10:3260"
    listen "10.10.251.10:3260"
    listen "10.10.252.10:3260"
    listen "10.10.253.10:3260"
    option "ha_shared" "on"
}

lun "s1t01e01" {
    ctl-lun "0"
    path "/dev/zvol/s1z6p01/s1p01z01"
    blocksize "512"
    serial "0cc47aeb7052000"
    device-id "iSCSI Disk   0cc47aeb7052000             "
    option "vendor" "TrueNAS"
    option "product" "iSCSI Disk"
    option "revision" "0123"
    option "naa" "0x6589cfc000000146a4302d61f1ef4116"
    option "insecure_tpc" "on"
    option "avail-threshold" "51054276273766"
    option "rpm" "7200"
}

lun "s1t02e02" {
    ctl-lun "1"
    path "/dev/zvol/s1z6p02/s1p02z02"
    blocksize "512"
    serial "0cc47aeb7052001"
    device-id "iSCSI Disk   0cc47aeb7052001             "
    option "vendor" "TrueNAS"
    option "product" "iSCSI Disk"
    option "revision" "0123"
    option "naa" "0x6589cfc0000004890eda588bdf208b7"
    option "insecure_tpc" "on"
    option "avail-threshold" "51054276273769"
    option "rpm" "5400"
}

auth-group "ag4tg1_1" {
    initiator-name "iqn.1998-01.com.vmware:esx01.corp.combobulate.test:558930432:64"
    initiator-name "iqn.1998-01.com.vmware:esx02.corp.combobulate.test:134567035:64"
    initiator-name "iqn.1998-01.com.vmware:esx03.corp.combobulate.test:1859839222:64"
    initiator-portal "10.10.250.0/24"
    initiator-portal "10.10.251.0/24"
    initiator-portal "10.10.252.0/24"
    initiator-portal "10.10.253.0/24"
    auth-type "none"
}

target "iqn.2005-101.org.freenas.ctl:s1t01" {
    alias "s1t01"
    portal-group "pg1" "ag4tg1_1"

    lun "11" "s1t01e01"
}

auth-group "ag4tg2_2" {
    initiator-name "iqn.1998-01.com.vmware:esx01.corp.combobulate.test:558930432:64"
    initiator-name "iqn.1998-01.com.vmware:esx02.corp.combobulate.test:134567035:64"
    initiator-name "iqn.1998-01.com.vmware:esx03.corp.combobulate.test:1859839222:64"
    initiator-portal "10.10.250.0/24"
    initiator-portal "10.10.251.0/24"
    initiator-portal "10.10.252.0/24"
    initiator-portal "10.10.253.0/24"
    auth-type "none"
}

target "iqn.2005-101.org.freenas.ctl:s1t02" {
    alias "s1t02"
    portal-group "pg1" "ag4tg2_2"

    lun "12" "s1t02e02"
}

Examining this output, we should be looking for where it says lun 11, and it looks like that is defined under the target section. The target section is what defines the LUN share name and the extent that the LUN belongs to. In this case the line says lun "11" "s1t01e01", so LUN 11 belongs to the s1t01e01 extent.

Now that we have the extent name, we should go back through the file and pull out the ctl-lun ID. Finding the extent section for s1t01e01, we will find ctl-lun "0", so now we know the ctl-lun ID is 0.

We now have all of the required info to start using ctladm. So now we will use readcap to look into the SCSI messages that ctl-lun 0 has been issuing. Please note, we may need to run the command several times to actually see the error message.

(bash) ctladm readcap 0
(7:0:0/0): READ CAPACITY(10). CDB: 25 00 00 00 00 00 00 00 00 00  Tag: 0/1
(7:0:0/0): CTL Status: SCSI Error
(7:0:0/0): SCSI Status: Check Condition
(7:0:0/0): SCSI sense: UNIT ATTENTION asc:29,1 (Power on occurred)

(bash) ctladm readcap 0
Disk Capacity: 124644229183, Blocksize: 512

(bash) ctladm readcap 0
Disk Capacity: 124644229183, Blocksize: 512

(bash) ctladm readcap 0
(7:0:0/0): READ CAPACITY(10). CDB: 25 00 00 00 00 00 00 00 00 00  Tag: 0x1b/1
(7:0:0/0): CTL Status: SCSI Error
(7:0:0/0): SCSI Status: Check Condition
(7:0:0/0): SCSI sense: UNIT ATTENTION asc:38,7 (Thin provisioning soft threshold reached)
(7:0:0/0): Info: 0x20

As we can see from the first three attempts, we were getting back info that wasn't exactly helpful for the error, but on the fourth attempt we struck gold. On the fourth attempt we can see that there is a SCSI Error and the error is Thin provisioning soft threshold reached.

We can also use ctladm reqsense to see the last error message as well.

(bash) ctladm reqsense 0
(7:0:0/0): REQUEST SENSE. CDB: 03 00 00 00 fc 00
(7:0:0/0): SCSI sense: UNIT ATTENTION asc:38,7 (Thin provisioning soft threshold reached)
(7:0:0/0): Info: 0x20

While this isn't great news, it is good news. We can now see the error message in SCSI that VMware is reporting. Which means that the error lies on the SAN somewhere...but where?

Verify Space

Description
Since we know that the errors are based on thin provisioning hitting or surpassing thresholds, we need to figure out the total amount of space we have, how much space we are using, and what the thresholds are. These next sections will go through tracking down this information and doing the comparisons to make sure that we are still within the thresholds.

Note, we have already verified that the errors exist on the SAN (reported by SCSI), so we will focus on the SAN and ignore VMware for now.

ZFS Primer
We won't go into a ton of detail on ZFS (Zettabyte File System). You can read more about it here in this article.

vdev (Virtual Devices): These are block devices that are built with files, partitions, or entire disks. It is recommended to use entire disks.
RAIDZ (Redundant Array of Inexpensive Disks for ZFS): These are used to combine disks into a single vdev. We can use these to create one big storage drive that has no redundancy, or has multiple disk fail over redundancy.
zpool (ZFS Pool): These are the logical grouping of vdev's. When creating a zpool you specify how many vdev's, where the vdev's are located, and what RAIDZ to use.
zvol (ZFS Volume): These are virtual volumes that represent block devices.

So, in our case, we have taken multiple hard drives (vdev's) and combined them together into a zpool with a RAID of RAIDZ-2 (which means we can lose 2 drives before we lose data). We then created a block storage volume (zvol) that we are using as our iSCSI LUN.
Let's go through the ZFS commands to view the entire zpool configuration.

(bash) zpool status
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:10 with 0 errors on Sat Jun 13 03:45:10 2026
config:

        NAME        State   READ WRITE CKSUM
        boot-pool   ONLINE     0     0     0
          mirror-0  ONLINE     0     0     0
            ada0p2  ONLINE     0     0     0
            ada1p2  ONLINE     0     0     0

errors: No known data errors

  pool: s1z6p01
 state: ONLINE
  scan: scrub repaired 0B in 12:13:36 with 0 errors on Sun Jun 14 12:13:37 2026
config:

        Name                                            STATE       READ WRITE CKSUM
        s1z6p01                                         ONLINE         0     0     0
          raidz2-0                                      ONLINE         0     0     0
            gptid/b2406590-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/b2490256-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/b2715c55-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/b283d2e4-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/b279a9d1-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/b28c2237-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0

errors: No known data errors

  pool: s1z6p02
 state: ONLINE
  scan: scrub repaired 0B in 00:00:25 with 0 errors on Sun May 24 00:00:26 2026
config:

        Name                                            STATE       READ WRITE CKSUM
        s1z6p02                                         ONLINE         0     0     0
          raidz2-0                                      ONLINE         0     0     0
            gptid/7f3eb7e0-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/7f30d49e-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/7f4c33d1-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/8058d271-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/80c91777-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0
            gptid/80e341ca-f559-11f0-bec0-0cc47aeb7052  ONLINE         0     0     0

errors: No known data errors

As we can see from the output, we have 3 pools. The first pool is just the boot-pool which contains the OS. The second pool is named s1z6p01 and it contains 6 drives configured in a RAIDZ-2. And the third pool is named s1z6p02 and it contains 6 drives also configured in a RAIDZ-2.

Drive / vdev Information
Let's go through how to map the gptid (GUID (Global Unique Identifier) Partition Table Identifier) back to the storage drives. This isn't really part of this troubleshooting, but it could help us out in the future if we have this knowledge.

(bash) glabel status
                                       Name  Status  Components
gptid/fa1a273d-bce8-11ec-8711-a0369f53074c      N/A  ada0p1
gptid/fa2ab6a5-bce8-11ec-8711-a0369f53074c      N/A  ada1p1
gptid/b2490256-f559-11f0-bec0-0cc47aeb7052      N/A  da0p2
gptid/b279a9d1-f559-11f0-bec0-0cc47aeb7052      N/A  da1p2
gptid/b2406590-f559-11f0-bec0-0cc47aeb7052      N/A  da2p2
gptid/b28c2237-f559-11f0-bec0-0cc47aeb7052      N/A  da3p2
gptid/b2715c55-f559-11f0-bec0-0cc47aeb7052      N/A  da4p2
gptid/b283d2e4-f559-11f0-bec0-0cc47aeb7052      N/A  da5p2
gptid/7f30d49e-11f1-11f0-92da-0cc47aeb7052      N/A  da10p2
gptid/7f3eb7e0-11f1-11f0-92da-0cc47aeb7052      N/A  da6p2
gptid/7f4c33d1-11f1-11f0-92da-0cc47aeb7052      N/A  da7p2
gptid/8058d271-11f1-11f0-92da-0cc47aeb7052      N/A  da11p2
gptid/80c91777-11f1-11f0-92da-0cc47aeb7052      N/A  da9p2
gptid/80e341ca-11f1-11f0-92da-0cc47aeb7052      N/A  da8p2
gptid/fa30ff2f-bce8-11ec-8711-a0369f53074c      N/A  ada1p3
gptid/fa1f48f0-bce8-11ec-8711-a0369f53074c      N/A  ada0p3

This just maps each of the gptid's back to the /dev so that we can identify the hard drives.
Next let's take a look at all of the sizes of the drives as well. Since TrueNAS is running on BSD, we aren't able to use lsblk so we will use something called geom. This will give us way more information than we need, but it may come in handy later. We won't be listing out all of the drives, but we will list the ones associated with this zpool (s1z6p01).

(bash) geom disk list
Geom name: da0
Providers:
1. Name: da0
   Mediasize 20000588955648 (18T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: SEAGATE ST20000NM002D
   lunid: 5000c500d86c370f
   ident: ZVT09NRT0000C21888MF
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da1
Providers:
1. Name: da1
   Mediasize 20000588955648 (18T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: SEAGATE ST20000NM002D
   lunid: 5000c500d8702dcf
   ident: ZVT09SH50000C22081JZ
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da2
Providers:
1. Name: da2
   Mediasize 20000588955648 (18T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: SEAGATE ST20000NM002D
   lunid: 5000c500d87019ab
   ident: ZVT09S260000C22080R1
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da3
Providers:
1. Name: da3
   Mediasize 20000588955648 (18T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: SEAGATE ST20000NM002D
   lunid: 5000c500d8b756fb
   ident: ZVT0GZAC0000W217ULHY
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da4
Providers:
1. Name: da4
   Mediasize 20000588955648 (18T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: SEAGATE ST20000NM002D
   lunid: 5000c500d871b6c3
   ident: ZVT09XSY0000W216SPCA
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

Geom name: da5
Providers:
1. Name: da5
   Mediasize 20000588955648 (18T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: SEAGATE ST20000NM002D
   lunid: 5000c500d85d8eb7
   ident: ZVT08KTY0000C216E70J
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

There is a lot of information here. The big things to pay attention to are the Name (device name), Mediasize (drive size), descr (drive model), and ident (drive serial number). We won't go into all of the details now, but we will reference back to these numbers in other sections.

Determine ZFS Space / Usage
From looking at the /etc/ctl.conf earlier, we know that under the extent (s1t01e01) information, we can see that the path specifies the zpool and the zvol (path "/dev/zvol/s1z6p01/s1p01z01"). In this case, the zpool is s1z6p01 and the zvol is s1p01z01.

With this information we can examine the zpool information to determine sizes.

(bash) zpool list s1z6p01
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
s1z6p01   109T  43.2T  65.9T        -         -     0%    39%  1.00x    ONLINE  /mnt

Some of these numbers are a little misleading. The 109T is the total size of all of the vdev's. If we go back to the output of the geom command, we will see that each of the drives are 18T in size. We know that we have 6 disks in this zpool based on the zpool status command that we ran earlier. 18 * 6 = 108 Not exact, but if we broke it down into bits and or bytes, then they would match up. We need to remember, that we don't have access to all of that 108T, we have RAID overhead, and ZFS has a habit of requiring 20% of free space overhead as well.

The ALLOC number is how much data has been written to the zpool. So in our case right now, there has only been 43.2 T written (allocated) to this zpool.

We should look at the size of the zvols as well, as the zvol is the actual block device that is shared out using iSCSI.

(bash) zfs list -t volume
NAME               USED  AVAIL    REFER  MOUNTPOINT
s1z6p01/s1p01z01  58.5T  43.8T    28.8T  -
s1z6p02/s1p02z02  58.5T  72.6T     656M  -

Reading these numbers, it looks like we have a LUN size of 58.5T, at least that is what would be shared out if the entire zvol was added to a single LUN, which is exactly what we did.

This is where it gets a bit tricky. The REFER is how much data has been written to the zvol. The referenced amount of data is 28.8T. But wait, this 28.8T does not match the ALLOC from the zpool...well, yeah it kind of does. The ALLOC, from the zpool, is the total amount of data that is written, this includes all of the data from REFER plus all of the RAIDZ overhead. A 6 disk RAIDZ-2 adds around 1.5 times the amount of space for redundancy. So 28.8 * 1.5 = 43.2 which matches up exactly.

Looking at all of these numbers, we are not using more than 50% of space anywhere, according to zfs.

We should probably verify that there aren't any quotas, reservations, or anything else configured on the zpool that could generate any threshold errors.

(bash) zfs get all s1z6p01
NAME     PROPERTY              VALUE                  SOURCE
s1z6p01  type                  filesystem             -
s1z6p01  creation              Mon Jan 19  9:10 2026  -
s1z6p01  used                  58.5T                  -
s1z6p01  available             14.1T                  -
s1z6p01  referenced            192K                   -
s1z6p01  compressratio         1.03x                  -
s1z6p01  mounted               yes                    -
s1z6p01  quota                 none                   -
s1z6p01  reservation           none                   default
s1z6p01  recordsize            128K                   default
s1z6p01  mountpoint            /mnt/s1z6p01           default
s1z6p01  sharenfs              off                    default
s1z6p01  checksum              on                     default
s1z6p01  compression           lz4                    local
s1z6p01  atime                 on                     default
s1z6p01  devices               on                     default
s1z6p01  exec                  on                     default
s1z6p01  setuid                on                     default
s1z6p01  readonly              off                    default
s1z6p01  jailed                off                    default
s1z6p01  snapdir               hidden                 default
s1z6p01  aclmode               passthrough            local
s1z6p01  aclinherit            passthrough            local
s1z6p01  createtxg             1                      -
s1z6p01  canmount              on                     default
s1z6p01  xattr                 on                     default
s1z6p01  copies                1                      default
s1z6p01  version               5                      -
s1z6p01  utf8only              off                    -
s1z6p01  normalization         none                   -
s1z6p01  casesensitivity       sensitive              -
s1z6p01  vscan                 off                    default
s1z6p01  nbmand                off                    default
s1z6p01  sharesmb              off                    default
s1z6p01  refquota              none                   default
s1z6p01  refreservation        none                   default
s1z6p01  guid                  15912384754104254868   -
s1z6p01  primarycache          all                    default
s1z6p01  secondarycache        all                    default
s1z6p01  usedbysnapshots       0B                     -
s1z6p01  usedbydataset         192K                   -
s1z6p01  usedbychildren        58.5T                  -
s1z6p01  usedbyrefreservation  0B                     -
s1z6p01  logbias               latency                default
s1z6p01  objsetid              54                     -
s1z6p01  dedup                 off                    default
s1z6p01  mlslabel              none                   default
s1z6p01  sync                  standard               default
s1z6p01  dnodesize             legacy                 default
s1z6p01  refcompressratio      1.00x                  -
s1z6p01  written               192K                   -
s1z6p01  logicalused           29.3T                  -
s1z6p01  logicalreferenced     42.5K                  -
s1z6p01  volmode               default                default
s1z6p01  filesystem_limit      none                   default
s1z6p01  snapshot_limit        none                   default
s1z6p01  filesystem_count      none                   default
s1z6p01  snapshot_count        none                   default
s1z6p01  snapdev               hidden                 default
s1z6p01  acltype               nfsv4                  default
s1z6p01  context               none                   default
s1z6p01  fscontext             none                   default
s1z6p01  defcontext            none                   default
s1z6p01  rootcontext           none                   default
s1z6p01  relatime              off                    default
s1z6p01  redundant_metadata    all                    default
s1z6p01  overlay               on                     default
s1z6p01  encryption            off                    default
s1z6p01  keylocation           none                   default
s1z6p01  keyformat             none                   default
s1z6p01  pbkdf2iters           0                      default
s1z6p01  special_small_blocks  0                      default

Alright, this was a bunch of information, but the big take aways are that there are no quotas and there are no reservations. So, now we can move on, the errors do not appear to be coming from ZFS.

Troubleshoot iSCSI

ctladm configs
So far we have shown that the errors aren't being generated by VMware or ZFS. So now it's time to dig deeper into iSCSI and try and track down what may be causing the errors from here. The first command we will try will be ctladm devlist -v just to see what configs are being used. A lot of this information is also present in /etc/ctl.conf so if the data looks familiar, that is why.
```
(bash) ctladm devlist -v
LUN Backend      Size (Blocks)    BS Serial Number    Device ID
  0 block         124644229184   512 0cc47aeb7052000  iSCSI Disk    0cc47aeb7052000
      lun_type=0
      num_threads=32
      vendor=TrueNAS
      product=iSCSI Disk
      revision=0123
      naa=0x6589cfc000000146a4302d61f1ef4116
      insecure_tpc=on
      avail-threshold=51054276273766
      rpm=7200
      file=/dev/zvol/s1z6p01/s1p01z01
      ctld_name=s1t01e01
  1 block         124639950976   512 0cc47aeb7052001  iSCSI Disk    0cc47aeb7052001
      lun_type=0
      num_threads=32
      vendor=TrueNAS
      product=iSCSI Disk
      revision=0123
      naa=0x6589cfc0000004890eda588bdf208b7
      insecure_tpc=on
      avail-threshold=51054276273769
      rpm=5400
      file=/dev/zvol/s1z6p02/s1p02z02
      ctld_name=s1t02e02
                        
```
As we can see here, a lot of this data is the same as what is in /etc/ctl.conf. Looking through this data, the only thing that stands out a little bit is avail-threshold, as this is the only field that has a large number.
When trying to find information on avail-threshold online, we run into several dead ends. The man page for ctladm calls out avail-threshold, used-threshold, and pool-avail-threshold as settings, but does not give a definition of those settings, which is a little odd. Doing some more digging, we can learn that avail-threshold is the amount of available space that is required before errors are generated.

If we take the setting of avail-threshold and convert it to terabytes, 51054276273766 / 1024^4 = 46.43, then compare that number to the available space of the zvol (zfs list -t volume) then we get somewhere. The available space of the zvol is 43.8T while the avail-threshold is set to 46.43T. Aha! So we have less available space, by 2.63T, than what is required by the avail-threshold. Which means that avail-threshold is doing its job and generating errors because we don't have enough free space.

Since we are using TrueNAS, we don't want to make configuration changes on the CLI because then the GUI may not match the settings of the CLI. This could cause lots of problems down the line. So if we go back into the GUI and navigate to Sharing -> Block Shares (iSCSI) -> Extents -> and edit s1t01e01 then we can see the default value for Available Space Threshold(%) is set to 80%.

Whoa! Wait a minute. The avail-threshold defaults to 80%? That means that we only have 20% of the LUN that we can use before errors begin to generate. That seems crazy, let's try flipping that number, and say that we only want to be notified when we only have 20% of free space remaining.

After we set the avail-threshold to 20% let's go back and check ctladm again and see what numbers we have now.
```
(bash) ctladm devlist -v
LUN Backend      Size (Blocks)    BS Serial Number    Device ID
  0 block         124644229184   512 0cc47aeb7052000  iSCSI Disk    0cc47aeb7052000
      lun_type=0
      num_threads=32
      vendor=TrueNAS
      product=iSCSI Disk
      revision=0123
      naa=0x6589cfc000000146a4302d61f1ef4116
      insecure_tpc=on
      avail-threshold=12763569068441
      rpm=7200
      file=/dev/zvol/s1z6p01/s1p01z01
      ctld_name=s1t01e01
  1 block         124639950976   512 0cc47aeb7052001  iSCSI Disk    0cc47aeb7052001
      lun_type=0
      num_threads=32
      vendor=TrueNAS
      product=iSCSI Disk
      revision=0123
      naa=0x6589cfc0000004890eda588bdf208b7
      insecure_tpc=on
      avail-threshold=51054276273769
      rpm=5400
      file=/dev/zvol/s1z6p02/s1p02z02
      ctld_name=s1t02e02
                        
```
Once again, let's take the setting of avail-threshold and convert it to terabytes, 12763569068441 / 1024^4 = 11.61, then compare that number to the available space of the zvol (zfs list -t volume) then we get a different picture. The available space of the zvol is still 43.8T while the avail-threshold is set to 11.61T. Finally, we now have an available threshold which is below, way below, what are actual available space is. Now, we should go check ctladm reqsense 0 again and see if the errors have gone away.
```
(bash) ctladm reqsense 0
(7:0:0/0): REQUEST SENSE. CDB: 03 00 00 00 fc 00
(7:0:0/0): SCSI sense: UNIT ATTENTION asc:38,7 (Thin provisioning soft threshold reached)
(7:0:0/0): Info: 0x20
                        
```
Well crap! We are still receiving the same error message.
Let's try something else, what if we disable TPC all together and see what happens.

Now, let's go back and run ctladm devlist -v again and see what the output is.
```
(bash) ctladm devlist -v
LUN Backend      Size (Blocks)    BS Serial Number    Device ID
  0 block         124644229184   512 0cc47aeb7052000  iSCSI Disk    0cc47aeb7052000
      lun_type=0
      num_threads=32
      vendor=TrueNAS
      product=iSCSI Disk
      revision=0123
      naa=0x6589cfc000000146a4302d61f1ef4116
      insecure_tpc=on
      rpm=7200
      file=/dev/zvol/s1z6p01/s1p01z01
      ctld_name=s1t01e01
  1 block         124639950976   512 0cc47aeb7052001  iSCSI Disk    0cc47aeb7052001
      lun_type=0
      num_threads=32
      vendor=TrueNAS
      product=iSCSI Disk
      revision=0123
      naa=0x6589cfc0000004890eda588bdf208b7
      insecure_tpc=on
      avail-threshold=51054276273769
      rpm=5400
      file=/dev/zvol/s1z6p02/s1p02z02
      ctld_name=s1t02e02
                        
```
As we can see, the avail-threshold is gone. So let's check reqsense again and see if it is still reporting errors.
```
(bash) ctladm reqsense 0
(7:0:0/0): REQUEST SENSE. CDB: 03 00 00 00 fc 00
(7:0:0/0): SCSI sense: NO SENSE asc:0,0 (No additional sense information)
                        
```
Huzzah! No more errors. If we go back into VMware, we should also see that there are no more thin-provisioning errors being generated either.
Closing Notes
We spent weeks and / or months trying to hunt down the cause of this. We tested numerous things to see if we could prevent the constant thin-provisioning error messages:
- Set avail-threshold to an astronomical number: Still generated error messages
- Set avail-threshold to almost zero: Still generated error messages
- Set reservation on s1z6p01/s1p01z01 (the zvol) to 58T: We did this in attempt to force ZFS to not display the zvol as thin-provisioned. Still generated error messages (both with astronomical and almost 0 avail-threshold).
- Set reservation on s1z6p01 (the zpool) to 58T: We did this in attempt to force ZFS to not display the zpool as thin-provisioned. This one was a real long shot, but worth a try. Still generated error messages (both with astronomical and almost 0 avail-threshold).
No matter what we tried, any time the avail-threshold was set, to any value, we would generate thin-provisioning errors. This might be a bug in how ctladm uses avail-threshold. What we did find, is if we disable TPC through the GUI, then the errors would go away.

We don't use thin-provisioning on any of our LUN's or VM's. We have always thick provisioned or lazy-zero provisioned storage. So, we are able to disable TPC and the avail-threshold without fear of ever over provisioning our data. If you are using thin-provisioning, then disabling this may also disable important warnings. But as we have seen, it doesn't matter what the settings are, ctladm will report errors, so it is your call.

This was a very complex and highly technical post. If you have any questions, or need further details on any of this, please reach out to info@combobulate.tech. We will try and be prompt with our response, and to also update this blog post.