Five layers, one migration — what a fabric cutover exposed about storage

The migration process involved moving a Windows workload and two TrueNAS servers from a legacy flat network to a new dual-TOR fabric on the workload VLAN. The plan was straightforward: repoint SMB mappings and ensure the workload and servers became healthy. However, the concept of "healthy" proved to be multi-layered and complex. The migration exposed five distinct layers of the storage stack, each with its own unique failure modes. These layers, presented in the order they surfaced during the migration, are:

1. Network reachability: This layer involves both physical link and L2/L3 configuration. A failure in this layer, such as a Windows host with an unrecognized network adapter or a TrueNAS server without the workload VLAN in its LAGG, could go unnoticed at higher layers.

2. Service reachability: This layer focuses on the SMB advertisement, which is independent of what lies behind it. An example of a failure here is a shared dataset not being mounted, even though the SMB service advertises the share correctly.

3. Filesystem state: At this layer, TrueNAS presents a cached view of pool status. If a ZFS dataset fails to mount, the SMB share remains advertised but cannot be used.

4. ZFS state: Here, the kernel sees the pool importable, disks as readable, and labels intact. However, if the ZFS pool isn't imported due to invisible disks, the filesystem state can still appear healthy at this layer.

5. Physical storage-path state: This final layer deals with the SAS and CAM layer under ZFS. For instance, if a JBOD shelf isn't powered on after a re-rack, the physical storage path can be compromised.

During the migration, each of these five layers failed at least once, revealing their issues independently. The layered framing is a retrospective perspective, and it's important to note that the day's experience may not have felt as methodical. The key takeaway from this migration experience is the importance of validating each layer individually, rather than assuming a healthy signal at one layer implies a healthy signal at the next.

A comprehensive migration checklist should ensure that both sides of a link can communicate at the address the change plan specifies, considering both the switch's side and the host's side.