ZFS dataset shadowing hid 5.5TB from me during an outage
My reverse proxy container stopped starting one evening, which meant every service behind it stopped resolving. Jellyfin, Immich, Nextcloud, the dashboard — all of it sits behind Nginx Proxy Manager in CT104, so when CT104 stays down, the whole *.bmlab.co.in namespace is down with it. It took me a few hours to find the actual cause, and the thing that cost me the most time was a warning that had…
A mini PC running Proxmox experienced an outage due to a ZFS dataset shadowing issue. The reverse proxy container, Nginx Proxy Manager, stopped starting, causing all services behind it to stop resolving. The main cause was a missing space in the pool vault, which was 100% full. The pool consisted of two ZFS pools: flash (2TB SSD) and vault (8TB IronWolf).
The issue stemmed from over-provisioned sparse raw images and copying downloads instead of hardlinking them, resulting in about 5.5TB of unnecessary data occupying the pool. To diagnose the issue, the user used du on obvious paths, which did not account for the full pool size. They also incorrectly assumed that duplicate content would not impact the pool's capacity.
The recovery process involved freeing up space by deleting verified duplicate downloads and moving rootfs volumes off the filled pool to a different storage.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
