SadServers
  • Scenarios
  • Labs
    All Labs Linux & Bash Web Servers Databases Data Processing Docker Kubernetes CI/CD Infrastructure as Code Tooling / Applications
  • Dashboard
  • Solutions
    For Individuals For Businesses
  • Ranking
  • Newsletter
  • Documentation
    FAQ Support Pro Accounts Pro+ Accounts Business Accounts Gift API CLI/TUI Privacy Troubleshooting Interviews
  • Blog
  • Pricing
  • Gift
    Gift Purchase Gift Redeem
  • About
Log In - Sign Up
  1. Labs
  2. Disk volumes
  3. Troubleshooting

Guide

Concepts and learning path

Troubleshooting

Failure modes and fixes

Cheatsheet

Commands to keep handy

Disk volumes troubleshooting

Partition table corrupted (disk shows wrong size or no partitions)

Backup first: sfdisk -d /dev/sda > sda-backup.txt (do this BEFORE any changes). Use fdisk -l /dev/sda to check visibility. Try parted /dev/sda print to see if GPT backup header is intact. For GPT, restore backup header: gdisk /dev/sda, then r (recovery) and then b (use backup header). For MBR, testdisk /dev/sda can search for lost partitions. Notice that the package testdisk might need to be installed to perform this last task.

Partition/Volume UUID conflict after cloning (Filesystem has duplicate UUID or wrong fs type, bad option, bad superblock)

To safely use the clone alongside the original volume, the filesystem UUID on the cloned volume must be changed to a new unique value. With the cloned volume attached to a working server, identify the clone's device name using lsblk. Assuming the device name is /dev/nvme2n1p1, mount to clean the journal mount -o nouuid /dev/nvme2n1p1 /mnt/clone, then umount the volume with umount /mnt/clone and regenerate the UUID based on the filesystem type: For XFS do xfs_admin -U generate /dev/nvme2n1p1 and for ext2/3/4 do tune2fs -U random /dev/nvme2n1p1. Verify the new UUID is unique with blkid | grep nvme2. Now /etc/fstab can be updated with the clone volume and the volume with the new UUID can be mounted normally with mount -a.

Filesystem full (No space left on device)

Run df -hT to find the full mount. Use du -xhd1 <mount> to locate large directories. Check for deleted-but-open files with lsof +L1 — space is not reclaimed until the process closes the file.

Inodes exhausted (space available, writes fail)

Check df -i. Millions of tiny files (mail queues, caches, sessions) can fill inodes first. Find dense directories with find /path -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -n or remove/rotate the offending data.

Mount missing after reboot

Inspect /etc/fstab for typos, wrong UUID, or missing nofail on non-critical mounts. Run mount -a to test. Check boot logs: journalctl -b | grep -i mount.

mount: wrong fs type, bad option, bad superblock

Verify filesystem type with blkid or lsblk -f. A mismatched fstab type (ext4 vs xfs) or corrupted superblock causes this. Run fsck on an unmounted filesystem if needed.

target is busy on umount

A process still has files open under the mount. Find it with lsof +f -- /mount/point or fuser -vm /mount/point. Stop the service or use a lazy unmount only when you understand the risk: umount -l.

Filesystem remounted read-only

Often I/O errors or a full disk triggered a protective remount. Check dmesg | tail for disk errors. Free space, fix hardware issues, run fsck, then mount -o remount,rw /.

LVM: extended disk but filesystem unchanged

Storage expansion may involve up to four layers, all of which must be checked and potentially resized.:

# 1. Partition (if not whole-disk PV) growpart /dev/nvme0n1 1 # 2. Physical volume pvresize /dev/nvme0n1p1 # 3. Logical volume lvextend -l +100%FREE /dev/myvg/mylv # 4. Filesystem resize2fs /dev/myvg/mylv # ext4 xfs_growfs /mount/point # xfs — needs mount point

Confirm each step with lsblk, pvs, lvs, and df -h.

Device name changed (/dev/sdb became /dev/sdc)

Kernel enumeration order can shift when disks are added. Prefer UUIDs in fstab:

blkid /dev/sdb1 # /etc/fstab: UUID=abc-123-def-456 /data ext4 defaults 0 2

Debugging workflow

1. Map the storage stack

lsblk -f df -hT df -i

2. Find what's consuming space

du -xhd1 /var | sort -h lsof +L1 | head

3. Validate fstab before reboot

findmnt --verify mount -a

Emergency space cleanup (safe patterns)

# Truncate a runaway log (keeps inode, frees space) # If the file is still open by a process, truncation frees space immediately. # Deleting the file would not. > /var/log/huge.log # Vacuum systemd journal (adjust size) journalctl --vacuum-size=500M # Remove old kernels (Debian/Ubuntu) apt autoremove --purge

Avoid deleting files from /proc, /sys (they are virtual filesystems — nothing there takes disk space and you can't meaningfully "delete" files from them anyway.), or random paths under /var/lib without knowing what owns them.

Practice scenarios

Hands-on Disk volumes scenarios on live Linux VMs: disk volumes

Cheatsheet →
SadServersSadServers

Real-world Linux and DevOps scenarios for hands-on learning and technical assessment.

Uptime Robot ratio (30 days)
Product
  • Scenarios
  • For Individuals
  • For Businesses
  • Pricing
Resources
  • FAQ
  • Blog
  • Newsletter
Company
  • About Us
  • Support
  • Privacy Policy
  • Terms of Service
  • Contact
Connect With Us
info@sadservers.com

Made in Canada 🇨🇦
Updated: 2026-06-13 16:06 UTC – 2d2950a