A production server’s disk is filling up. Walk through how you’d diagnose and fix it.
Start with df -h to confirm which mounted filesystem is actually full, don’t assume it’s the obvious one, /var filling from logs is far more common than / or /home. Then use du -sh /* at the root of the full filesystem and walk down directory by directory to find where the space actually went, rather than guessing.
The gotcha every senior admin checks
If du and df disagree, meaning du says the visible files only account for a fraction of used space, that means a process is holding a deleted file open. Deleting a log file doesn’t free its space until every process with that file descriptor open closes or restarts. Find it with lsof +L1 (lists open files with a link count of zero, meaning deleted but still held), then either restart the offending process or, if that’s too disruptive, truncate the file descriptor directly with : > /proc/<pid>/fd/<fd> as an immediate, non-disruptive fix.
Best practice
Once the immediate fire is out, fix the actual cause: set up log rotation (logrotate) with sane size and retention limits if logs were the culprit, or add disk usage alerting at 80% so this never becomes a 2am page again. A one-time cleanup without addressing rotation just delays the same incident.
Edge case interviewers probe for
Ask what you’d do if the disk hit 100% and services are actively failing to write, meaning you can’t even write new log files or often can’t SSH in cleanly if the OS needs disk space for session files. Know that you may need out-of-band access (console, IPMI) and that deleting files under extreme pressure should target known-safe things first (rotated old logs, package caches like apt/yum cache) before touching anything application-related.
Common mistake
Deleting a large file that a running process still has open and being confused when df doesn’t show the space freed. This is the single most common Linux disk-space gotcha and a strong signal the candidate has actually operated production Linux systems.
What the interviewer is checking
Whether you know the difference between “file removed from the directory listing” and “space actually freed on disk,” and whether you fix the underlying cause (rotation, alerting) rather than just clearing the symptom.
Imagine a library where you “remove” a book from the catalog by crossing its name off a list, but the book itself still physically sits on the shelf taking up space until the last person reading it returns it. If someone’s still reading that book, crossing it off the catalog doesn’t free up shelf space, the book is still there.
That’s exactly what happens when you delete a log file a running program still has open: the file’s entry disappears from the folder listing, but the actual disk space isn’t freed until the program closes the file or restarts. Checking lsof for deleted-but-still-open files is like checking who still has a “removed” book checked out.
Why interviewers ask this
It’s one of the most common real Linux admin incidents, and the deleted-but-open-file gotcha specifically separates people who’ve actually operated servers from those who’ve only read about it.
What a strong answer signals
That you diagnose before deleting anything, and that you fix the root cause (rotation, alerting) instead of just clearing space once.
Common follow-ups
- How would you set up alerting before this becomes an emergency?
- What’s the difference between inode exhaustion and space exhaustion?
- How would you safely truncate a log file that’s actively being written to?
Advanced variation
“The disk shows space available but you can’t create new files,” which expects knowledge of inode exhaustion, a separate resource from raw disk space, checked with df -i.
A web server’s /var partition hit 100% overnight. df -h pointed to /var, but du -sh /var/* only accounted for 40% of the reported usage. Running lsof +L1 revealed the application server was still holding a 30GB debug log open that had been deleted during a manual cleanup earlier that day without restarting the process. Truncating the open file descriptor directly freed the space immediately without needing a disruptive restart, and adding a logrotate rule with a 500MB size cap prevented the same log from growing unbounded again.
# 1. Confirm which filesystem is actually full
df -h
# 2. Find where the space went, one level at a time
du -sh /var/* 2>/dev/null | sort -rh | head -10
# 3. If df and du disagree, look for deleted-but-open files
lsof +L1 2>/dev/null
# 4. Safe immediate fix: truncate the open file descriptor
# (replace PID and FD with values from the lsof output above)
: > /proc/PID/fd/FD
# 5. Long-term fix: add a logrotate rule
# /etc/logrotate.d/myapp
# /var/log/myapp/*.log {
# size 500M
# rotate 5
# compress
# missingok
# }- 1Use df -h to find the full filesystem, then du -sh to find what’s actually consuming space.
- 2If du and df disagree, a process is likely holding a deleted file open; find it with lsof +L1.
- 3Truncating an open file descriptor frees space immediately without a disruptive restart.
- 4Fix the root cause with logrotate and disk alerting, not just a one-time cleanup.
- 5Inode exhaustion is a separate failure mode from space exhaustion; check both with df -i and df -h.