Linux Interview Questions (2026): 33 Commands & Scenarios, With Answers

The FHS defines a standard layout most distros follow closely enough that muscle memory transfers between them. /etc holds static configuration, things a human or a package manager edits and that don't change on their own. /var holds variable data that grows and shrinks at runtime: logs, mail spools, package caches, databases in some setups. /usr holds installed software and the bulk of user-facing binaries, historically kept separate from the root filesystem so it could be mounted read-only or shared over a network.

A candidate who says config lives in /var, or that logs live in /etc, hasn't actually gone looking for either one at 2am.

The first character is the file type (- for a regular file, d for a directory, l for a symlink). The next nine split into three groups of rwx: owner, group, everyone else. r is 4, w is 2, x is 1, and you add them per group. chmod 750 sets owner to 7 (rwx), group to 5 (r-x), and everyone else to 0 (nothing), which prints as -rwxr-x---.

chmod 750 deploy.sh
ls -l deploy.sh
# -rwxr-x--- 1 alice devs 812 Jan 4 09:12 deploy.sh

Candidates who've only ever used chmod +x tend to freeze the first time an interviewer asks for the octal value directly instead of the symbolic form.

chmod changes what the permission bits allow. chown changes who owns the file, the user and the group both, with chown user:group file. They're independent, changing one doesn't touch the other.

A regular user can chmod their own files freely, but can't chown a file to give it away to someone else, not without root or sudo. That restriction exists so a user can't dodge disk quotas by handing a huge file off to somebody else's ownership. -R recurses through a directory for both commands, and it's worth pausing before running either recursively on something like /, which has actually happened to people.

They're two different syntax traditions that both survived into modern Linux. ps aux uses BSD-style flags (no dash), sorts loosely by process start, and shows %CPU and %MEM columns directly. ps -ef uses UNIX System V-style flags (with a dash), shows PPID (parent PID) more prominently, and skips the CPU and memory percentages by default.

Neither is more correct, and most Linux ps implementations (procps-ng) accept both syntaxes on the same box. Pick one and know its columns cold; interviewers care less which one and more that you can read PID, PPID, STAT, and the command off whichever you land on.

-i ignores case. -v inverts the match, printing lines that don't match instead of ones that do. -c prints a count of matching lines instead of the lines themselves. -r recurses through a directory tree.

grep -i "error" app.log
grep -v "DEBUG" app.log | grep -c "WARN"
grep -rn "TODO"./src

-n adds line numbers, which matters more than it sounds like once you're grepping something you're about to open in an editor. Combining flags (-rin) is normal, not showing off.

Both tell the kernel which interpreter should run the script. #!/bin/bash hardcodes the path, which works fine as long as bash actually lives at /bin/bash, true on most Linux distros. #!/usr/bin/env bash instead asks env to find bash wherever it sits in the current PATH, which matters more than it sounds on systems where it isn't at /bin/bash, some macOS setups with a newer Homebrew bash, some minimal containers, NixOS by design.

It's a small thing, but reaching for the env form by default is one of those tells that a candidate has actually shipped a script that ran somewhere other than their own laptop.

ifconfig comes from the older net-tools package, which stopped seeing meaningful upstream development years ago and doesn't know about newer kernel networking features at all. ip, part of iproute2, is the actively maintained replacement and the one that actually understands things like multiple routing tables and modern interface naming.

ip addr show
ip route show
ip link set eth0 up

Plenty of boxes still have ifconfig installed out of habit and it still basically works for a quick look, but if an interviewer asks you to do anything beyond "show me the IP," reaching for ip is the answer that signals current knowledge instead of a decade-old habit.

df -h reports space at the filesystem level, reading block-allocation counters the kernel already tracks, fast, but it has no idea which files those blocks belong to. du -sh actually walks a directory tree and sums up the apparent size of every file underneath it, which is why it's slower and why it only ever reports on the part of the tree you point it at.

df -h /
du -sh /var/log

Run both against the same mount point and they usually roughly agree. When they don't, that gap is worth chasing, and the next question is exactly why.

username:x:UID:GID:comment:home-directory:shell, in that order. The x in the second field is a placeholder, not the actual password hash. Password hashes moved into /etc/shadow decades ago, a file only root can read, because /etc/passwd itself has to stay world-readable so ordinary tools can resolve usernames to UIDs, and leaving real hashes sitting in a world-readable file was exactly the kind of thing that got exploited before the shadow suite existed.

getent passwd alice
id alice

getent passwd works whether user info comes from /etc/passwd directly or from something like LDAP, which plain cat /etc/passwd won't show you on a box using centralized auth.

minute, hour, day-of-month, month, day-of-week, in that fixed order, followed by the command to run. 0 3 * * * matches minute 0, hour 3, every day of month, every month, every day of week, so it fires at 3:00 AM daily.

0 3 * * * /usr/local/bin/backup.sh
@reboot /usr/local/bin/startup-check.sh

@reboot is a special string standing in for the whole five-field schedule, running the command once at system startup instead of on any recurring interval. crontab -e opens the current user's crontab for editing; crontab -l lists it without opening an editor.

-u nginx scopes the output to just that unit's log entries instead of the entire system journal. -f follows the log live, the same idea as tail -f but reading from the binary journal instead of a text file.

journalctl -u nginx -f
journalctl -u nginx --since "1 hour ago"
journalctl -p err -b

-p err filters to error-priority-and-above entries, and -b scopes to the current boot. Some services still write their own plain-text logs under /var/log alongside whatever systemd captures, nginx's access log among them, and journalctl won't show you that traffic log at all since it only captures what a service writes to stdout, stderr, or the systemd journal API directly.

An inode is a data structure the filesystem keeps for every file that stores everything about the file except its name and its actual contents. That means the owner uid and gid, permission bits, timestamps (atime, mtime, ctime), file size, a link count, the file type (regular, directory, symlink, device), and pointers to the data blocks on disk. Two things trip people up: the file name lives in the directory entry, not the inode, which is why a hard link can point at the same inode under two completely different names in two different directories. And ctime is not "creation time," it's the time the inode's metadata last changed, chmod, chown, and rename all bump it, which people confuse with mtime constantly.

You can see all of this directly with stat on a file, which prints the inode number and every field above. If df -h shows plenty of free space but new files still fail to create, check df -i instead. Filesystems have a fixed number of inodes set at creation time (mkfs), and a directory full of millions of tiny files can exhaust that count long before it exhausts the actual bytes.

stat myfile.txt
df -h /var
df -i /var

PATH is a colon-separated list of directories that the shell searches, in order, when you type a command name without a full path. Bash doesn't inherently know where ls lives, it walks each directory in PATH left to right, something like /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/bin, and runs the first matching executable it finds with that name. If two directories both contain an ls, the one listed earlier in PATH wins, which is exactly the situation when someone installs a second copy of a tool and gets confused about why which python still points at the old one.

echo $PATH shows the current value, and which -a ls (or type -a ls) lists every match across the whole search path, not just the first one. PATH is scoped per shell session, so export PATH=... only changes the current shell and its children. That's why editing.bashrc and checking an already-open terminal shows no change, the new value only applies to shells started after the edit unless you explicitly source ~/.bashrc to reload it.

Every process starts with three open file descriptors, 0 for stdin, 1 for stdout, and 2 for stderr. Stdin is where a program reads input from unless told otherwise, stdout is where normal output goes, and stderr is a separate channel just for error and diagnostic messages, kept apart from stdout so the two can be handled differently. That separation is what lets you pipe a program's real output into another tool while its warnings still print to your terminal instead of getting mixed into the data.

To send only stderr to a file: command 2> errors.log. To send only stdout: command > out.log. To send both to the same file you redirect stdout first, then point stderr at wherever stdout is now going: command > combined.log 2>&1. Order matters here, writing 2>&1 before the > target just duplicates the current terminal output, not the file. command 2> /dev/null is the standard move when you want to keep a script's normal output but throw away warnings cluttering the log.

A package manager resolves dependencies, downloads a package from a configured repository, verifies its signature, unpacks files into the right locations across the filesystem, not just one binary but config files under /etc, man pages, systemd unit files, shared libraries, runs any pre or post install scripts, and records what got installed so it can be upgraded or removed cleanly later. apt install nginx on Debian or Ubuntu pulls.deb packages from apt repositories; dnf (or the older yum) does the equivalent on RHEL, Fedora, or CentOS with.rpm packages, and the two package formats aren't compatible with each other.

Copying a binary from another server skips all of that. It might happen to work if every shared library version matches exactly, but nginx dynamically links against libssl, libpcre, zlib and others, and a mismatched or missing version on the target box gets you an "error while loading shared libraries" failure at runtime instead of a clean error at install time. You'd also lose the systemd unit file, default config, and log rotation config that the package would have installed, plus any clean way to remove it later. ldd /usr/sbin/nginx on the source machine shows exactly which shared libraries it depends on, which is the fast way to see why a copied binary would or wouldn't run somewhere else.

A hard link, made with ln original.txt hardlink.txt, points at the exact same inode as the original file, same data blocks, same permissions, same everything except the filename. Delete original.txt and hardlink.txt still works fine, because the underlying data only goes away once the inode's link count drops to zero and nothing has it open.

ln original.txt hardlink.txt    # hard link, same inode
ln -s original.txt symlink.txt   # symlink, separate inode, stores a path
stat -c '%i %n' original.txt hardlink.txt symlink.txt

A symbolic link, ln -s, is a different inode that just stores the path to the target as its content. Delete the original and the symlink still exists, it just points at nothing, a dangling link that errors out the moment you try to open it. Symlinks can cross filesystems and point at directories. Hard links can't do either, they only work within a single filesystem, and you can't hard link a directory yourself (the. and.. entries mkdir creates are hard links, but that's the kernel doing it, not something a tool lets you call directly).

SGID on a directory (chmod g+s shared/) makes every new file created inside inherit the directory's group instead of the creating user's primary group, and it propagates to subdirectories created inside it too. That's the fix for a shared team folder where files keep landing with the wrong group and nobody can read each other's output.

The sticky bit (chmod +t, or the leading 1 in chmod 1777) does something unrelated: on a world-writable directory like /tmp, it stops any user from deleting or renaming a file they don't own, even though the directory's own write permission would otherwise allow it. Without it, /tmp being 777 would let anyone delete anyone else's temp files.

D is uninterruptible sleep, almost always a process waiting on kernel-level I/O, a slow disk, a hung NFS mount, a device driver that isn't responding. The kernel deliberately won't deliver any signal, SIGKILL included, to a process in that state, because letting a signal interrupt it mid-syscall could leave a kernel data structure half-updated.

ps -eo pid,stat,comm | awk '$2 ~ /D/'

The process usually clears on its own once the I/O it's stuck on finishes or times out. If it never clears, the fix is almost never the process itself, it's whatever it's stuck talking to: an unresponsive disk, a dead NFS server, or a USB device that fell off the bus.

Lower its scheduling priority with nice or renice rather than killing it. Niceness ranges from -20 (highest priority) to 19 (lowest), default 0, and a plain user can only raise their own process's niceness, never lower it, since lowering niceness raises priority and a user shouldn't be able to grab more CPU than the scheduler intends without root.

nice -n 10./batch_job.sh    # start it deprioritized
renice 10 -p 4821        # or deprioritize one already running

renice -p works on a PID that's already running, nice only applies when you're starting a new one. Reaching for renice before reaching for kill is usually the answer interviewers want to hear, since killing a three-hour job to save an interactive terminal is rarely the actual trade-off worth making.

awk splits each input line into fields on whitespace by default, $1 is the first field, $2 the second, and so on. $NF isn't a fixed number, it's the last field on that specific line, whatever its position, which makes it useful when field count varies row to row. $0 is the whole line, unsplit.

ps aux --sort=-%mem | awk '{print $4, $11}' | head -6
awk -F: '{print $1}' /etc/passwd

-F sets a different field separator, colon for /etc/passwd instead of the default whitespace. Interviewers use this question to see whether a candidate reaches for awk at all, or defaults to piping everything through grep and cut and losing the actual field-aware part of the tool.

s/foo/bar/g is a substitute command: find foo, replace with bar, g means every occurrence on the line, not just the first. -i edits the file in place instead of printing the result to stdout.

sed -i 's/foo/bar/g' file.txt     # GNU sed (Linux)
sed -i '' 's/foo/bar/g' file.txt    # BSD sed (macOS)
sed -n '5,10p' access.log       # print only lines 5-10

GNU sed, the one on basically every Linux distro, accepts -i with no argument for an in-place edit with no backup. BSD sed, the default on macOS, requires an explicit backup suffix argument right after -i, even if that argument is just an empty string. Run the Linux-style command unmodified on a Mac and it either errors or, worse, treats the next argument as the script and silently does nothing useful. I've seen this exact mismatch break a script that worked fine in CI and failed the instant someone ran it locally on a laptop.

set -e exits the script as soon as any command returns non-zero, with real exceptions that catch people off guard. A command that's part of an if, while, or until condition doesn't trigger it, since a non-zero exit there is the expected way conditionals work, not a failure. A command on the left side of && or || is exempt for the same reason. And inside a pipeline, only the exit status of the last command counts by default, so grep failing to match anything in the middle of a three-command pipe won't stop the script under plain set -e.

set -euo pipefail
cmd1 | cmd2 | cmd3  # pipefail makes ANY failure here stop the script

set -o pipefail closes that last gap by making the whole pipeline's exit status reflect any command in it that failed, not just the last one. Scripts that only set -e and skip pipefail are a common source of "it kept running after something clearly broke."

netstat gathers its information by parsing text files under /proc/net, which gets slower as connection count grows, since it's re-reading and re-parsing plain text on every call. ss talks to the kernel directly over netlink sockets, which is why it stays fast even on a box with tens of thousands of open connections, a real difference, not just a style preference.

ss -tulpn
# -t tcp, -u udp, -l listening only, -p process, -n numeric ports

netstat isn't gone, plenty of scripts and habits still reach for it, but ss is the one that shows up in current documentation and the one worth defaulting to.

curl -I sends a request and prints only the response headers, status code, content-type, server header, without pulling down the body, useful for checking whether an endpoint is up and what it's claiming about itself without downloading a whole page. dig queries DNS directly and shows the full resolution chain: the answer, the authoritative name servers, and the TTL on the record, which ping never surfaces since ping only cares about the final resolved IP.

curl -I https://example.com
dig example.com +short
dig example.com NS

ping tells you whether a host responds to ICMP at all, which is a narrower and sometimes misleading signal since plenty of production hosts block ICMP entirely while serving HTTP traffic just fine.

Usually not. Linux uses spare RAM as page cache for recently read files instead of leaving it idle, since unused memory doing nothing is arguably wasted memory. The used column includes that cache, which makes a healthy box look almost full even when it isn't under any real pressure.

free -h
#        total  used  free  shared buff/cache available
# Mem:      31Gi  6.2Gi  1.1Gi  412Mi  24Gi    24Gi

The available column, added to free some years back, is the number that actually matters, it estimates what the kernel could hand a new process right now, including memory it would happily reclaim from cache under pressure. Adding free plus buffers plus cached by hand the way admins used to do it is close but not exact, and available exists specifically so nobody has to do that math themselves anymore.

su switches to another user's shell, root by default, and needs that target user's own password (or root's, if you're already root). sudo runs one command as another user, checked against rules in /etc/sudoers, using your own password, not the target's.

That difference is the whole reason sudo won out over shared root logins on most teams: every sudo invocation gets logged with the actual user who ran it, access can be revoked per person without rotating a shared secret everyone knows, and sudoers can restrict a specific user to a specific set of commands instead of handing over the whole shell. Losing an employee who had the root password used to mean rotating it everywhere. Losing one with sudo access means removing one line.

start runs a unit right now. enable creates a symlink from the unit into a target's.wants directory (WantedBy=multi-user.target in most service files) so it starts automatically on the next boot. They're independent axes: you can enable a service without starting it yet, or start one without enabling it, and it won't survive a reboot.

systemctl start nginx
systemctl enable nginx
systemctl status nginx
journalctl -u nginx -f

disable removes that boot-time symlink, but the unit file itself still works, someone can still systemctl start it manually. mask goes further: it hard-links the unit to /dev/null, which blocks it from starting at all, manually or automatically, until it's explicitly unmasked. Reaching for disable when the actual goal is "nobody should ever be able to start this" is a common miss, since disable alone doesn't stop that.

journalctl -u <service> -b -1 pulls that unit's log from the previous boot, not the current one, which is exactly what you need after an unexpected reboot wiped the terminal you'd have been watching. journalctl -b -1 with no unit filter shows the entire previous boot's system log, useful for spotting a kernel panic or an OOM kill that took the whole box down rather than just one service.

journalctl -u myapp -b -1
journalctl -b -1 | grep -i "oom|panic"

If the box has systemd-coredump enabled, coredumpctl list and coredumpctl gdb narrow it down to the exact crashing process and let you inspect the actual core, but plenty of boxes don't have that turned on by default, so the journal is often the only trail you get.

-type f restricts the match to regular files so directories don't get swept up, and -mtime +7 matches files whose content was last modified more than 7 days ago. find's time math works in whole 24-hour blocks rather than calendar days, which catches people off guard right at the boundary. The semicolon form runs a brand new rm process once per matching file, so a directory with 50,000 old files forks rm 50,000 times. Swapping the terminator to a plus sign batches as many matched paths as fit on one command line into a single rm invocation, which is dramatically faster and is essentially what piping through xargs accomplishes manually.

find /var/log -type f -name "*.tmp" -mtime +7 -exec rm {} +

Two habits senior engineers actually apply here: run the identical find command without -exec first to see exactly what would match before attaching rm to it, and quote the -name pattern so the shell doesn't glob-expand it before find ever sees it. It's also worth remembering -mtime tracks content modification time; if you actually need files untouched (not even read) for 7 days, that requires -atime, which only works reliably if the filesystem is tracking access times at all, and plenty are mounted noatime for performance, which makes -atime useless there.

xargs takes lines of input and builds one or more command lines out of them, which matters any time the command you want to run doesn't itself read a list of paths from stdin. grep is the common example, grep doesn't take a file list on stdin by itself, so find. -name "*.log" | xargs grep -l ERROR runs grep across everything find turned up. Historically xargs was also how people batched find results into one command before the -exec plus terminator existed, and it's still more portable since not every find implementation supports that terminator.

find. -name "*.log" -print0 | xargs -0 grep -l ERROR

The footgun is filenames containing spaces or newlines. Plain xargs splits input on whitespace, so a file named "my report.txt" becomes two separate arguments, "my" and "report.txt", and either fails outright or, worse, silently operates on the wrong thing. The fix is find... -print0 paired with xargs -0, which uses a null byte as the separator instead of whitespace, and a null byte can never legally appear inside a filename, so it's the one delimiter that's always safe. Anyone who's gotten burned by this once never forgets the -print0 again.

Each line in /etc/fstab tells the system how to mount a filesystem at boot: device (or UUID/LABEL), mount point, filesystem type, a comma-separated list of options, and two numbers for dump backup and fsck pass order. Using UUID= instead of a raw device path like /dev/sdb1 is standard practice now because device letters can shift between reboots, a new drive added, a controller reordered, while the UUID stays tied to that specific filesystem regardless.

The three options matter most on filesystems where untrusted content lands, a shared /tmp, an NFS mount, a USB drive someone plugs in. noexec blocks binaries on that mount from being executed directly, nosuid strips the effect of setuid and setgid bits so a malicious binary can't silently escalate to another user's permissions, and nodev prevents device files on that mount from being interpreted as real device nodes. None of these is airtight on their own, you can still copy a noexec binary elsewhere and run it, or invoke it through an interpreter, but stacking all three on something like /tmp meaningfully raises the bar and is standard hardening on any box that isn't fully trusted end to end.

Swap is disk space the kernel uses as overflow when physical RAM comes under pressure, pages that haven't been touched recently get written out so the RAM they occupied can go to something that needs it right now. On a server that never runs low on memory, swap mostly sits idle, but it still serves a purpose beyond overflow: it gives the kernel somewhere to put genuinely cold pages, an idle daemon's memory, say, so page cache for actively used files gets more room to work with, and it's also required for hibernation on machines that use it.

vm.swappiness, a value from 0 to 100 with 60 as the default on most distros, tunes how aggressively the kernel reaches for swap versus reclaiming page cache. A low value like 10 tells the kernel to strongly prefer dropping cache pages and only swap application memory as a last resort, the usual choice on a database server where you want the working set to stay in RAM and let disk-backed cache shrink first. A value near 100 lets the kernel swap application memory out more readily to preserve more page cache, which can help a system that's mostly serving static files. The real gotcha in production is that even a small amount of swap usage on a latency-sensitive service, a JVM app in particular, can cause massive GC pause spikes the moment any of its heap gets swapped out, so plenty of teams just set swappiness very low or disable swap entirely on those hosts instead of tuning around it.

Every process has a limit on how many file descriptors it can hold open at once, sockets, open files, and pipes all count against it. The default soft limit on most distros is something modest like 1024, fine for a shell but nowhere near enough for something like a busy Nginx or a JVM app handling thousands of concurrent connections, each holding at least one socket fd open. Once a process hits that ceiling, new connections and new file opens start failing with EMFILE, which shows up in logs as exactly that message.

ulimit -n shows the current shell's limit, but the fix for a service isn't running ulimit -n 65536 in a terminal, that only affects that shell and anything spawned from it afterward, not a systemd-managed daemon. For a systemd unit you set LimitNOFILE=65536 under [Service] in the unit file or an override drop-in, and for anything still relying on the older mechanism you'd add a line to /etc/security/limits.conf like appuser soft nofile 65536. Checking /proc//limits on the actual running process confirms the limit that took effect, since a misconfigured PAM stack or a systemd default can silently cap you below what you thought you set.

A zombie, state Z in ps, is a process that already finished executing but still has an entry in the process table because its parent hasn't called wait() to collect its exit status yet. It isn't consuming CPU or memory beyond that one process table slot, it's just waiting to be reaped. An orphan is the opposite situation, a process whose parent died before it did, which the kernel handles by reparenting it to init, or whatever holds PID 1, systemd on most modern distros, so it never floats around without a parent.

A handful of zombies usually means nothing, they clear the moment the parent finally calls wait, or when the parent itself exits and the zombie gets reparented to PID 1, which reaps it immediately. The real problem is a parent with a bug that never reaps its children at all, in that case zombies pile up one per finished child, and since the process table has a fixed size, enough of them can eventually block new processes from being created system-wide. That's the actual failure mode, not resource usage from the zombies themselves. You can't kill a zombie directly, kill -9 does nothing to a process that's already dead, the only fix is getting the parent to reap it or killing the parent so init inherits and cleans up.

Bash reads different startup files depending on whether a shell is interactive and whether it's a login shell. An interactive login shell reads /etc/profile and then the first of ~/.bash_profile, ~/.bash_login, or ~/.bashrc that it finds. A non-login interactive shell, which is what most terminal emulators actually open, reads ~/.bashrc directly. Non-interactive shells, exactly what cron or a remote ssh command spawns, don't read either one by default, which is precisely why aliases and PATH changes defined only in.bashrc silently don't exist in that environment.

The fix depends on what's missing. If it's PATH, set it explicitly at the top of the script or the crontab entry rather than relying on an inherited environment. If it's a variable or function you genuinely need, source the file explicitly near the top of the script, though needing to do that is usually a sign the logic belongs in a real script rather than an interactive-only alias to begin with. Aliases specifically only expand in interactive shells even when the defining file is sourced, so leaning on an alias inside a non-interactive script is generally the wrong tool no matter what you source; a shell function or an actual script does the same job without that restriction.

A raw partition has a fixed size the moment you create it, and growing it later usually means unmounting, resizing the partition table entry, and hoping nothing else on the disk is in the way. LVM adds a layer of abstraction: physical volumes, real disks or partitions, get pooled into a volume group, and logical volumes are carved out of that pool to behave like partitions to the filesystem sitting on top, but they can be resized, or even span multiple physical disks, without touching the underlying partition table at all. Growing a logical volume that's filling up is routine work, extend the LV, then resize2fs or xfs_growfs the filesystem on top, both while it's mounted and in active use in most setups.

LVM also gives you snapshots, a point-in-time, copy-on-write view of a logical volume that's genuinely useful for taking a consistent backup of a live database without stopping it, or testing an upgrade with a rollback path available. The tradeoff is an extra layer of indirection, recovering a wrecked filesystem takes a bit more work when you first have to reconstruct which physical volumes map to which logical volumes, and it's one more concept that has to be understood correctly during disaster recovery rather than "here's the partition, here's the data." For a single-disk box that will never need resizing or snapshots, plain partitions are simpler and skipping LVM entirely is a fine call.

They're not three competing firewalls, they're layers. Actual packet filtering happens in the kernel's netfilter subsystem. iptables was the traditional userspace tool for writing rules into it, evaluated top to bottom in ordered chains like INPUT, OUTPUT, and FORWARD, where the first matching rule usually wins. nftables is the newer kernel subsystem and replacement backend, faster and with cleaner syntax, and modern distros now ship iptables as a compatibility shim that translates old-style commands into nftables rules rather than using the legacy path directly. firewalld sits above both, a daemon with a "zones" abstraction, public, trusted, internal, that you configure through firewall-cmd, and it writes the actual low-level rules into nftables or iptables on your behalf.

The failure people actually run into is having firewalld active and separately running raw iptables commands, or an automation playbook that does, since the two don't share state. firewalld can silently overwrite or ignore rules added by hand the next time it reloads, and vice versa. The fix is picking one management layer per box and sticking with it; if firewalld is running, manage everything through firewall-cmd --permanent plus a reload rather than hand-editing rules alongside it. systemctl status firewalld tells you which one's actually in charge before touching anything.

cp -r only works locally or through something like an NFS mount, has no concept of resuming, and if it's interrupted you're starting over from zero with no way to know what already made it across intact. scp copies over SSH and is fine for a one-shot transfer, but the same problem applies, if the connection drops at 190GB you're restarting the whole transfer, since scp has no way to skip files that already arrived correctly.

rsync is built around exactly this problem. It compares source and destination first, by size and modification time by default, or a full checksum with -c, and only transfers what's actually different. If a transfer is interrupted, --partial keeps the incomplete file instead of deleting it, so a second run can pick up close to where it left off rather than restarting from scratch. For an initial full copy, rsync -avz --progress --partial user@host:/source/ /dest/ over SSH is the standard move, -a preserves permissions, ownership, timestamps and symlinks, -z compresses in transit which helps a lot on a slow link and barely at all on a fast local one, and running the identical command again after any interruption is what makes it resumable, since it skips everything already matching and finishes the rest. The other feature people underuse is --delete, which makes the destination an exact mirror by removing files no longer on the source, great for keeping a backup in sync and genuinely dangerous if you ever get source and destination backwards.

logrotate runs on a schedule, typically daily via cron or a systemd timer, and works through config files under /etc/logrotate.d/, usually one per service, that define how a log should be handled: rotate weekly or at a size threshold, keep some number of old copies, compress the older ones, and run any commands the service needs afterward. A typical rotation renames access.log to access.log.1, shifting older numbered copies up, gzips the older files to save space, and once a configured count is reached, rotate 7 say, deletes the oldest copy entirely so retention never grows unbounded.

The part that actually trips people up is that renaming a file doesn't tell a process that already has it open by file descriptor to start writing somewhere new. A process holds an open fd to the inode, not the filename, so after a naive rotation the service keeps happily appending to what is now access.log.1 internally, while the freshly created access.log stays empty and the old inode silently keeps growing. That's exactly why logrotate configs for something like Nginx include a postrotate block that sends the process a signal, nginx -s reopen, or a plain SIGHUP to the right pid, so it closes and reopens its log file by name and picks up the new one instead of continuing to write into the file that just got renamed out from under it. Forgetting that postrotate hook is the single most common reason someone finds a rotated log file mysteriously still growing weeks after it should have been retired.

You generate a key pair with ssh-keygen, the private key stays on your machine and the public key gets appended to ~/.ssh/authorized_keys on the server. When you connect, the server sends a challenge, your client signs it with the private key, and the server verifies that signature against the public key it already has on file. Your private key itself never crosses the wire, that's the entire point, unlike password auth where the secret has to be transmitted every single time even over an encrypted channel.

sshd is deliberately paranoid about permissions on the client side and will refuse to use a private key, or refuse to trust an authorized_keys file, if it's writable by group or world. The standard permissions are chmod 700 on ~/.ssh itself, chmod 600 on the private key, and chmod 600 on authorized_keys. The reasoning is that if any other local user could write to those files, they could swap in their own key or read yours, so ssh fails closed instead of silently trusting a file it can't vouch for. This usually bites people after copying a.ssh directory from another machine with tar or rsync without resetting permissions, and the fix is just running chmod again rather than anything exotic.

/proc is a virtual filesystem, it doesn't live on disk at all. Every file under it gets generated by the kernel the moment something reads it, which is why /proc/cpuinfo, /proc/meminfo, and /proc/[pid]/status always reflect the live state of the machine instead of a snapshot that goes stale.

That live-state property makes /proc genuinely useful past trivia. kill -0 pid checks whether a process exists without sending a real signal, and underneath it's really just confirming /proc/pid exists and you have permission to signal it. /proc/pid/fd lists every open file descriptor a process holds, which is the fastest way to answer "what does this PID actually have open right now" without installing anything extra.

/usr/bin/passwd is owned by root and has its SUID bit set, so when any user runs it, the process runs with root's privileges instead of the calling user's, just long enough to write the new hash into /etc/shadow, a file regular users can't touch directly. That's the entire reason passwd works without sudo.

chmod 4755 /usr/local/bin/tool  # leading 4 sets SUID
ls -l /usr/local/bin/tool
# -rwsr-xr-x 1 root root 45032... tool

And no, not reliably. The Linux kernel ignores the SUID bit on scripts that start with a shebang line (#!/bin/bash and friends), because of a well-documented race condition between the kernel reading the shebang and the interpreter actually opening the file. Set SUID on a shell script and chmod will happily accept it, ls -l will happily show the s, and it will do nothing at execution time. If you need SUID-style privilege escalation for something scripted, the usual answer is a small compiled C wrapper, not the script itself.

On Linux, kill sends SIGTERM (signal 15) by default, a polite request a process can catch, ignore, or use to run cleanup before exiting. kill -9 sends SIGKILL, which the kernel enforces directly, no handler, no cleanup, no chance for the process to object, which is exactly why it should be the second thing you try, not the first (signal(7), Linux man-pages).

kill 4821    # SIGTERM (15), ask nicely
kill -9 4821   # SIGKILL, no negotiation
kill -1 4821   # SIGHUP (1), often means "reread your config"

SIGHUP (1) originally meant the controlling terminal hung up on a process. Almost nobody relies on that meaning anymore. Most long-running daemons, nginx and sshd among them, repurposed SIGHUP as a signal to reload configuration without a full restart, a detail that trips up candidates who only know SIGHUP as "the one that used to matter for terminals."

awk -v since="$(date -d '1 hour ago' '+%d/%b/%Y:%H')" 
 '$0 > since' /var/log/nginx/access.log 
 | awk '{print $1}' 
 | sort 
 | uniq -c 
 | sort -nr 
 | head -5

awk pulls the first field out of an Nginx combined-log line, which is the client IP. sort groups identical lines together so uniq -c can collapse each run into a count. The second sort -nr sorts those counts numerically, descending, and head -5 keeps the top handful. The date filter up front is the part people skip and then wonder why yesterday's traffic is drowning out the last hour's.

None of these tools was built to talk to any of the others directly, they just share plain text over a pipe, which is the actual design idea behind the whole toolchain and worth saying out loud if an interviewer asks why this approach instead of a script in Python.

If $DIR is unset or accidentally empty, that line expands to rm -rf /, a bare slash with nothing between it and the trailing one, which is exactly the kind of thing that has taken down real production boxes. Unquoted variables also break on any value containing spaces, since bash word-splits an unquoted expansion into separate arguments.

rm -rf "$DIR/"    # quoted: an empty DIR still deletes "/", but at least predictably
set -u         # unset variables become a hard error instead of expanding to nothing
: "${DIR:?DIR must be set}"

Quoting alone doesn't fully save you here, an empty quoted "$DIR/" still resolves to /. The real fix is set -u so referencing an unset variable errors out immediately, plus a guard like the parameter-expansion check above that fails loudly with a message instead of silently walking into a root deletion. I still quote every variable expansion out of habit even when I'm confident it's set, because being confident is exactly the state I was in the one time it wasn't.

sudo ss -tulpn | grep :8080
sudo lsof -i :8080
sudo fuser -n tcp 8080

Any of the three works. ss -tulpn filters to listening sockets and shows the PID and process name in the last column. lsof -i :8080 lists every open file (sockets count as files on Linux) bound to that port. fuser -n tcp 8080 is the terse option, it just prints the PID.

The sudo part matters more than it looks. Without root, ss and netstat still show that a socket exists on that port, but the process name and PID columns come back blank for sockets owned by a different user, since a regular user isn't allowed to see what another user's processes are doing. Run the same command as root and the identity resolves. Once you've got the PID, kill it, or more usefully, ask why it's still running before you kill anything, since "address already in use" after a crash is often a previous instance of the exact service you're trying to restart.

Almost always a deleted-but-still-open file. Delete a file that a running process still has open, and Linux unlinks the directory entry immediately, so du can't see it anymore since du only walks what's actually linked into the directory tree. But the kernel won't reclaim the underlying disk blocks until every process holding that file descriptor closes it or exits, so df still counts every one of those blocks as used.

lsof +L1           # files with a link count under 1: deleted but still open
lsof | grep deleted
: > /proc/<pid>/fd/<fd>    # truncate it live without restarting the process

A log file a service opened at startup and kept writing to after a log rotation deleted the old path is the classic version of this. Restarting the process reopens its log handle against the current path and reclaims the space. If a restart isn't an option right that second, truncating through /proc/pid/fd/N gets the space back without killing anything, though I'd only reach for that on something I fully understood was safe to zero out live.

Cron runs jobs with a minimal environment, not the one your interactive shell builds from.bashrc or.profile. PATH under cron is typically just /usr/bin:/bin, so any command your script calls by bare name that lives somewhere else, a tool installed via a version manager under your home directory, say, simply isn't found, and the job fails before it does anything useful.

0 3 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

Fix it two ways at once: use absolute paths inside the script instead of relying on PATH, and redirect both stdout and stderr to a log file the way the line above does. By default cron only mails output if MAILTO is configured and local mail delivery actually works, which on most modern boxes it doesn't, so the failure output just vanishes into nothing unless you redirect it somewhere you'll actually look.

Linux load average counts processes that are running or in uninterruptible sleep (D state, the same state from the process-signals section earlier on this page), not just ones actively burning CPU. A pile of processes stuck waiting on slow disk I/O or a hung NFS mount inflates load average the same way CPU-bound processes would, even though every one of them is technically idle from the CPU scheduler's point of view.

vmstat 1 5
ps aux | awk '$8 ~ /D/'

vmstat's wa column (I/O wait) is the tell. High wa with low CPU usage and a high load number points straight at storage or network-mounted filesystems, not compute, and reaching for a CPU profiler here is exactly the wrong instinct an interviewer is checking for.

A container is not a lightweight VM, it's an ordinary process running on the host's kernel, just given a restricted view of the system. Namespaces are what create that restricted view. A PID namespace makes a process think it's PID 1 in its own isolated process tree even though the host sees it as some ordinary PID. A mount namespace gives it its own filesystem view, which is how it gets a completely different root filesystem built from image layers. A network namespace gives it its own interfaces, routing table, and IP, connected back to the host through a virtual ethernet pair. A UTS namespace gives it its own hostname, and a user namespace can remap root inside the container to an unprivileged uid on the host, which is what makes rootless containers possible. Each namespace type answers "what can this process see," not "how much can it use."

cgroups answer the second question. A control group lets the kernel enforce hard limits on CPU shares, memory, block IO, and process count for a group of processes, which is exactly what a memory or CPU flag on a container run configures at the implementation level, by writing into cgroup filesystem entries. Namespaces plus cgroups plus a union filesystem for image layers is the entire mechanism, there's no hypervisor, no virtualized hardware, no second kernel. The container's process is directly schedulable by the host kernel like any other process, which is why containers start in milliseconds and VMs don't. The tradeoff is that isolation is only as strong as the kernel's namespace and cgroup implementation, a container escape is fundamentally a kernel bug being exploited rather than a hypervisor being bypassed, which is a big part of why kernel patching matters more for container security than people tend to expect.

When the kernel genuinely can't satisfy a memory allocation and swap, if any, is exhausted too, it invokes the out-of-memory killer rather than let the whole system deadlock. It doesn't kill whatever asked for memory last, it walks every process and computes an oom_score for each one, heavily weighted by how much memory the process is actually using, with some adjustment for things like root-owned processes, and kills whichever has the highest score, on the theory that killing the biggest consumer frees the most memory for the least damage.

That heuristic has no idea which process is actually important to you. A database legitimately using most of a box's RAM as intended, a large buffer pool, say, can easily have a higher oom_score than the small runaway script that's actually leaking, since the kernel only sees memory footprint, not business priority. You influence this through /proc//oom_score_adj, a value from -1000 to 1000 that shifts a process's score, setting it to -1000 makes the kernel effectively never pick that process, which is exactly what protects sshd on some distros by default so you don't lose remote access at the worst possible moment, while a positive value makes a process a preferred sacrifice. The longer-term fix is not relying on the OOM killer as your memory strategy at all, set real limits, cgroup memory caps on containers, sane heap sizes on JVM apps, so a leak gets contained and killed within its own boundary instead of triggering a system-wide OOM event that has to guess across every process on the host.

The first move is strace -p attached to the already-running process, which shows every syscall it's making, or critically, the one syscall it's currently blocked in, without needing to restart anything. If the process is genuinely hung, you'll see it stuck on the last line with nothing new appearing, and that last syscall tells the actual story. Stuck in read() or recvfrom() usually means it's waiting on a socket where the other end never responds, a downstream service that isn't answering, or a connection where a firewall silently drops packets instead of resetting them. Stuck in futex() often points at lock contention between threads. Stuck in a filesystem read against a file descriptor on an NFS mount can mean the NFS server itself has gone unresponsive and the kernel is blocking indefinitely waiting on it.

For a multi-threaded process, strace -f -p follows every thread, not just the main one, which matters because the thread actually holding a lock everyone else is waiting on might not be the one that looks stuck from a coarse view. For a lighter first look before attaching a full tracer, ps -o pid,stat,wchan -p shows which kernel function the process is blocked in without the overhead of strace, useful on a box where you don't want to slow down something that's already struggling. The general rule senior engineers apply here is simple: don't guess, attach and read the actual syscall the process is doing the moment it's stuck, that's almost always enough to tell whether the bug is a network dependency, a lock, disk IO, or something else entirely.