Guide

Concepts and learning path

Troubleshooting

Failure modes and fixes

Cheatsheet

Commands to keep handy

systemd troubleshooting

Start here: `systemctl --failed`

When the problem is unknown, list every failed unit at once instead of guessing which service to check:

systemctl --failed
systemctl list-units --state=failed

Includes services, mounts, timers, and other unit types. Then systemctl status UNIT and journalctl -u UNIT on the one that matters. After fixing: systemctl reset-failed UNIT.

Service failed to start

Run systemctl status myservice for the exit code and last log lines. Full output: journalctl -u myservice -b --no-pager. Check ExecStart= path, permissions, and the User= the service runs as.

Service runs but immediately exits

systemctl start succeeds briefly, then the unit shows inactive (dead) or flips to failed — the main process exited right away. Common causes:

Type=oneshot without RemainAfterExit=yes — systemd marks the unit dead after the command finishes
Script daemonizes itself — forks into the background while Type=simple expects the main process to stay in the foreground
Missing foreground mode — app supports -D, --no-daemon, or run vs start; systemd got a wrapper that exits after spawning

Debug:

systemctl status myservice
journalctl -u myservice

Fix: match Type= to how the process behaves (forking for classic daemons, simple with a foreground command, or oneshot + RemainAfterExit=yes for setup scripts). See Type= mismatches below.

Service not running after reboot

Started but not enabled? systemctl is-enabled myservice should show enabled. Run systemctl enable myservice. Also verify [Install] WantedBy= exists in the unit file.

Restart loop (start-limit-hit)

systemd stopped retrying after too many rapid failures. Check logs for the root cause, fix it, then systemctl reset-failed myservice and systemctl start myservice. Tune RestartSec= and StartLimitBurst= if needed.

Unit changes ignored

After editing a unit file or drop-in, run systemctl daemon-reload before restart. Confirm the effective config with systemctl cat myservice.

Dependency failed or job canceled

A required unit failed first. List dependencies: systemctl list-dependencies myservice. Check Requires= vs Wants= — Requires is hard; failure blocks startup.

Timer not firing

Timers need both .timer and .service units, and the timer must be enabled. Check schedule with systemctl list-timers mytimer.timer and logs with journalctl -u mytimer.service.

`Type=` mismatches

Wrong service type causes hang or premature "active" state:

Type=simple   # default — main process is ExecStart
Type=forking  # classic daemon that forks (legacy apps)
Type=oneshot  # runs once and exits (often with RemainAfterExit=yes)

Environment and PATH issues

Services get a minimal environment — not your shell's .bashrc:

# In the [Service] section:
Environment="PATH=/usr/local/bin:/usr/bin"
EnvironmentFile=/etc/myapp/env

Permission and sandboxing failures

Modern unit files often harden services with sandbox directives. systemd may block filesystem or privilege operations with little obvious error — increasingly common in production and distro-shipped units.

Typical directives that cause silent breakage:

ProtectSystem= — mounts /usr, /boot, /etc read-only (or stricter)
PrivateTmp= — private /tmp and /var/tmp; paths differ from your manual test
ReadOnlyPaths= / ReadWritePaths= — only listed paths writable
NoNewPrivileges= — blocks setuid/setgid and some capability escalations

Symptom: command works when you run it as the service user in a shell, but fails under systemctl start — permission denied, read-only filesystem, or missing socket under /tmp.

systemctl cat myservice | grep -E 'Protect|Private|ReadOnly|ReadWrite|NoNew'
journalctl -u myservice -n 50
# Compare: manual run vs systemd
sudo -u myapp /opt/myapp/bin/server

Fix: add ReadWritePaths=/path/myapp/needs, relax ProtectSystem= in a drop-in, or adjust the app to use allowed directories. Prefer tightening incrementally over disabling all sandboxing.

Debugging workflow

0. Unknown failure — list all failed units

systemctl --failed

1. Status and recent logs

systemctl status myservice
journalctl -u myservice -n 100 --no-pager

2. Effective unit and dependencies

systemctl cat myservice
systemctl list-dependencies myservice

3. Run ExecStart manually as the service user

sudo -u deploy /opt/myapp/bin/server

If manual works but systemd does not, check sandbox directives — see permission and sandboxing failures above.

systemctl show myservice -p ExecMainStatus,Result,FragmentPath

Masking vs disabling

disable stops auto-start; mask prevents start entirely (symlink to /dev/null). Unmask before re-enabling:

systemctl mask myservice    # hard block
systemctl unmask myservice  # undo mask

Practice scenarios

Hands-on systemd scenarios on live Linux VMs: systemd

Cheatsheet →